World Models — Reading & Resources: Deep Dives + Annotated Idea Map
This is the single reading-and-resources hub for the World Models series. It has two halves: Part I — Deep Dives reads each key paper along the same axes (gist · pipeline · benchmark & dataset · results · stated future work · scope & unique ideas) so the page doubles as an idea-generation worksheet; Part II — Thematic Index is the broader, faster annotated map across the active-perception, world-model, and embodied-agent literature. The concepts behind all of this are introduced in Part 0: From Language Models to World Models.
Throughout, a 🎯 marks the papers closest to my current research direction — active, task-aligned perception for embodied agents (“moving to see better”). Annotations are paraphrased in my own words; several entries are fast-moving preprints, so open each link and confirm title/authors/venue before citing in a formal document. Full plain-text citations are in the References.
- Embodied AI Agents: Modeling the World — the position paper that makes the world model the core of embodied agents.
- World-in-World — closed-loop evidence that visual realism ≠ task success.
- WorldPrediction — a benchmark showing high-level procedural planning is largely unsolved.
- V* — "where to look" as a learnable skill inside a multimodal model.
Part I · Deep Dives
A reading list tells you what exists; this part tells you why each paper matters. Papers are grouped; reference numbers in the References follow this order.
Group A — World models & the data/agenda layer
1 · WorldPrediction
Gist. Visual realism ≠ planning ability — high-level, abstract-action planning is largely unsolved, and the authors suspect perception is the bottleneck.
- Problem & pipeline. Do models have a world model good enough for high-level, long-horizon procedural planning (not just low-level motion)? Formalized as a partially observable semi-MDP. Given initial and final visual states, the model must pick the correct action (WM track) or the correctly ordered action sequence (PP track) from counterfactual distractors. The clever trick: “action equivalents” (the same action in a different context) as distractors, so models can’t cheat using low-level background continuity; heavy human filtering.
- Benchmark / dataset / metrics. Two tracks (WM, PP) scored by discriminative accuracy; data drawn from procedural-video sources (COIN, CrossTask, EgoExo4D, IKEA-ASM).
- Results. Best models ≈ 57% (world modeling) and 38% (procedural planning) versus humans near-perfect; bigger models barely help planning.
- Scope & unique ideas. A ready-made target to test whether better perception lifts planning. The “action-equivalents” construction is a clean, reusable way to force semantic over pixel reasoning — worth copying for any new benchmark.
2 · Action100M
Gist. Automated, hierarchical annotation can replace manual labeling at scale, and predictive (JEPA) representations keep improving with more data.
- Problem & pipeline. Action understanding needs huge, open-vocabulary, hierarchically labeled video, but manual labeling doesn’t scale. They build ~100M labeled segments from 1.2M instructional videos with a fully automated pipeline: (i) hierarchical temporal segmentation using V-JEPA 2 embeddings; (ii) multi-level captions arranged as a Tree-of-Captions; (iii) aggregation by a reasoning model (GPT-OSS-120B) under multi-round Self-Refine to reduce hallucinations and emit structured fields (action, actor, captions).
- Benchmark / dataset / metrics. The dataset itself (~100M segments / 1.2M videos); evaluated by zero-shot action-recognition transfer and data-scaling curves. Descends from HowTo100M and the JEPA representation line.
- Results. A trained VL-JEPA model shows consistent data-scaling gains and strong zero-shot action recognition.
- Scope & unique ideas. A data substrate for perception / world-model work; the motion→task hierarchy suits multi-level reasoning. Practical caveat: only a 10% preview is public; the full set is gated.
3 · Embodied AI Agents: Modeling the World
Gist. Perception, planning, and memory should be unified inside a world model; embodiment plus social/mental modeling is the agenda.
- Problem & pipeline. How should embodied agents (avatars, wearables, robots) be built to learn and act like humans? A research-vision paper, not an experiment. Proposes unifying multimodal perception + reasoning-for-action + memory inside a world model, and adds mental world models that infer user intentions and social context (a Theory-of-Mind layer).
- Benchmark / dataset / metrics. None of its own; it organizes the field and references existing benchmarks, contributing a virtual/wearable/robotic agent taxonomy.
- Results. No single model, no accuracy number — its value is the framing.
- Scope & unique ideas. The umbrella that legitimizes a perception-quality-inside-world-models direction. Its Theory-of-Mind thread is itself a possible project (user-intention modeling).
Group B — “Moving to see better” (active perception)
4 · World-in-World 🎯 Core to the project
Gist. Judge world models by closed-loop task success, not pixels — the most direct evidence for a task-coupled metric.
- Problem & pipeline. Do generative world models actually help agents succeed, or do we only measure visual quality? They build the first closed-loop platform with a unified online planner and a standardized action API, across four environments (perception, navigation, manipulation) where task success is the primary metric.
- Benchmark / dataset / metrics. Task success rate (primary) plotted against visual-quality scores (FID/FVD-style), revealing weak correlation.
- Results. High visual quality does not translate into task success; scaling post-training on action-observation data beats upgrading the generator; more inference-time planning compute helps.
- Scope & unique ideas. Cite as the empirical backbone of a perception-aligned, task-coupled reward; reuse its closed-loop, task-success evaluation philosophy to show a quality reward improves outcomes.
5 · MANIQA 🎯 Core to the project
Gist. A single image can be scored for human-perceived quality with no reference, and attention across both channel and space matters.
- Problem & pipeline. No-reference image quality assessment (score perceptual quality from the image alone) was weak on GAN-type distortions. Extract features with a ViT, then a Transposed Attention Block (channel dimension) and a Scale Swin Transformer Block (spatial dimension), with a dual-branch patch-weighted score head.
- Benchmark / dataset / metrics. LIVE, TID2013, CSIQ, KADID-10K; correlation with human MOS via SRCC/PLCC.
- Results. State-of-the-art on those datasets and first place in the NTIRE 2022 no-reference challenge.
- Scope & unique ideas. A concrete starting architecture for a “view-quality” metric Q; learn its MOS-regression recipe and adapt it from “image quality” to “view usefulness for a task.” (Note: Group D shows human-MOS quality transfers poorly to robots — so adapt the architecture, not the labels.)
6 · Rein-EAD (Reinforced Embodied Active Defense) 🎯 Core to the project
Gist. Moving to re-observe, driven by a dense uncertainty reward, beats single-shot perception — the closest existing mechanism to “move to see better.”
- Problem & pipeline. Passive perception is fragile to adversarial and 3D attacks. A recurrent feedback loop takes multiple looks; an uncertainty-aware guided dense reward shapes where to move and observe; scene understanding is rebuilt over steps rather than judged from one frame.
- Benchmark / dataset / metrics. Adversarial robustness / defense metrics under patch and 3D attacks (not perceptual-quality indices).
- Results. Improves robustness while staying computationally efficient.
- Scope & unique ideas. Its RL loop and dense-reward design are directly reusable — swap the uncertainty/defense reward for a human-aligned quality reward.
7 · RL4VLM 🎯 Core to the project
Gist. Attach an RL reward to a perception-capable model and you get an agent — the recipe for turning any reward into behavior.
- Problem & pipeline. VLMs perceive well but decide poorly. Treat the VLM as a policy that reasons step-by-step (chain-of-thought) and then emits actions, trained end-to-end with task reward in gym-like multimodal environments.
- Benchmark / dataset / metrics. Interactive decision tasks (card games, embodied / ALFWorld-style), scored by success/reward.
- Results. Improves decision-making over prompting and supervised baselines.
- Scope & unique ideas. The practical template for wiring a custom (e.g. perceptual-quality) reward into a perception-capable agent; study how they balance reasoning and action.
8 · V* 🎯 Core to the project
Gist. Deciding where to attend, rather than consuming everything at once, is itself a learnable, performance-critical skill — the model-side analog of active perception.
- Problem & pipeline. Multimodal LLMs miss small details in high-resolution images because they look once, globally. The SEAL framework uses context to guide an iterative search that zooms to the relevant region before answering.
- Benchmark / dataset / metrics. Introduces V*Bench for fine-detail visual-QA accuracy.
- Results. Accurate on fine-detail visual questions where standard MLLMs fail.
- Scope & unique ideas. Conceptual grounding for “where to look”; even with a moving robot, the search-then-decide structure transfers.
Group C — Multi-agent & state tracking
9 · Solaris
Gist. World models can be multi-agent and must keep shared views consistent — directly relevant to shared visual understanding.
- Problem & pipeline. Can a video world model serve multiple agents with consistent shared views (a “multiplayer” world model)? Built in Minecraft, predicting consistent first-person views for two players via a staged single→multiplayer pipeline (bidirectional, causal, and Self-Forcing training), backed by a multiplayer data system (~12.6M multiplayer frames).
- Benchmark / dataset / metrics. Co-observation consistency, grounding, memory, movement, and building metrics.
- Results. A technical report — treat the numbers as preliminary.
- Scope & unique ideas. If you keep a multi-agent strand, this is the reference for cross-agent visual consistency, and its consistency metric is reusable.
10 · VSTAT
Gist. Models can describe a frame but cannot track state over time — a concrete, quantified gap that motivates world-model and memory work.
- Problem & pipeline. Can video MLLMs track fine-grained visual state over time (counts, attributes, order)? A benchmark of procedural videos with state-tracking questions: 834 clips and ~1,500 questions across synthetic, self-recorded, and real videos, forcing a running mental model of object states.
- Benchmark / dataset / metrics. VSTAT accuracy, with the large human–model gap as the headline index.
- Results. Best model (~Gemini-3.1 Pro) ≈ 44.4 versus humans ≈ 90.5.
- Scope & unique ideas. Motivation and a possible evaluation axis for world-model / memory work; its human-vs-model gap framing mirrors the broader “metrics don’t equal usefulness” argument.
Group D — Quality + active perception for embodied AI
The trio most directly on the “moving to see better” idea: each uses image quality to guide or judge embodied perception. Two of them carry a finding that reshapes the framing — read the idea seed after this group.
11 · Active View Selector (AVS) 🎯 Core to the project
Gist. Reframes where to capture next for 3D reconstruction / novel-view synthesis as 2D image-quality assessment — go where the current rendering looks worst (“boost where it struggles”) — making active view selection fast and representation-agnostic.
- Problem & pipeline. Given a partial reconstruction, where should the camera go next? Prior methods (ActiveNeRF, FisherRF) compute 3D uncertainty / information gain — slow (FisherRF builds a Hessian over 200M+ parameters, ~5–8 s per view) and tied to a specific 3D representation. AVS instead scores each candidate viewpoint by the predicted quality of its rendering. Since there is no ground-truth image for a candidate view, they train a cross-reference IQA network (inspired by CrossScore) that predicts the SSIM map of a rendering from the rendering plus several real reference views. A lightweight RepViT-backbone variant runs in ~0.5 s.
- Benchmark / dataset / metrics. Reconstruction / NVS quality versus selection cost; zero-shot generalization to egocentric ARIA smart-glasses data it was never trained on; compared against FisherRF and no-reference IQA baselines (MANIQA, MUSIQ).
- Results. Beats FisherRF on reconstruction quality while running 14–33× faster with less than half the GPU memory; notably, plain no-reference metrics (MANIQA, MUSIQ) are already strong baselines for view selection.
- Stated future work. Integrate the low-latency selector into real-time SLAM and AR/VR; use it to guide a user or robot to capture images that achieve an objective (better synthesis or coverage); fine-tune on egocentric data.
- Scope & unique ideas. The closest published thing to “moving to see better,” but the quality it optimizes is reconstruction fidelity (SSIM), not task usefulness — swapping that objective for a task-aligned reward is open space. Selection here is greedy and passive (score all candidates, pick the worst); there is no learned movement policy — an RL agent that learns where to go is unclaimed. The cross-reference trick (“is this view good given the views I already have”) maps neatly onto an agent with memory.
12 · Image Quality Assessment for Embodied AI (Embodied-IQA) 🎯 Core to the project
Gist. Proposes the topic IQA for Embodied AI: predict how usable an image is for a robot’s task rather than how pretty it looks to a human — and shows a large human–robot vision gap.
- Problem & pipeline. Builds a Perception–Cognition–Decision–Execution pipeline (framed via the Mertonian system) and a large database: 1,230 reference images, 30 distortion types at 5 levels (~36,900 distorted images), with 5M+ annotations. Cognition is scored by 15 VLMs, Decision by 15 VLAs (on 7-DoF pose), and Execution by 1,500 real-world robot-arm runs (success = 100; failure deducts centimeters of error; collision = 0).
- Benchmark / dataset / metrics. Correlation of IQA metrics with embodied scores via SRCC/PLCC; Decision-versus-Execution correlation.
- Results. Standard IQA metrics that correlate ~0.9 with human opinion drop to ~0.75 at best (TOPIQ) on embodied data, and no-reference methods fall below 0.6. Distortion sensitivity differs from humans (a denoise can hurt a robot badly, while strong block-interpolation barely matters if it misses the target object). Decision correlates with real Execution above 0.6, but Cognition alone correlates below 0.5 — a VLM’s judgment is not enough.
- Stated future work. Develop more accurate quality indicators for embodied AI; integrate the human, VLM, and VLA paradigms and pick the right one per use case; extend Perception to vision–tactile fusion and Execution to legged / quadruped robots; build an automated real-world pipeline to scale execution labels.
- Scope & unique ideas. Direct evidence that human-aligned quality is the wrong target for a robot. Everything here is static scoring of given images — there is no agent that moves to raise the score; the active-perception loop is missing entirely. They also show no single VLM or VLA is a reliable judge (subject correlations of ~0.25–0.3), an argument for a learned, aggregated quality / reward model.
13 · Embodied IQA for Robotic Intelligence (EPD / MA-EIQA) 🎯 Core to the project
Gist. Builds robot-quality labels purely from task performance (no human in the loop) and ships the first no-reference IQA model designed for robots.
- Problem & pipeline. Same core thesis as Embodied-IQA (robot quality ≠ human quality, invoking the Moravec paradox), but the labels come from RL reward, not opinion. The EPD database has 12,500 reference/distorted pairs from 100 episodes across two tasks (push, pick) in the SAPIEN/ManiSkill simulator, with 25 distortions at 5 levels. The label for each image is the RL episode reward (from PPO, SAC, and TDMPC2 agents) normalized to a 0–5 score. They also propose MA-EIQA: a ResNet-50 backbone, a multi-scale (PANet-style) encoder that fuses semantic and texture detail, and a CBAM-style embodied-attention module — kept lightweight (48.83M parameters, far smaller than MANIQA).
- Benchmark / dataset / metrics. Human-MOS versus embodied-MOS correlation; SRCC/PLCC across 16 IQA methods.
- Results. Human and embodied MOS barely correlate (PLCC ~0.13–0.21); across 16 IQA methods none exceeds ~0.6 SRCC; MA-EIQA reaches SOTA among no-reference models and beats the full-reference ones. Color distortion hurts tasks most; motion blur and noise least — the opposite of human sensitivity.
- Stated future work. Expand the embodied-preference database; add vision–tactile sensing; cover more robot body types.
- Scope & unique ideas. They already use RL reward as the quality label, but only offline, to build a static dataset. Using a learned quality model as a live dense reward that drives an agent’s movement is the step they do not take. The label is task-specific (push, pick); a quality signal that generalizes across tasks — or that an agent can use before it knows the exact task — is open. They vary distortion on fixed viewpoints; you care about viewpoint — nobody has unified distortion-robustness and viewpoint-usefulness under one task-aligned signal.
A cross-cutting idea seed
Read together, these papers point one direction.
Five of them report the same thing from different angles. WorldPrediction, World-in-World, and VSTAT show that today’s models look good but act, plan, and remember poorly, and pixel-level metrics fail to predict task usefulness. Embodied-IQA and EPD sharpen this with hard numbers: human perceptual quality correlates poorly with embodied task success (PLCC as low as 0.13–0.21; human-trained metrics that hit ~0.9 on people fall to ~0.5–0.75 on robots). So a human-aligned view-quality reward — an intuitive first pitch — is probably the wrong target for a robot doing a task.
Three honest ways to respond (picking one sharpens the contribution):
- Pivot the reward from human-aligned to task-aligned view usefulness (what Embodied-IQA and EPD argue for), and make the novelty the active movement none of them do.
- Keep human alignment but scope it to human-facing agents (wearables, assistive glasses), where AVS-style capture guidance and human preference genuinely matter.
- Make the human-versus-task quality gap itself the research question: when do they agree, when do they diverge, can one model serve both.
The unclaimed center. AVS moves the camera but optimizes reconstruction quality with no learned policy; Embodied-IQA and EPD define task-aligned quality but only score static images — nothing moves. Stated precisely, the open problem is: an agent that learns a movement policy whose reward is a learned, task-aligned (not pretty) view-quality signal, evaluated in closed loop by task success. Each half exists in the literature; the combination does not. Concrete openings: a task-aligned view-usefulness model used as a dense intrinsic reward for an active-perception RL policy (EPD’s reward-as-label × AVS’s move-to-improve); a cross-reference usefulness signal for an agent with memory (“is this new viewpoint more useful than the ones I already hold”); a new active-perception benchmark where an agent moves and each viewpoint is labeled by downstream task success (filling the static-only gap in Embodied-IQA / EPD); and a lightweight-by-design reward model (both AVS’s RepViT and MA-EIQA’s 48M-parameter CNN exist because the loop must run in real time).
Part II · Thematic Index
The broader, faster map — organized by theme, meant as raw material for idea generation: read across a cluster, find the gap, build there. Entries already covered above link back to their deep dive.
1 · Foundations & framing
The classics that define the vocabulary, plus the position papers that set today’s agenda.
2 · World models for embodied agents
Surveys that organize the field, and the two method families that dominate: video-generation vs. latent-prediction world models.
3 · Active perception — “moving to see better”
Agents that choose where to look or move in order to perceive better — increasingly trained with RL rather than hand-designed next-best-view rules.
4 · Agentic & multi-agent embodied AI
Agents that reason, plan, and coordinate — often LLM/VLM-driven, increasingly as multi-agent systems.
5 · Advanced MI — foundation models, spatial cognition & VLAs
The generalist layer: models that unify perception, language, and action, and increasingly reason about space and time.
6 · Evaluation & benchmarks
The most important recent shift: judging world models by closed-loop task success and state-tracking, not pixels.
7 · Quality as a task-aligned signal for embodied perception 🎯 Project-aligned cluster
The newest, most directly relevant thread: using image quality to guide or judge embodied perception — and the finding that human-perceived quality transfers poorly to robot task success.
How to use this map for idea generation
A simple recipe I follow: (1) pick a cluster above; (2) read the surveys to get the taxonomy; (3) read 2–3 methods and their evaluation; (4) look for a mismatch — e.g. metrics that don’t predict task success (World-in-World), or a capability humans have that models lack (state tracking, spatial recall), or a quality signal that ignores movement (Group D); (5) propose the smallest experiment that closes that gap. The recurring theme across the 2025–2026 literature — visual realism is not utility — is itself a fertile source of problems.
Verification note. Links point to real, recent papers; the Saining Xie / NYU and Meta entries are taken from official pages, and the four “start here” IDs were independently confirmed. Several other entries — including the Group D trio — are preprints that may later change venue, title, or version. The reframing and the idea openings are my own analysis, not claims from the papers. Confirm title/authors/venue at the source before formal citation.
References
- D. Chen, W. Chung, Y. Bang, Z. Ji, and P. Fung. WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planning. arXiv:2506.04363, Meta FAIR, 2025. https://arxiv.org/abs/2506.04363
- Meta FAIR. Action100M: A Large-scale Video Action Dataset. arXiv:2601.10592, 2026. https://arxiv.org/abs/2601.10592
- P. Fung et al. Embodied AI Agents: Modeling the World. arXiv:2506.22355, Meta, 2025. https://arxiv.org/abs/2506.22355
- J. Zhang et al. World-in-World: World Models in a Closed-Loop World. arXiv:2510.18135, ICLR 2026 (Oral). https://arxiv.org/abs/2510.18135
- S. Yang et al. MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment. CVPR Workshops, 2022. arXiv:2204.08958. https://arxiv.org/abs/2204.08958
- Reinforced Embodied Active Defense (Rein-EAD). IEEE TPAMI, 2025. arXiv:2507.18484. https://arxiv.org/abs/2507.18484
- Y. Zhai et al. Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning (RL4VLM). NeurIPS, 2024. arXiv:2405.10292. https://arxiv.org/abs/2405.10292
- P. Wu and S. Xie. V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs. CVPR, 2024. arXiv:2312.14135. https://arxiv.org/abs/2312.14135
- G. Savva, O. Michel, …, S. Xie. Solaris: Building a Multiplayer Video World Model in Minecraft. arXiv:2602.22208, NYU, 2026. https://arxiv.org/abs/2602.22208
- VSTAT: Benchmarking Visual State Tracking in Multimodal Video Understanding. arXiv:2606.03920, NYU, 2026. https://arxiv.org/abs/2606.03920
- R. Wang, Y. Bhalgat, C. Li, and V. A. Prisacariu. Active View Selector: Fast and Accurate Active View Selection with Cross-Reference Image Quality Assessment. arXiv:2506.19844, University of Oxford, 2025. https://arxiv.org/abs/2506.19844
- C. Li et al. Image Quality Assessment for Embodied AI. arXiv:2505.16815, 2025. https://arxiv.org/abs/2505.16815
- J. Zhang et al. Embodied Image Quality Assessment for Robotic Intelligence (EPD / MA-EIQA). arXiv:2412.18774, Shanghai Jiao Tong University, 2025. https://arxiv.org/abs/2412.18774