PointGoal
Point-goal navigation
Navigate toward a goal specified in the robot frame, through floor clutter.
Navigation foundation model
See What I Need, Dream What I Want
A Navigation Foundation Model on a Self‑Constructed Representation Space
∗Equal contribution†Project leadCorresponding authors
1Peking University2University College London, University of London3DataCanvas
00
Abstract
What visual representation should a navigation policy act on? A central idea of representation learning is that the task’s own error should shape the representation. Navigation policies built on vision foundation models largely set this idea aside: the encoders’ pretraining objectives decide much of what the vision-language backbone sees. We present RepresentNav, a navigation foundation model built on a self-constructed representation space: a visual representation constructed by the policy’s own objectives out of the internal structure of vision foundation models. RepresentNav builds this space with Representation Residuals, a compact three-layer transformer that attends over features spanning the full depth of multiple complementary vision foundation models, so that the policy learns to see what it needs for navigation. In the same space it dreams what it wants: before every action a latent chain-of-thought predicts the representation of the scene a moment ahead. The dream is trained end-to-end with the policy as a joint-embedding predictive architecture (JEPA), with a next-representation prediction loss and a regularizer on its targets.
RepresentNav sets a new state of the art on VLNVerse, VLN-CE R2R, EVT-Bench, and two point-goal benchmarks, and its gains carry over to a physical robot amid clutter and moving people. On the VLNVerse fine-grained benchmark, RepresentNav achieves a success rate of 83.80%, outperforming the previous state of the art by 20.05 percentage points, and on EVT-Bench distracted tracking it reaches 85.5% with a single camera (+10.1). Analysis shows where they come from. With the same encoders, the self-constructed space clearly outperforms concatenating their outputs and learns to weight their layers unevenly, and a dream in it cuts collisions by a quarter. A single dreamed latent decodes into the depth, semantics, and 3D occupancy of the scene a moment ahead, more accurately than pretrained encoders applied to the real future frame, although occupancy is never supervised, and the rollout tracks the robot’s own motion over the next second: the dream emerges as a compact latent world model.
Method overview
Learn what to see. Predict what comes next.
One representation space carries perception, foresight and action.
a
Representation Residuals
Three trainable transformer layers attend over layers spanning the full depth of three frozen vision foundation models. A learnable [CLS] summarizes each view.
See what I need → bLatent chain-of-thought
Before acting, the language model rolls out four continuous latents, each predicting the future [CLS] tokens; VISReg keeps the targets from collapsing.
Dream what I want → cFrom foresight to action
A planning head maps the latent-end state to eight future waypoints, turning latent reasoning into a navigation trajectory.
Watch it navigate →01Representation Residuals
See what I need.
Attention residuals read every depth of three frozen foundation models; the policy’s own objective decides what its representation makes explicit.
- ❄Frozen
- Trainable
- αAttention residual
- [CLS]
- Representation token
01
Observe
The 3D illustration needs WebGL. The explorer below still shows every readout.
3D occupancy120 × 80 × 20 voxels
Current representation · t = 0Constructing the scene…
↔ Drag to rotate · Scroll to zoom
Height02 m
Interactive illustration reconstructed to explain the method: real RGB inputs, illustrative readouts, not checkpoint inference.
02Latent chain-of-thought
Dream emerges as a
latent navigation world model.
Before every action, the policy dreams in the space it sees through: four continuous latents predict the representation of the scene a moment ahead.
- Text token
- Representation token
- latent_start / end
- Latent state h
- [CLS] target
01
Observe & instruct
The 3D illustration needs WebGL. The explorer below still shows every decoded state.
Open a dreamed state
3D occupancy120 × 80 × 20 voxels
h₁ · t + 0.4 sConstructing the scene…
↔ Drag to rotate · Scroll to zoom
Height02 m
Decoded latents are post-hoc visualizations, reconstructed to explain the method; they are not part of the control loop.
03From representation to action
Real-world demonstrations
Navigation through clutter, following language instructions, and tracking a moving target in real environments.
Cite
BibTeX
@misc{chen_representnav,
title = {See What I Need, Dream What I Want:
A Navigation Foundation Model on a
Self-Constructed Representation Space},
author = {Chen, Jiawei and Wang, Yinxi and Yang, Zihan and
Gao, Meijun and Liu, Yaowei and Zhang, Shuming and
Li, Zecheng and Mi, Xiangju and Liu, Yuxuan and
Huang, Runzhong and Li, Tianlin and Zhang, Xun and
Wang, Zhengyang and Miao, Xu and Yang, Tong},
year = {2026},
eprint = {2610.31762},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2610.31762}
}