REALM — Region-to-Entity Alignment for Last-3-Meter Navigation
Stopping within 3 m of a waypoint is not the same as seeing the object. We measure this Last-3-Meter Grounding Gap and close part of it with REALM, a plug-and-play refinement module that works on top of any VLN agent.
Vision-and-Language Navigation (VLN) agents may satisfy conventional success criteria while still failing to establish reliable object-level grounding, because current evaluation protocols mainly reward stopping within a 3-meter radius and largely ignore the agent’s final orientation and target visibility. We formalize this limitation as the Last-3-Meter Grounding Gap and introduce three instance-centric metrics to quantify proximity precision, target visibility, and final-view grounding. To mitigate this gap, we propose REALM (Region-to-Entity Alignment for Last-3-Meter Navigation), a plug-and-play, architecture-agnostic refinement module that decouples fine-grained target approaching from long-horizon navigation. REALM uses a visibility-aware stopping strategy to reduce premature termination and improve final viewpoint alignment. We further construct REVERIE-AIM, which provides object-instance-level goals and 180K short-horizon training samples for final-stage target approaching. Evaluations across four diverse VLN backbones show that REALM consistently improves proximity precision and visual grounding success, suggesting its potential applicability.
On REVERIE and REVERIE-CE, an episode counts as a success if the agent stops within 3 m of a pre-annotated waypoint. That waypoint sits near the object, not at it, and the agent’s final heading is never checked. Drag the agent in Figure 1 to see where the two standards disagree.
The agent stops within τ (0.1 m or 0.5 m, geodesic) of navigable points around the specific target instance.
The target is visible in the final camera frame and covers at least 1% of the image.
The detector’s box at the final view matches the ground truth (IoU ≥ 0.5), separating navigation errors from detection errors.
Long-horizon navigation and short-horizon target approach are different problems. REALM is a decoupled refinement stage that runs only after the upstream policy issues stop, using egocentric RGB and the original instruction, with no depth, maps or changes to the upstream model.
A graph-based, video-based or zero-shot VLN policy follows the instruction and stops near the target region.
UniNaVid-7B fine-tuned with LoRA predicts the next four low-level actions from the RGB video and instruction, repositioning the robot until the target is close and in view.
BERT-QA extracts a short target phrase (“flowers on the table”) that queries OWLv2 for the final bounding box.
Trained with imitation learning alone, the refinement policy tends to stop too early, often before the target has even come into view. Once it stops, the episode is over and there is no chance to recover.
VSP adds a training penalty that targets exactly this mistake. Whenever the model is inclined to stop while the target is still not visible, it is penalized, and the more confidently it wants to stop, the larger the penalty. When the target is visible, stopping is not penalized at all. The agent therefore keeps searching and adjusting its viewpoint until it can actually see the object, and is still free to stop as soon as it does.
REVERIE remapped into the Habitat continuous simulator, with region-level waypoints replaced by object-instance goals, plus a dense short-horizon split for learning the final approach. The first embodied navigation dataset to combine continuous space, language goals, instance-level targets and object-proximity endpoints.
REVERIE-AIM val-unseen. Pick a backbone to compare the upstream policy’s own stopping point with the same episodes after REALM refinement.
20 episodes on a Hello Robot Stretch 3 (RealSense D435if RGB-D, inference on an RTX 4090) across a robot lab, a school lobby and a foyer. UniNaVid alone vs. UniNaVid + REALM.
@inproceedings{shi2026realm,
title = {From Region Arrival to Instance-Level Grounding in Vision-and-Language Navigation},
author = {Shi, Xiangyu and Yang, Ruoxi and Tao, Wei and Zhang, Jiwen and Qiao, Yanyuan and Wu, Qi},
booktitle = {Conference on Robot Learning (CoRL)},
year = {2026}
}