Embodied systems need geometric perception that exploits available observations beyond images alone. Recent feed-forward 3D models incorporate geometric priors, including camera poses, intrinsics, and depth. However, handling noisy poses, preserving accurate priors, and recovering physical scale require more than simply accepting these inputs. We introduce the Vision-Prior Geometry Grounded Transformer (VPGGT), a VGGT-based framework that extends OmniVGGT for prior-aware embodied perception.
We formulate sensor-motivated pose corruptions from ground-truth trajectories for training and introduce a parameter-free Prior Residual Connection (PRC) to mitigate prior dilution, where predictions are less accurate than their supplied pose priors. Our noise formulation targets camera poses; supplied intrinsics and depth receive no additional corruption. We further introduce Metric Global Attention, which conditions a global scale token on available pose and depth scales and predicts a shared metric scaling factor for the geometric outputs.
Experiments across four datasets show that PRC improves translation-direction accuracy and joint pose AUC over a matched training baseline when camera priors are provided for all views, under both exact and corrupted poses. These results support explicit prior access during refinement as a useful addition to feature-level conditioning.
We use prior to denote a supplied geometric estimate. For a higher-is-better pose metric $M$, let $M_{\mathrm{prior}}$ and $M_{\mathrm{out}}$ be the scores of the input prior and the model prediction on the same eligible image pairs. We define their gap as $d_M = M_{\mathrm{prior}} - M_{\mathrm{out}}$ and refer to $d_M > 0$ as prior dilution — the model produces camera estimates less accurate than the priors it was given, even when those priors are exact.
When the prior outperforms the matched no-camera-prior reference $M_0$, closure of the baseline-to-prior gap is $\kappa_M = \tfrac{M_{\mathrm{out}} - M_0}{M_{\mathrm{prior}} - M_0}$, with dilution $\mathcal{D}_M = 1 - \kappa_M$.
This motivates training the model with PRC so that the refinement operator learns corrections that respect, rather than dilute, informative priors.
Given images $\mathcal{I} = \{I_i\}_{i=1}^{N}$ and optional camera observations $\mathcal{C}$ and masked depth observations $\mathcal{D}$, VPGGT predicts a shared positive scale $\hat{s}$ together with per-view camera parameters, depth maps, and pointmaps. It builds on an OmniVGGT backbone (24 alternating frame/global attention pairs, token dimension 1024) and retains the inherited camera, depth, and pointmap heads.
The camera head already refines predictions through residual updates. PRC changes the reference for these updates from an estimated camera state to the supplied prior wherever one is available, following the residual-learning principle of learning corrections to a reference.
After recording the initial prediction $\mathbf{r}_i^0$, PRC resets the state to the scale-aligned prior, $\mathbf{u}_i^0 = \mathrm{sg}[\,m_i \mathbf{a}_i + (1-m_i)\mathbf{r}_i^0\,]$, where $m_i$ indicates prior availability and $\mathrm{sg}$ denotes stop-gradient. Remaining iterations apply the original refinement operator. PRC adds no parameters and leaves the computation unchanged when camera priors are absent.
MGA retains the physical-scale information removed when normalizing geometric priors. A shared encoder maps the log scale and coverage of each available modality (camera, depth) to a feature, using learned modality-specific placeholders when a scale is missing. The resulting scale token is appended to the multi-view sequence for global attention, bypasses frame attention, and finally predicts a positive factor $\hat{s} = \exp\!\big(h(\mathrm{LN}(\mathbf{t}_s^L))\big)$ that scales depths, translations, and pointmaps without changing their heads.
We generate corrupted camera trajectories in metric units before view sampling and normalization. Drift profiles combine sequence-level biases with random motion increments, while the C3VD profile perturbs each pose independently. Each training item uses either the clean trajectory or one of four fixed corrupted versions, consistently across its views.
| Training data | Translation | Rotation |
|---|---|---|
| TartanAir | Time-dependent | Time-dependent |
| TUM RGB-D | Time-dependent | Time-dependent |
| Virtual KITTI 2 | Step-dependent | Time-dependent |
| C3VD | Independent | Independent |
Pose-corruption profiles. Drift is applied to relative-motion increments.
Under matched training from the same released OmniVGGT checkpoint, PRC improves camera-prior utilization, particularly at full coverage. With exact camera priors and no depth input, PRC raises RTA@5° by +15.00 points on TUM RGB-D and +33.58 points on C3VD, with AUC@30° gains of +9.42 and +19.06 points respectively.
| Method | TUM RGB-D | C3VD | ||||
|---|---|---|---|---|---|---|
| RRA@5↑ | RTA@5↑ | AUC@30↑ | RRA@5↑ | RTA@5↑ | AUC@30↑ | |
| RGB only | ||||||
| OmniVGGT | 99.67 | 38.00 | 61.94 | 87.51 | 14.67 | 40.88 |
| VPGGT | 96.22 | 50.44 | 69.38 | 100.00 | 27.41 | 62.58 |
| w/ exact camera prior ($C_e$) | ||||||
| OmniVGGT | 100.00 | 70.78 | 78.95 | 98.62 | 45.23 | 71.44 |
| VPGGT | 96.89 | 72.56 | 80.39 | 100.00 | 77.48 | 88.14 |
| w/ noisy camera prior ($C_n$) | ||||||
| OmniVGGT | 100.00 | 58.89 | 72.21 | 98.67 | 36.35 | 61.05 |
| VPGGT | 97.11 | 67.44 | 76.50 | 100.00 | 56.05 | 72.95 |
$R$, $T$, $A$ denote RRA@5°, RTA@5°, and AUC@30° (percent). Bold marks the unique best network within each input setting; ties at displayed precision are unmarked. Only poses are corrupted; intrinsics and supplied depth remain ground truth.
Scope. Gains on TUM/C3VD do not establish uniform cross-domain improvements; on TartanAir-V2 subsets OmniVGGT remains stronger in most settings. Prior availability alone does not guarantee prior preservation, particularly at partial coverage. See the paper for complete matched-training tables, stricter thresholds, directed metrics, metric-scale evaluation, and limitations.
