True physical agents must go beyond memorizing scene configurations and instead reason about spatial relationships, dynamics, and causality to act in novel situations. We characterize this through two forms of generalization: cross-task transfer within the same environment, and cross-environment transfer across distinct physics simulators. Conventional non-VLM agents (e.g., DQN) collapse perception and decision-making into a single opaque mapping, learning spurious visual-to-action correlations that fail to transfer. Vision-language models offer a different paradigm by replacing reactive mappings with explicit causal reasoning, yet the two dominant training paradigms — Supervised Fine-Tuning (SFT) and success-driven RL — each introduce structural limitations.

We identify that both SFT and success-driven RL fail to supervise the connection between CoT reasoning and physical reality, leading to two failure modes: hallucinated CoT that contradicts physical reality, and reasoning-action misalignment where the model takes a visual shortcut that bypasses its own stated reasoning. VAORA resolves both by requiring the reasoning trace to remain consistent with the visual observations of the environment — both the initial scene and the post-interaction outcome — within a shared symbolic space.

Visual Alignment Reward. To anchor the reasoning trace to a scene independent of the model's action, the grounding reward r_G aligns the VLM's perception of the initial scene configuration with the real initial observation. A model that cannot accurately perceive the scene before acting will inevitably produce reasoning inconsistent with physical reality, so this reward directly attacks hallucinated CoT at its source.

Visual-Action Alignment Reward. Two rewards ground the reasoning trace in the physical consequences of the action. The collision reward r_C aligns the model's predicted collision events with the post-interaction outcome, while the placement reward r_P aligns the predicted ball placement with its actual position in the first post-interaction frame — together addressing reasoning-action misalignment.

Expert-Guided (EG) Optimization. Sparse binary task-success rewards in continuous action spaces often cause policy collapse. We replace the binary success signal with a smooth, dense success probability estimated by a pre-trained DQN expert, r_DQN. The visual-action alignment rewards are gated by r_DQN so that reasoning quality is rewarded only when the action is physically plausible, while r_G is optimized outside the gate. The two components are jointly optimized with GDPO.