Abstract
World-action models couple imagined visual futures with robot actions, but RGB-only imagination leaves geometry and motion implicit, limiting robust robot control under distribution shifts. We propose S4NDBOX, a 4D world-action model that jointly predicts RGB, depth, view-specific geometric motion fields, and robot actions, explicitly connecting visual evolution to 3D robot motion. The explicit 4D future supports action generation and makes world–action agreement measurable before execution.
Across both simulated and real-world distribution shifts, S4NDBOX improves robustness over RGB-only control. On RoboTwin, after training only on clean scenes, S4NDBOX improves success in the hard (domain-randomized) setting by 13.3 percentage points over the RGB-only baseline. We further evaluate zero-shot simulation-to-real transfer on Stack Cups, training on simulated demonstrations and deploying on a physical robot without real-world task demonstrations or fine-tuning. Under additional shifts in object pose, lighting, and clutter, with camera-pose shifts additionally evaluated in the single-view setting, our 4D model improves the mean task score over RGB-only by 41.7 and 40.0 points in three-view and single-view settings, respectively.
Method
Most world-action models represent the imagined future primarily through RGB. Appearance can implicitly encode geometry and motion, but these quantities stay entangled in pixel space. A visually plausible future need not preserve the geometry implied by the predicted action, and appearance can change substantially across domains even when the underlying spatial relations and robot motion do not. S4NDBOX instead makes geometry and motion explicit.
Autoregressive 4D world–action modeling. RGB, depth, and motion streams jointly form a future 4D stage. These modalities attend to one another and to the observed history; the subsequent action stage can additionally attend to the complete future 4D prediction, while future 4D tokens cannot access future actions. All modalities share a single Transformer with modality-specific I/O projections.
Factorized 4D prediction
Given an observation history $H_k$, S4NDBOX predicts a future 4D chunk $Z_k=\{I_k^v,D_k^v,M_k^v\}_{v=1}^{V}$ — RGB, metric depth, and a view-dependent motion stream — followed by its aligned action chunk $A_k$:
$p_\theta(Z_k,A_k\mid H_k) = p_\theta(Z_k\mid H_k)\,p_\theta(A_k\mid H_k,Z_k).$
The three visual modalities first mutually constrain the same imagined future; that future then serves as the basis for action inference. Future 4D tokens are predicted independently of future actions.
View-dependent motion
The motion stream is instantiated differently per camera. Fixed cameras use endpoint optical flow: source pixels are lifted to 3D with metric depth and camera geometry, propagated between the two endpoints, and reprojected with the target camera, which accounts for camera motion without accumulating adjacent flow estimates. Moving wrist cameras instead use world-frame ray directions $r_t(u) = R_{wc,t}K^{-1}\bar u / \|K^{-1}\bar u\|_2$, which directly expose camera orientation. The RDR variant therefore combines head-camera endpoint flow with wrist rotation-ray maps.
All three streams are encoded with a shared frozen Wan 2.2 VAE into a common three-channel visual space, with multi-view latents packed into one T-shaped canvas so that cross-modal correspondence is preserved and view identity is encoded spatially. Aligned tokens share 3D RoPE coordinates; action vectors are embedded into the same token space at temporal positions between consecutive visual timestamps.
World–action agreement
A predicted 4D future and its paired action describe the same robot evolution in complementary representations, so they should agree geometrically. We exploit this redundancy by rotating the observed wrist rays according to the predicted action to obtain action-implied rays $\tilde r$, and comparing them against the predicted future rays $\hat r$. A high angular residual indicates disagreement between the imagined future and its paired action, rather than an error attributable to either prediction alone. Multiple world–action hypotheses sampled from the same state can then be ranked before execution, with the lowest-residual candidate committed.
Generalization under Scene Distribution Shifts
We train on 50 clean demonstrations per task (2,500 total over 50 tasks) and evaluate in both clean and domain-randomized scenes, following RoboTwin's hard protocol, with 50 trials per task and setting across all 50 tasks. The clean setting (Easy) uses fixed white backgrounds, lighting, and table height with no distractors. The domain-randomized setting (Hard) varies textures, lighting, and table height and introduces clutter.
| Method | 4D | Easy▼ | Hard▼ | Mean▼ | |
|---|---|---|---|---|---|
| Single-task supervised fine-tuning | |||||
| π0 Single-task SFT | 46.4 | 16.3 | 31.4 | ||
| Robotics Diffusion Transformer Single-task SFT | 34.5 | 13.7 | 24.1 | ||
| 3D Diffusion Policy Single-task SFT | 55.2 | 5.0 | 30.1 | ||
| Action Chunking Transformer Single-task SFT | 29.7 | 1.7 | 15.7 | ||
| Diffusion Policy Single-task SFT | 28.0 | 0.6 | 14.3 | ||
| Vision–language–action models | |||||
| GigaBrain-0.7* VLA | 66.8 | 67.9 | 67.3 | ||
| π0.5 VLA | 70.7 | 46.0 | 58.4 | ||
| X-VLA VLA | 68.0 | 20.9 | 44.5 | ||
| World–action models | |||||
| AHA-WAM WAM | 64.3 | 3.2 | 33.8 | ||
| FastWAM WAM | 77.8 | 1.9 | 39.9 | ||
| X-WAM* WAM | ✓ | 70.0 | 25.8 | 47.9 | |
| LingBot-VA◆ WAM | 80.7 | 34.6 | 57.7 | ||
| 4D-WAM◆ WAM | ✓ | 81.5+0.8 | 41.8+7.2 | 61.6+3.9 | |
| Ours: S4NDBOX (RGB + depth + motion)◆ WAM | ✓ | 86.8+6.1 | 47.9+13.3 | 67.3+9.6 | |
Average success rate (%). Easy and Hard evaluate clean and domain-randomized scenes after clean-only task training; 50 trials per task and setting across all 50 tasks. Green figures are gains over LingBot-VA, the shared backbone (◆). * RoboTwin 2.0 trajectories appear in the pretraining corpus; GigaBrain-0.7 additionally pretrains on 37.3K embodied hours against LingBot-VA's 16K. Bold marks the best value in a column, underline the second best.
Success rate (%), sorted by Hard. Labelled values are our own row and the best Hard score; hover any bar for exact values, or switch to the table for every number.
Our method reaches 86.8% in Easy and 47.9% in Hard. Against LingBot-VA, the RGB-only baseline it shares a backbone with, that is 6.1 and 13.3 percentage points, the larger gain appearing under randomization. FastWAM, which retains video co-training but omits future prediction at inference, drops from 77.8% to 1.9% — video co-training alone does not ensure robustness to these shifts. Among 4D world–action models we exceed 4D-WAM and X-WAM on Hard by 6.1 and 22.1 points.
GigaBrain-0.7 remains stronger on Hard at 67.9%, but its pretraining exposure is far broader — 37.3K embodied hours plus 272M vision–language samples, against the inherited LingBot-VA backbone's 16K manipulation hours — so that comparison is not budget-matched.
Single-View RDF: Geometry and Motion Consistency
We present six selected single-view cup-stacking rollouts under variations in object pose, lighting, and scene clutter. For each example, we provide interactive reconstructions from predicted and observed RGB–depth sequences, together with a flow-based tracking diagnostic on visible robot surfaces. These qualitative examples are separate from the multi-view RDR evaluation reported below.
Pose 1 — Interactive 4D reconstruction
Open full viewer ↗Each frame is independently back-projected into the robot base frame. Prediction and observation are compared at matching action indices; missing frames are not interpolated. Rotate either point cloud to inspect its geometry, or use the timeline to examine temporal changes. “4D” denotes a sequence of 3D reconstructions, not persistent point correspondences.
Flow-based surface tracking
Left: observed flow and depth. Right: predicted flow and depth. Red points indicate tracked locations; cyan crosses indicate visible surface references obtained from measured joint configurations and forward kinematics (FK). Magenta crosses denote missing tracked depth. Reinitialization occurs every eight flow updates (24 action steps). Playback is five displayed frames per second, not wall-clock speed.
Tracking protocol and interpretation
Tracking begins once reliable robot surfaces are visible. At each window boundary, surface points are initialized from the current observation and assigned fixed coordinates in their corresponding robot-link frames. Within the window, optical flow propagates the image locations and depth lifts them to 3D; action targets and FK are not used to correct the propagated points. The observed and predicted streams share the same initialization.
The reported error is the mean finite 3D distance to the visible, measured-FK surface reference, in millimetres. Initialization samples are excluded. Missing tracked depth reduces coverage rather than contributing zero error. At each boundary, the previous window endpoint is scored before the new initialization. Point sets may differ across windows; the results therefore characterize short-window consistency, not uninterrupted long-horizon tracking.
The recorded observed-flow stream uses kinematic/depth information in its construction. Its agreement with FK is a consistency diagnostic, not an independent ground-truth accuracy estimate. These selected examples do not establish a statistical relationship between flow alignment and task success.
Zero-Shot Simulation-to-Real Transfer on Stack Cups
Both the RGB-only and the geometry-augmented policy are trained on the same 1,000 scripted demonstrations collected in CoppeliaSim, then deployed on a calibrated Franka Research 3 without real-world task demonstrations or fine-tuning. We compare post-convergence checkpoints under two observation layouts — three views (two side cameras and one wrist camera) and a single left-side view — and, on top of the simulation-to-real gap, real-world shifts in cup configuration, lighting (two strong, one dim, one medium) and added clutter, four episodes per layout per shift. In the single-view layout we additionally shift the camera pose, over eight episodes.
A complete three-cup stack scores 1, and the stack has to stay standing after release. Retries within an episode are allowed, each failed attempt costs 0.5, and the episode score floors at 0. The mean pools all evaluated episodes within a layout. This is not a binary success rate.
| Method | Pose | Light | Clutter | Cam pose | Mean | Δ Mean |
|---|---|---|---|---|---|---|
| Three views: two side cameras and one wrist camera | ||||||
| RGB-only | 0.0 | 12.5 | 0.0 | – | 4.2 | baseline |
| RGB + depth + motion (ours) | 37.5 | 25.0 | 75.0 | – | 45.8 | +41.7 |
| Single view: left-side camera only | ||||||
| RGB-only | 37.5 | 0.0 | 37.5 | 34.4 | 28.8 | baseline |
| RGB + depth + flow (ours) | 87.5 | 75.0 | 50.0 | 65.6 | 68.8 | +40.0 |
Graded task score, 0–100, under object pose, lighting, clutter and camera-pose shifts; camera pose is evaluated in the single-view layout only (– = not evaluated). Mean pools all evaluated episodes within each view layout. Bars are scaled to the full range; bold marks the geometry-augmented policy against RGB-only within the same layout — the two layouts are separate comparisons and should not be ranked against each other.
The geometry-augmented policy beats RGB-only in every shift in both layouts, by 41.7 points with three views and 40.0 with the left view alone. The widest gaps are lighting in the single-view layout, where RGB-only scores 0.0 against 75.0, and clutter with three views, 0.0 against 75.0. Under camera-pose shift in the single-view layout, RGB-only scores 34.4 against 65.6.
Simulation training data
The policy is trained on simulated demonstrations. The clip below is one such demonstration — a three-cup stack, with its three views in step. Camera layout differs between simulation and the real lab, so this is not the real-robot configuration.
Synchronised LEFT / RIGHT / WRIST views. Real time (1×), view names as burned into the clip.
RGB-only vs. ours under the same condition 10× SPEED
Three conditions where the RGB-only policy fails to complete the stack, each beside a successful rollout of the geometry-augmented policy under the same condition. Played at 10×. These use a three-cup Stack Cups setup, separate from the graded evaluation above, so they illustrate failure modes rather than contribute to any score. Two further RGB-only failures were left out because their matching success recording could not be identified.
Fails. The arm engages the cups but no stack is formed.
Succeeds. The same cluttered workspace, ending in a completed three-cup stack.
Fails. The arm hovers over the cups without completing a grasp-and-place.
Succeeds. Same illumination, stack completed.
Fails. The gripper descends between two cups and the episode ends with no stack.
Succeeds. Same illumination, stack completed.
Selected clips, not a success-rate summary. Conditions were matched from the recordings themselves — clutter by the extra objects on the table, and the two lighting conditions by illumination — not from a lab log.
Robot demonstration
Stack Cups under moving coloured lighting · 10× speed
BibTeX
@misc{s4ndbox2027,
title = {Verifiable 4D World-Action Model for Robust Robot Control},
author = {Anonymous Authors},
year = {2027},
note = {Working draft under review}
}