Physical AI often assembles perception, forecasting and control as a chain of separately trained systems. Each interface can lose information or introduce inconsistency. A video may suggest that an object moved left, a depth estimate may place it elsewhere, and an action model may still reach toward its old location. Good component scores do not guarantee a coherent account of the event.
Thinker-WM brings appearance, metric geometry and action into a common dynamics representation. It preserves distinct encoders and decoders because the outputs still have different structures. The proposal is therefore more specific than putting several modalities into one model: it asks whether they can evolve through the same underlying state while retaining usable interfaces.
One dynamics core supports several requests
Thinker-WM, introduced October 13, maps appearance, geometry and action into a shared dynamics representation while retaining modality-specific encoders and decoders. The authors train on human video, robot data and simulation, with depth supervision available for part of the robot corpus. Read the research paper.
A Unified Dynamics Transformer jointly evolves the shared representation, with modality-specific processing around it. The same parameterisation supports world prediction, policy inference, action-conditioned generation, video-to-action generation, inverse dynamics and joint video-action generation. This creates several ways to interrogate overlapping learned dynamics.
The training corpus combines approximately 20,000 hours of egocentric video, 10,000 hours of real-robot data and 1,000 hours of simulation, according to the paper. About 4,000 hours of the robot data include aligned depth supervision. Not every sample therefore provides every modality, an important detail when interpreting multimodal transfer.
Missing modalities make the supervision question important
The training mixture includes human video, robot trajectories and simulation, but only part of the real-robot corpus has aligned depth. This heterogeneity is central to the unification claim. The model is not simply trained on one perfectly aligned three-channel dataset where every frame comes with a matching physical action and metric map.
Different samples can therefore teach different aspects of the shared representation. Human video supplies appearance and motion patterns; robot data connects observations to executable commands; aligned depth anchors geometry where available. These are descriptions of the roles implied by the data design, not a claim that every benefit has been isolated quantitatively. A useful comparison should ask which transfer survives when one source is reduced or removed. Otherwise the scale of the combined corpus can be mistaken for evidence that all modalities contribute equally or that sharing alone explains the benchmark gains.
Recoverable motion is one part of the evidence
The authors report leading results on EWMBench, WorldOlympiad and WorldArena, plus controlled studies of action conditioning, geometric interaction and multi-function training. A lightweight probe also recovers motion information from frozen visual representations.
The probe establishes availability of a signal to a learned readout. It does not by itself establish that every action decoder depends on that signal in the intended way. Stronger causal checks would remove or alter the information and inspect downstream changes. This distinction prevents a promising representation result from becoming an unsupported account of how all model decisions are made.
Cross-modal agreement can become a diagnostic
A coupled model offers an opportunity to test whether its outputs constrain one another. Alter the proposed action while holding the scene fixed, then inspect whether depth, appearance and task state change consistently. Alternatively, alter an object’s location and ask whether both the visual forecast and executable command respond.
These proposed interventions make the larger value of unification measurable. Shared parameters can save duplication, but the stronger benefit would be mutually consistent physical predictions. The corresponding risk is correlated error: one mistaken assumption can propagate into several convincing outputs. Evaluating the joint event matters more than treating every output as a separate demonstration.
Questions, news tips or corrections? Contact [email protected].
