PredActor

Getting a humanoid to walk on command, then change direction with a gamepad, is not only a matter of making the model "understand instructions". For a 50 Hz controller, every control cycle has just 20 milliseconds to receive body feedback, update predictions, and produce an action. Before denoising has finished, the body may already have drifted away from the previous prediction.

Anyone who has taken a policy from simulation to a real robot meets another gap: simulation can hand over the position and velocity of every body part directly, while onboard sensors more readily provide proprioceptive information such as joint states and inertial measurements. If guiding future motion requires the policy to read the full whole-body state, state estimation becomes an extra problem to solve; if the model only outputs actions, requirements such as "move toward the target at a given speed" are hard to express directly.

What we set out to resolve is how these two constraints can hold at the same time: generate actions directly from feedback a real robot can obtain, while keeping an interface that text and motion goals can steer. To that end, we (RoboParty Lab) keep future states inside the model in PredActor, predicting them together with actions; the policy is updated from proprioceptive history, text selects behaviour, goals on predicted states guide motion, and deployment no longer chains a separate motion-reference tracking policy.

We have deployed this system on a G1's Jetson Orin NX, running onboard at 50 Hz. The full control callback has a p95 latency of 19.383 ms; goal reaching in simulation reaches 44/45, and on the real robot we show text-driven behaviour switching and action responses under external disturbances. These kinds of evidence answer the compute budget, the guidance capability, and the physical execution questions respectively, and cannot substitute for one another. Below we first look at how behaviour is executed, then unpack why this design can serve both feedback and guidance.

The experiment was completed relatively early and has not yet been deployed or tested on RoboParty RP1; an RP1 version will follow in later iterations.

01 After the instruction switches, the body motion has to keep up

For a user, "walk", "squat down", and "walk again" are just a few instructions; for the controller, the starting point of every switch is the robot's body state at that moment, not the first frame of a preset animation.

The G1 real-robot experiments include standing, walking, jogging, and squatting, as well as transitions between walking, squatting, and walking again. The policy continuously receives proprioceptive history and generates subsequent actions from the current task condition. The same policy must both change behaviour and keep the post-transition motion physically executable.

To explain these switches, two things must first be separated: selecting a motion is not the same as already having the action the robot should execute at this instant. In PredActor we try to put task requirements and body feedback into the same policy, but once actions are output directly, where should the goals that used to be expressed on motion trajectories go?

02 After removing the tracking stage, how is motion controllability preserved?

The key is the interface between the generated result and the robot. Diffusion Policy applies diffusion to action learning: it produces a segment of actions from observations, executes part of it, and updates the subsequent plan with new observations. This receding horizon control (RHC) feeds feedback back into the generation process. For humanoid whole-body control, one still has to decide: should the model generate a motion reference for people to watch, or actions that can be handed to the robot to execute?

Choosing a motion reference requires another controller to track it. The generator handles motion expression and the tracker handles body execution; the division of labour is clear, but it requires the reference produced by the former to fall within the capability of the latter. A motion can look reasonable yet be hard for the robot to execute from its current body state; once disturbed, recovery work still falls mainly to the tracking stage.

Generating actions directly can shorten this execution path. But action-only diffusion, a diffusion policy that predicts only actions, has no explicit future state trajectory. For requirements such as "walk toward the goal point" or "maintain a certain motion direction", users can describe the desired motion but can hardly write down the corresponding joint actions directly. This is precisely why internal state prediction is introduced.

Jointly predicting states and actions offers an interface: goals can be written on future states rather than directly as joint actions. But existing representative methods rely on privileged inputs such as whole-body states, which require extra estimation at deployment; the learned behaviour conditions and the motion goals specified at test time also need to enter the generation process separately. So merely changing the action output into a joint state-and-action output does not yet solve all the problems of real deployment.

Our design choice is: take proprioceptive history as the external input, use future states as the internal guidance interface, and finally output actions directly. Joint state–action modelling is not an independent first for this work; what is worth attention is how this representation combines with real-robot observations, the two kinds of guidance, and onboard feedback execution. It reduces deployment's dependence on external full-state inputs and a separate motion tracking module, at the cost that the policy itself must learn state prediction and action generation well and meet the online compute budget.

So "direct" means removing the separate motion-reference tracking policy, not cancelling low-level joint control. Another distinction to draw is between supervision and input: full whole-body states can still be used for training supervision, while deployment does not require these states to be supplied externally to the policy.

03 Text selects behaviour, predicted trajectories receive motion goals

Classifier-free guidance (CFG); classifier guidance (CG).

One inference pass can be understood as two connected processes: first, denoise step by step from proprioceptive history and optional task conditions to obtain future states and actions; then take the action from them and send it to the joint controller. Predicted states are not issued as a motion reference but stay inside the model for the objective function to evaluate and guide.

Structurally, we represent states and actions with interleaved tokens, preserving the asymmetric attention relationship of Diffuse-CLoC. The encoded observation history and task conditions enter the model through cross-attention, that is, the sequence being denoised reads this conditioning information. This connection both preserves the role of states in predicting subsequent actions and lets task requirements participate in generation, rather than only filtering results after generation.

Next we must distinguish "which action to perform" from "what goal the motion must satisfy". They correspond to CFG and CG respectively; the two act differently and can be used together.

Using CFG to strengthen learned behaviour conditions

Classifier-free guidance (CFG) combines predictions "with task conditions" and "without task conditions" to strengthen the generated result's response to the condition. Put plainly, it makes the model execute the currently selected behaviour more distinctly. In PredActor, text encoding is used to select the motion semantics covered by training, such as standing, walking, jogging, and squatting; it is not equivalent to understanding and executing arbitrary natural-language tasks.

Using CG to adjust the direction and speed of future motion

If the requirement is to approach a target position, or to bring speed close to a desired value, the deviation can be computed on predicted states and the gradient used to give an adjustment direction. This is the classifier guidance (CG) here: test-time motion requirements are written as differentiable objectives that guide the predicted trajectory during denoising.

No extra image classifier is attached here. CG adjusts predicted states within the current gradient step and does not directly rewrite the current action prediction; the joint denoising then lets the updated state information influence action generation. In this way, "future speed closer to the desired value" can become an optimisation objective without the user translating the requirement into joint actions. CFG handles learned behaviour conditions, CG handles test-time computable motion goals, and the two complement each other within the same generation process.

In the joint text-and-gamepad demonstration this division is especially intuitive: text selects the learned motion behaviour, while the gamepad continuously adjusts motion requirements during execution.

The same internal state interface can also take a goal-point cost, adjusting the predicted trajectory and robot motion as the goal changes.

The outdoor G1 video shows steering guidance and behaviour switching on the real robot. This demonstration enables the steering input wz, with the planar velocity inputs vx and vy disabled. It therefore provides qualitative evidence for outdoor steering control and cannot be equated directly with real-robot validation of all speed and goal-point settings in simulation.

Behaviour conditions need not be discrete switches only.

The semantic interpolation experiment gradually changes the mixing weight of two conditions, presenting intermediate behaviours between learned conditions such as standing and raising a hand, or standing and running.

04 Reactions after disturbance also depend on the current behaviour

The demonstrations above mainly change task inputs. External interaction changes the body itself: after pushing, moving an arm, or blocking the path, the policy no longer faces the previously predicted state. What needs observing then is whether the new feedback actually changes the following behaviour.

The real-robot interaction video first shows standing, then walking. Under the standing instruction, external interaction is accompanied by arm extension, changes in stance width, body lowering, or lateral stepping; under the walking instruction, the robot adjusts its steps when the path is blocked. For practitioners, the observation point here is whether the response relates to the current behaviour, not merely whether the trajectory deviated. The video presents behaviour adjustments in these specific situations; recovery success rates still require repeated trials.

External-force experiments in the real robot and in simulation are used to observe response patterns, while quantitative statistics are presented in the experiments section.

These clips show how feedback updates appear at the behaviour level, but why can it cope with states that deviate from the original trajectory? The answer lies not only in the network structure but also in whether training covers such deviations.

05 Making training cover states after the policy deviates from demonstrations

Removing the separate tracker at deployment does not mean its physical execution experience is no longer needed during training. Collecting only successful demonstrations still leaves a problem: once the student policy makes an error or receives a push, it may enter body states that demonstrations rarely cover; continuing to learn from the same batch of ideal trajectories may not fill these gaps.

The initial data first answers "where behaviour comes from". The motion library contains generated motions and AMASS (Archive of Motion Capture as Surface Shapes) motion-capture data; BABEL (Bodies, Action and Behavior with English Labels) labels and the MotionCLIP motion–text alignment model provide semantic associations. A frozen tracking teacher then executes the reference motions and collects states and actions. Observation noise, action noise, and pushes are added during collection so that the data is not limited to ideal execution.

With an initial student policy in place, it is then run on its own. The teacher starts from the states the student actually reaches and generates demonstration segments aligned with the reference motion; these new data are used for the next round of student training. This DAgger-like (Dataset Aggregation) iteration shifts the sampling focus onto the states the student itself will encounter.

What changes here is the way data coverage is built: not only adding demonstrations of "what the teacher would do", but also supplementing the states of "what the student encounters after going astray". The model jointly learns action reconstruction, an action-variation regulariser, and future state reconstruction; the teacher provides supervision during training, and at deployment the student policy generates actions directly. Engineering-wise, it moves part of the tracking capability into the training pipeline rather than eliminating the cost of building the teacher, collecting data, and iterating training.

06 Not generating the whole future from scratch is what makes new feedback fast to incorporate

Training settles what the policy has seen; deployment must also settle how long each update takes to compute. Diffusion generation usually involves multiple denoising rounds; if the whole future is generated from scratch after each new observation, the compute cost and the feedback update rate come into conflict.

This G1 configuration controls 29 actuated joints and maintains a 20-step state–action prediction. This prediction length is not the control period: the policy updates 50 times per second, and each 20 ms budget must accommodate the complete callback work of network computation, guidance, and sampling updates.

This reuse must match the noise schedule; it is not copying the previous clean trajectory over directly. Onboard operation further adopts two-step DDIM (Denoising Diffusion Implicit Models, a diffusion sampling method), caches reusable conditions, precomputes the sampling schedule, and optimises the C++ runtime. Latency compensation interpolates between adjacent clean actions to reduce the effect of the mismatch between inference time and the actual execution instant. The trade-off here is not simply "fewer steps is better", but jointly considering sampling quality, disturbed performance, and the cost of the complete control callback.

The source is paper v3 Section 4.2 and Appendix D.3. The timing covers the complete callback from receiving the state message to producing the host-side action tensor, not only the neural network forward pass; but this timing experiment does not drive the actuators and does not include the time for the actuators to complete their physical response. 593/600 falling within budget shows onboard operation is feasible and does not constitute a hard real-time guarantee. The lesson for deployers is that network speed, callback tail latency, and real-robot closed-loop performance need to be checked separately.

07 The experiments test the combination of steerability and stability

We conduct simulation evaluation in a unified MuJoCo (Multi-Joint dynamics with Contact) environment, testing goal reaching, text matching, and disturbance response separately. Of the seven configurations, six learned policies each evaluate three independently trained models; the combination of the ARDY motion generator and the SONIC tracker uses a single-model reference, so it cannot be treated as the same repetition setup as the three-model statistics. ARDY stands for Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation, and SONIC for Supersizing Motion Tracking for Natural Humanoid Whole-Body Control.

The goal-guidance and text-retrieval results come from paper v3 Section 4.1 and Table 2. Text retrieval measures how well the executed motion matches the target text; the valid windows for PredActor and DiffuseLoco w TextCFG are 119/135 and 126/135 respectively, and should be read together with the scores. This comparison supports PredActor obtaining a higher text-matching score under this evaluation, but the differences of the whole system cannot all be attributed to the single design of joint state prediction.

In the same simulation disturbance evaluation, PredActor's survival-duration ratio is 0.511 and DiffuseLoco w TextCFG's is 0.563; it is not "the proportion of trials fully passed". The positive meaning of this group of results is that PredActor, while gaining motion guidance capability and a higher text-retrieval score, still retains disturbance resistance close to the conditional action diffusion baseline. "Close" describes the observed numerical range and does not imply improved disturbance resistance or statistical equivalence.

The G1 real-robot trials include six settings, five trials each; falls, incomplete motions, and safety stops all count as failures. The repeated trials supplement failure cases that demonstrations cannot show: the policy can already complete the tested transitions and disturbance responses, but not successfully every time. The small-scale repeated trials support capability under these specific settings and cannot be extrapolated to reliability under arbitrary contacts or environments.

08 What is still missing to move from existing demonstrations to more complex tasks

Putting these results together, PredActor's value is not merely chaining one fewer module. It shows that such a combination can enter an onboard closed loop: body feedback comes from proprioception, behaviour selection comes from conditional guidance, motion requirements act on internal predicted states, and finally the same policy produces executable actions.

This combination still has boundaries. Which behaviours text can select depends on the coverage of motion and semantic data; when goal guidance is too strong, generation may drift outside the training distribution; and sustained multi-contact motions such as crawling and climbing have not yet been sufficiently validated. This means that widening the instruction range, strengthening guidance, and adding complex contacts cannot be solved by tuning the same parameter alone.

For policy researchers, this work offers a direction worth continuing to test: when outputting actions directly, internal predicted states can still serve as the motion guidance interface, without requiring the deployment side to first provide full privileged states. For deployment engineers, it gives an implementation path that includes data aggregation, receding-horizon sampling, latency compensation, and onboard timing, not just a network structure. What still needs testing next: how to preserve physical executability under stronger guidance, what data more contact behaviours require, and how to further reduce tail latency. These questions determine whether this route can extend from existing demonstrations to more complex tasks.