P-BFM

P-BFM (Perceptive Behavioral Foundation Models), the latest research from RoboParty Lab members, is now released, exploring how the same humanoid behaviour foundation model can flexibly execute different tasks on complex terrain.

P-BFM models "what to do" and "how to execute it on the current terrain" separately: through forward–backward (FB) representations, motion references, target body states, and reward functions are connected to a shared intent space, and a perceptive policy then generates motion together with the environment geometry. The model uses single-stage joint training, with only 40 flat-ground LaFAN motion sequences as the motion prior; without modifying reference motions or preparing terrain-specific demonstrations, it learns to adjust step height, posture, foothold position, and contact mode through online interaction with diverse simulated terrain.

P-BFM's value lies not only in letting the robot adapt to terrain, but in connecting multiple behaviour interfaces with terrain-aware execution. With explicit root instructions (ERI), the policy can control the target heading and planar velocity while tracking a motion; once pretraining is complete, the same frozen policy can support teleoperation, motion tracking, target-state reaching, and reward-function-based behaviour inference without task-specific retraining, and has terrain-aware disturbance recovery. The policy has been transferred zero-shot to a Unitree G1 equipped with an onboard depth camera.

01 From motion reuse to terrain adaptation

Flat-ground motions and complex terrain have direct kinematic and contact-constraint conflicts. Stairs require changing foot-lift height, obstacles restrict the leg-swing path, and slopes change body posture and support conditions; sitting, lying down, and standing up also involve switching contact between the hands, legs, torso, and environment. Extending the motion library alone is therefore not enough to solve terrain adaptation; the controller must also re-coordinate whole-body motion according to the environment.

One class of methods first generates terrain-adapted references through offline optimisation, then trains a perceptive tracking policy. As behaviour categories, terrain parameters, and initial states increase, the behaviour–terrain combinations that must be covered expand rapidly, and complex contact also makes reference optimisation harder. P-BFM brings the adaptation process into online policy learning, using flat-ground demonstrations as the motion prior and learning how to execute on different terrains through environment interaction.

The fixed walking-intent experiment shows different motion realisations corresponding to a fixed walking intent: the robot adjusts step height and footholds on stairs, changes posture on slopes, and chooses feasible support positions among discrete obstacles. The intent stays unchanged while joint trajectories and contact modes vary with the terrain.

In the real-robot teleoperation experiment, the operator moves on flat ground while the robot executes the corresponding behaviour on stairs. Real-time retargeting provides body motion goals, and the robot adjusts its motion according to its own depth observations; the ground-contact and standing-up clips further show multi-part support and contact switching.

02 How the FB representation unifies different tasks

From successor state visitation to reward inference

FB learns the long-term visitation relationship of the policy to future body states after taking actions, and approximates the discounted successor state visitation distribution with the inner product of a forward representation F and a backward representation B. B represents target body states, while F combines the current state, action, and behaviour intent to describe how the policy will visit these states in the future.

This decomposition lets downstream reward functions be converted directly into the policy's behaviour intent. Let x be a body feature and r(x) a reward function; encoding body states sampled from experience and weighting them by reward gives the intent direction:

zᵣ ∝ Σᵢ r(xᵢ) B(xᵢ).

The corresponding action value can be approximated as:

Qᵣ(s, a) ≈ F(s, a, zᵣ)ᵀ zᵣ.

The reward defines the body states one hopes to achieve, and the FB representation links these preferences with the policy's long-term behaviour. After pretraining, a new reward function is connected to the same policy by recomputing the intent vector, without re-optimising model parameters for each reward task. At inference, the intent vector is normalised to the scale used in training.

F participates in value estimation during training; real-robot execution is produced directly by the perceptive policy as 29 joint targets, which are then handed to the low-level PD controller.

Intent encoding and perceptive execution are modelled separately

P-BFM's backward representation B receives only body features, not depth, terrain geometry, or root instructions. Given the same target body state, the intent encoding stays consistent. The policy, in turn, receives the intent, proprioceptive state, and depth observations together and generates actions according to the actual environment.

F and the auxiliary value network in training also use terrain information to evaluate the long-term effect of actions in the current environment. B is responsible for expressing behaviour goals, and the perceptive policy is responsible for adapting to the execution environment. This division of inputs lets task representations be reused across terrain while leaving room for adjusting step height, footholds, and whole-body contact.

03 How the flat-ground motion prior and terrain interaction are trained together

Single-stage joint training

In the same pretraining stage, P-BFM jointly learns the behaviour representation, the perceptive policy, and the velocity estimator. The 40 flat-ground LaFAN sequences provide the body states that construct behaviour prompts and also serve as positive samples for the conditional discriminator D(x,z). The discriminator evaluates body motion together with the current intent, associating its reward with the behaviour one hopes to execute.

The motion prior influences the policy through the value objective rather than requiring the executed trajectory to match the demonstration frame by frame. The policy samples online across seven categories of simulated terrain and updates its control behaviour from physical feedback such as collisions, support changes, foot slipping, and blocked progress. Demonstrations thus constrain behaviour features, while terrain interaction determines how these behaviours are realised in the environment.

The joint training objective includes FB value, the conditional motion prior, physical auxiliary terms, and heading and velocity supervision. They support task representation learning, motion quality, physical constraints, and root control respectively; the velocity estimator is also supervised with the simulator's ground-truth velocity. The paper reports a training scale of 8 H200 GPUs, about 10 hours, and 192 million environment steps, with training experience made up of flat-ground demonstrations and online simulation interaction.

Depth perception and root velocity estimation

The policy uses a shared CNN and GRU to encode 8 frames of depth history, and uses visual features together with proprioception for control. Each frame is 18×32 pixels, sampled at intervals of about 0.1 seconds, covering about 0.7 seconds of observation history. This lets the model use both the surrounding geometry and how it changes as the robot moves.

Depth features are also used to estimate the root's three-dimensional linear velocity. The estimator fuses depth, proprioception, and action history, is supervised with ground-truth velocity in simulation, and runs on onboard observations at deployment. It receives neither behaviour intent nor goal instructions, so estimating the current motion state and specifying the desired motion are done separately.

The real robot uses a RealSense D435i depth camera. Camera observations are aligned to the ray layout used in training and processed by cropping, smoothing, and normalisation; both the maximum depth and the invalid-depth replacement value are 2.5 metres. After observation alignment, the model is deployed to the Unitree G1 with frozen parameters.

04 ERI: controlling root motion while preserving behaviour

Why explicit root instructions are still needed

Similar joint motions do not necessarily correspond to the same global displacement. The robot may keep alternating steps yet gradually drift away from the target direction, or step in place in front of an obstacle. Although the behaviour latent variable contains body motion information, it may still be unable to constrain heading and forward velocity precisely.

ERI supplies the target heading and planar velocity directly to the policy and sets up separate value supervision for the two kinds of instruction. During motion tracking, the behaviour intent and root instructions come from the same reference, controlling whole-body behaviour and global displacement respectively, maintaining the target direction and forward velocity while adapting to the terrain.

Instruction expression and local adaptation

The target horizontal velocity is converted from the world coordinate frame to the robot's current heading frame, and the target heading is input as a periodic representation of the relative angle. The policy thereby obtains direction and speed requirements relative to its current heading.

A validity mask distinguishes "target velocity is zero" from "velocity unspecified": the former requires stopping, while the latter applies no corresponding velocity supervision. ERI constrains planar motion; the vertical motion needed to go up and down stairs and local contact adjustments are handled by the perceptive policy.

Ablation: motion tracking and root control must be evaluated separately

After removing the heading instruction, the heading error on the SOMA evaluation rises from 6.7° to 14.6°; after removing the velocity instruction, the horizontal velocity error rises from 0.21 m/s to 0.32 m/s. The joint-angle MAE of all three configurations is about 0.13 rad, showing that similar local motion tracking quality can accompany clearly different root motion accuracy.

The configuration without velocity instructions has a heading error of 6.3°, slightly lower than the full model. The experiments therefore mainly support that the two kinds of instruction each improve the corresponding control dimension, and also show that joint-angle error alone is insufficient to measure the execution quality of whole-body motion.

05 One frozen policy, three task inputs

After pretraining, the model parameters stay fixed. Motion references, target body states, and reward functions construct different behaviour intents through the B representation and are executed by the same perceptive policy.

Reward inference: specifying behaviour preferences with a reward function

A reward function can express requirements such as forward velocity and pelvis height. P-BFM reward-weights and normalises the B representations of experienced body states to convert these requirements into an intent vector. Given a reward and a set of sampled states, the inferred intent stays fixed, and the policy generates continuous actions according to the terrain where it is located.

This way specifies behaviour by state preferences without providing a complete motion trajectory. The real-robot experiments show stair walking under velocity and pelvis-height preferences, with the velocity and height values being task settings.

Motion tracking: terrain adaptation driven by a reference motion

Motion tracking takes a reference sequence as input. Starting from the current reference instant, P-BFM selects up to 8 frames of body states, encodes them with B, averages and normalises them to obtain the current behaviour intent. The reference window advances with time, so the intent continuously reflects changes in the motion.

The perceptive policy follows the reference behaviour while adjusting joint motion and contact mode according to the terrain. Real-robot demonstrations include dancing on a platform and a dynamic side flip near stairs, showing execution of motion references under complex contact conditions.

Teleoperation connects through the same tracking interface to real-time retargeted body state goals. The input switches from pre-recorded motions to the operator's real-time motion, and the robot continues to adapt to the surrounding terrain according to its own depth observations.

Goal reaching: generating transition motions from a target body state

Goal reaching needs only one target body state. P-BFM encodes it with B and normalises it to obtain the behaviour intent; the intermediate motion from the current state to the goal is generated by the policy.

On stairs or slopes, approaching the same goal may require different support positions and contact sequences. The real-robot demonstrations present body state transitions on stairs, with the robot coordinating whole-body contact while sitting, lying down, supporting itself, and standing up.

06 Recovering motion using the surrounding terrain

After the robot is disturbed, the recovery actions available depend on the state of imbalance and the surrounding geometry. Obstacles may restrict the direction of leg swing, and stair surfaces may provide new support. Recovery requires adjusting body posture, foothold position, and task execution at the same time.

In the real-robot clips, the robot steps onto stairs to establish support after being pushed, and lifts a leg over a nearby obstacle after being kicked. These motions show how footholds and support adapt to the surrounding terrain during recovery.

Tracking recovery evaluation in simulation

The paper evaluates disturbance recovery on 281 complete 10-second windows from 180 SOMA recorded motions. A horizontal force of 50, 100, 150, or 200 N is applied to the pelvis at 2.0–2.2 seconds, and the reference motion keeps playing. The metric RR@5s measures the proportion that recovers or maintains target heading and velocity tracking after the force is removed; success requires that within 5 seconds after removal the heading error is below 30° and the horizontal velocity error does not exceed 0.5 m/s, and that this holds until the end of the test for at least 1 second; always satisfying the tolerance also counts as success, and a reset does not count as success.

07 Tracking and control accuracy on motions outside the demonstrations

The paper uses 994 SOMA recorded motions to evaluate out-of-demonstration tracking capability. These motions are not among P-BFM's 40 LaFAN training sequences. The evaluation matches the robot model and terrain conditions, covering seven initial terrain categories and ten terrain rows, with five paired trials for each motion and terrain condition.

The comparisons include UFO, an FB-based unsupervised control framework, and PMT, which achieves perceptive tracking through offline reference adaptation. The evaluation reports joint angle, horizontal velocity, heading, and foot-slip metrics separately, to observe performance at both the motion reproduction and global control levels.

Compared with UFO, P-BFM's joint-angle MAE drops from 0.16 rad to 0.13 rad, horizontal velocity error from 0.37 m/s to 0.21 m/s, and heading error from 28.9° to 6.7°. The improvements appear in both local motion tracking and root control, consistent with the observations from the ERI ablation above.

The evaluation compares states directly against the reference clock without post-hoc time alignment; after an ordinary tracking failure, subsequent execution is also counted as long as the simulation can continue. Some baseline runs end early on reaching the map boundary, and metrics are computed over the recorded period, so run durations are not fully identical.

Out-of-demonstration generalisation here is relative to P-BFM's training motion set; PMT's full training coverage is unknown. Both training and evaluation use the seven terrain categories configured in the paper; the standard deviations in the table represent variation between evaluation runs and do not involve multiple independently trained models.

From motion prior to perceptive control

P-BFM shifts the role of motion demonstrations from covering behaviour–terrain combinations one by one to providing a reusable motion prior. The FB representation unifies task inputs, online interaction learns terrain adaptation, and ERI adds global motion constraints, connecting different control tasks to the same perceptive policy.

The current results cover multiple kinds of motion, terrain, and real-robot contact behaviour. The capability range still depends on the motion prior, the environments explored, and the learned representation; broader environment coverage and more complex whole-body contact remain to be further validated.