Language-Conditioned Humanoid Manipulation on the Unitree G1

A humanoid robot that takes a plain-English command, looks at the scene, and executes the manipulation — end to end. A natural-language instruction routes to the right skill, an Action Chunking Transformer policy reads the robot's camera and joint state, and a 30 Hz ROS 2 control loop turns predictions into smooth, physically-grounded motion on a Unitree G1 in high-fidelity MuJoCo simulation. Both the single-arm and the two-handed tasks clear a rigorously validated evaluation at 100% success.

Demo

Each clip shows an overview camera alongside the robot's own egocentric view — the same frame the policy sees.

Single-arm pick-and-place — reach, grasp, lift, and place, run as one continuous behavior from a language command:

Bimanual box lift — both palms hold and raise the box by friction alone, under full contact physics with no attachment constraints:

Results

Every rate below comes from a pre-registered gate on a validated evaluation: baseline "floor" policies (untrained and mean-action) score 0%, so a passing number proves real learned skill rather than a leaky test.

Robustness to domain randomization

The single-arm policy was trained with per-episode visual randomization — lighting, table and object color, up to three distractor objects at random positions, and camera jitter — and holds up strongly when those perturbations are active at evaluation:

Graceful out-of-distribution degradation

Pushed beyond the range it ever trained on, the policy degrades smoothly instead of collapsing — the signature of a see-and-reach policy rather than one that memorized a fixed band:

The signature result: diagnosing a broken sensor

The single-arm task sat at a stubborn 54% plateau that survived every modeling idea thrown at it — a CVAE variant, a 5× increase in training data, and reshaped data distributions. Each was pre-registered and each failed to move the number.

Rather than keep tuning the model, a cheap observation-resolution probe asked a blunt question: how many pixels of the target object does the policy actually see? The answer was zero. The camera was aimed 95° off the workspace — the policy had been effectively blind the whole time, reaching toward the average object position (a textbook regression-to-the-mean undershoot, measured at 5 cm short at the band edge).

Re-aiming the camera was the entire fix. Success jumped from 54% to 100%, and the corrected policy now reaches the target with zero undershoot at every displacement. The lesson — verify the sensor before blaming the model — is exactly the kind of systematic root-causing that separates a working robotics stack from a stuck one.

Engineering: an evaluation you can trust

The headline numbers are only as good as the ruler that measures them, so the evaluation harness was built to be adversarial to its own results:

• Baseline floors on every eval — untrained and mean-action policies are graded alongside the real one; a valid evaluation pins them near zero. This tripwire caught a bug where an untrained network appeared to "win," and a case where a constant pose gamed a too-narrow task band.
• Disjoint test / selection seeds — checkpoints are selected on one block of seeds and graded on a completely separate one, so the reported number is never the number that was optimized against.
• Pre-registered, falsifiable gates — each phase's acceptance bar and rollback plan are written before the run, and directional claims are confirmed with a paired exact McNemar test plus Wilson 95% confidence intervals.

That discipline shows up directly in the two-handed task. Its first evaluation was declared invalid by its own floor row — a constant pose part-solved a too-easy setup — so instead of claiming the pass, the task was made harder (a one-way ratchet on the box-placement range). The harder setting exposed a real grasp failure at the edges of the range, caught automatically by a demonstration-acceptance gate; an improved scripted expert fixed it, regenerating a clean dataset. The retrained policy then cleared the unchanged 85% bar at 100% success, with a separation of +98 points over the floor (one-sided McNemar p ≈ 2 × 10⁻¹⁵).

The whole project runs like a small research lab: an automated multi-agent development pipeline with CI-style quality gates, 150+ unit and integration tests written test-first, deterministic seeding end to end, MuJoCo physics smoke tests, and checkpoint backup to Hugging Face.

System architecture

flowchart LR
    CMD["ROS 2 /vla/task_goal<br/>natural-language command"] --> RTR["Command router"]
    RTR --> EMB["Task embedding"]
    SIM["MuJoCo, 500 Hz<br/>Unitree G1"] --> IMG["Egocentric RGB<br/>640 x 480 → 224 x 224"]
    SIM --> JST["Joint state"]
    IMG --> ENC["ResNet-18 encoder"]
    EMB --> DEC["Transformer decoder<br/>4 layers, 20 query tokens"]
    ENC --> DEC
    JST --> DEC
    DEC --> CHK["Action chunk<br/>20 future joint targets"]
    CHK --> ENS["Temporal ensembling"]
    ENS --> PDC["PD control with<br/>gravity compensation, 30 Hz"]
    PDC --> SIM
    SIM --> ST["ROS 2 /vla/status"]

Each policy consumes an egocentric camera frame, the current joint state, and a task embedding, then predicts a chunk of 20 future joint targets — about two-thirds of a second of motion. Overlapping chunks are blended by temporal ensembling into a smooth 30 Hz target stream, which a PD controller with per-joint gravity compensation tracks while MuJoCo steps the physics at 500 Hz. Composite behaviors such as pick-then-place switch the task embedding at a grasp trigger, with a latch that prevents re-grasping after an intentional release.

Method

Policy. ~15.6M-parameter Action Chunking Transformer. A ResNet-18 encoder (frozen through its third stage) embeds the egocentric frame; joint state and the task embedding join as additional tokens; a 4-layer, 4-head transformer decoder with 20 learnable query tokens emits the action chunk. Trained with AdamW, cosine annealing, MSE loss, and mixed precision — the full single-arm policy trains in about 90 minutes on a single laptop GPU.

Data. Demonstrations are generated by scripted inverse-kinematics experts using iterative Jacobian IK with reachability checks, recorded at 30 fps and stored in HDF5 with a LeRobot-format converter. Every generated demonstration passes an acceptance gate before it can enter the dataset.

Control & integration. PD torque control with per-joint gravity compensation, physics at 500 Hz under control at 30 Hz. A ROS 2 Jazzy node subscribes to /vla/task_goal, runs inference in a thread-safe loop against the simulation, and publishes status on /vla/status; a rosbridge WebSocket endpoint lets external clients issue commands live.

Stack. PyTorch · MuJoCo 3 · ROS 2 Jazzy · ACT + ResNet18 · imitation learning · transformers · CVAE ablations · domain randomization · Jacobian IK experts · HDF5 data pipelines.

Further reading

The condensed engineering lessons from this build — controller design, IK fragility, ACT training practice, and bimanual contact — are written up in 50 Lessons from Building a Humanoid VLA System.