The paper that started this
Earlier this year, Shi, Jiang, Tessler and Peng published "GPC: Large-Scale Generative Pretraining for Transferable Motor Control," one of the most elegant results in physics-based character animation. It showed that the "pretrain a big generative model, then adapt it cheaply" playbook that transformed language and vision also works for making simulated humanoids move.
GPC works in three stages. First, reinforcement learning compresses motion into a discrete vocabulary: an encoder squeezes a short window of upcoming reference motion into a small latent vector, and Finite Scalar Quantization (FSQ) snaps it onto a fixed grid. Every tenth of a second of motion becomes a discrete token. A decoder, conditioned on the character's proprioception plus that token, outputs motor commands, trained end to end with PPO against an imitation reward inside a physics simulator. Second, a GPT-style transformer models sequences of these skill tokens autoregressively. Motion becomes a language: sampling from the transformer generates behavior, subject to gravity, contact, and balance. Third, small conditional adapters steer the frozen prior toward downstream tasks.
The catch is in the first line of the abstract: large-scale. Nobody has 680 hours of motion for their character, their robot, their art style.
The question
Can you train a GPC-quality generative motor controller from a small motion library — tens of minutes, not hundreds of hours — on a single GPU, without the skill vocabulary collapsing?
Small data is where this recipe breaks, and it breaks in measurable ways: the tracker memorizes its clips instead of learning recovery, and the prior, starved of token diversity, degenerates into repeating one motion. The plan is a recipe of three countermeasures — physics-grounded augmentation, decoder self-rollout distillation, and anti-collapse regularization — each evaluated on data-efficiency curves at five, fifteen, thirty and sixty minutes of motion. Everything runs on one NVIDIA L4, a modest cloud GPU, deliberately.
What exists today
The project is an independent re-derivation, built from the papers, and Stage 1 now works end to end. The physics stack is MuJoCo-Warp, DeepMind's GPU-batched MuJoCo, which exposes simulation state as zero-copy PyTorch tensors: the environment, the PPO learner, and the FSQ bottleneck live in one framework with no serialization boundary. A 21-degree-of-freedom humanoid steps at roughly half a million simulation steps per second across 4096 parallel worlds, in under two gigabytes of memory. The full training loop sustains about 34,000 environment steps per second, with PD position servos at 180 Hz and the policy at 30 Hz — the layering DeepMimic used.
The first milestone gate asked: does the discrete bottleneck cost tracking quality? At an identical 65-million-step budget:
- Unconstrained MLP baseline: 100% tracking success at 7.1 cm mean per-joint error
- FSQ-bottlenecked tracker: 100% tracking success at 9.6 cm
Discreteness costs about 2.5 cm, and the vocabulary stays healthy — code perplexity around three thousand, no collapse. Skills survive quantization. The gate passed.
The data testbed is Ubisoft's LAFAN1: 77 mocap clips retargeted to the humanoid by whole-body damped-least-squares inverse kinematics over fifteen keypoints per frame. Ten seconds of motion retargets in about one second.
What the bugs taught me
Each of these silently produces "RL just doesn't learn" rather than an error.
The DeepMimic reward exists as a weighted sum and as a product of exponential terms. The
product form evaluates to roughly e^-50 for a freshly initialized policy
— zero reward everywhere it visits, a flat gradient landscape. Training was inert
until I switched to the published weighted sum.
Check what your reward evaluates to for a random policy, not just a good one.
Updating observation-normalization statistics between collecting a rollout and running the PPO epochs corrupts every stored log-probability; the adaptive learning-rate controller reads the inflated KL and throttles the learning rate into the floor. The fix is one line: freeze the statistics within an iteration.
MuJoCo's free-joint angular velocity is body-local, not world-frame — verified with a test scene, not assumed. Naive keyframe interpolation slid the reference feet 17 cm sideways and 11 cm underground; every clip now gets its feet pinned before it goes near the reward. And on diverse data, discrete code flips read as huge policy jumps to a KL-based trust region, strangling the learning rate: discrete representations need a looser trust region than continuous ones.
What is training now
A single FSQ tracker is learning to physically reproduce thirty minutes of LAFAN1 — 61 clips from walking to dance. Deliberately vanilla, it is the baseline every contribution gets measured against. Next comes the central idea: the Stage-1 decoder generalizes beyond its training clips, so roll it out under perturbed skill tokens, keep the physically stable trajectories, and train the transformer prior on that enlarged corpus. A few real clips in, unlimited on-distribution token sequences out. If the data-efficiency curves bend, that is the headline result.
Credits
GPC: Yi Shi, Yifeng Jiang, Chen Tessler, Xue Bin Peng — arXiv:2606.29148. The architecture re-derived here is theirs. FSQ: Mentzer, Minnen, Agustsson, Tschannen (2023). DeepMimic: Xue Bin Peng, Pieter Abbeel, Sergey Levine, Michiel van de Panne (2018). LAFAN1: Harvey, Yurick, Nowrouzezahrai, Pal — Ubisoft La Forge, used under its non-commercial research license. MuJoCo and MuJoCo-Warp, plus the humanoid model: Google DeepMind. PPO conventions follow the legged-robotics community, particularly legged_gym.
Code, configs, and retraining commands: github.com/5usu/gpc-small