ABC at Individual Scale: Reproducing a Bimanual Behavior-Cloning Stack on One A100

ABC — "Scalable Behavior Cloning with Open Data, Training, and Evaluation" (UC Berkeley / MIT / Amazon FAR / XDOF / CMU, 2026, abc.bot) — is one of the largest open efforts in real bimanual manipulation. It ships a 3,553-hour teleop dataset, two flow-matching chunk policies, a MuJoCo sim suite with near-real correlation, and a set of clean scaling claims. It was trained on 64–96 H200 GPUs.

I wanted to know a narrower thing: which of those results are reachable, reproducible, or falsifiable on a single A100-40GB — the machine an individual researcher actually has. That constraint is not a footnote; it is the entire experiment. Batch size, optimizer memory, and step budget are all bounded by one 40 GB card, and it turns out several of the paper's headline behaviors depend on exactly the throughput I don't have.

Everything below ran on that one GPU. The code is on github.com/5usu/abc-a100 (upstream: amazon-far/abc).

What the paper gives you

• ABC-130K — 3,553 h / 134,806 episodes / 195 tasks of real bimanual teleop on an $8k YAM station with 3× RealSense D405. The largest open teleop dataset.
• ABC-DiT (2B: DINOv3 ViT-B + 1.93B DiT head) and ABC-VLA (4.3B Gemma-3 + 44.7M adaLN flow head), both flow-matching chunk policies emitting 30×14 actions at 30 Hz.
• ABC-Sim — MuJoCo(-Warp) environments plus 400 h of sim teleop, with sim↔real correlation r ≈ 0.9.
• Findings: DINOv3-xattn beats CLIP; DiT is more compute-efficient while VLA wins at batch 9216; bigger DiT means lower loss at fixed steps (their Fig. 21); train loss and validation action error predict real-world success while validation loss does not; the finetune recipe is lr 1e-5.

At the time of this work only a minimal release existed: the DiT single-task pipeline, one "bottles-in-bin" task with data and a sim env, and a single 75k from-scratch checkpoint. The VLA code, the abcdl dataloader, all robot / DAgger / VR infrastructure, the remaining sim tasks, and the 200k base checkpoints were slated to ship "end of July." So much of the reproduction had to be built from the paper's spec rather than run from released weights.

What I built on one GPU

1. Single-GPU adaptation of the trainer — gradient accumulation with correct optimizer-step semantics, configurable pretrained-checkpoint init, tuned defaults, peak-VRAM logging. The VRAM profile is unforgiving: micro-batch 8 is safe, 16 sits at the 40 GB ceiling with expandable_segments, and 32 OOMs — the static AdamW state of the 2B model alone is ~31–32 GB.
2. A DINOv3 weight converter (convert_dinov3.py) — HuggingFace format back into the original DINOv3 layout (fused QKV in q,k,v order with masked K-bias, register→storage tokens, layer_scale→ls.gamma, and so on). Verified: zero unexpected keys, finite features. This converter exists nowhere else publicly, because Meta gates the original .pth.
3. An ABC-VLA reproduction from the paper spec (the official code was unreleased) — Gemma-3-4B + 8×512 QK-Norm attention pooling + a 5120→512 adaLN conditioning path + a 45.6M flow head + the k=8 diffusion-draws trick, plus the matching Gemma token layout and SigLIP 896px preprocessing, a frozen-bf16-backbone trainer over a real+sim mixture, and a closed-loop sim evaluator. Parameter match to the paper: backbone 4.32B (paper 4.3B), head+interface 45.6M (paper 44.7M).
4. Eval infrastructure — 50-world protocols, per-world seed reproduction, rollout video/GIF rendering, and delivery.

Result 1 — the finetune learning rate makes or breaks the policy

The single sharpest lesson. Starting from the released bottles_75k.pt checkpoint and finetuning on the bottles task (sim eval, 50 worlds unless noted):

*10-world eval. Champion checkpoint: cache/bottles_ft_full_lr5_2k.pt.

• The finetune LR is the whole ballgame. The pretraining LR of 1e-4 collapses a converged checkpoint to 0% at any data scale — more data does not rescue it. The paper's lr=1e-5 recipe lifts the released model from 12% to 18%.
• Gains saturate by ~2k steps at this batch size; the duration curve is non-monotonic and 2k / 10k / baseline differ by only 1–3 successes.
• Offline metrics predict collapse-vs-healthy perfectly but cannot rank similar checkpoints: action error improved monotonically (0.156→0.084) across 2k/5k/10k while sim success went 18%→10%→16%.

Result 2 — the size-scaling curve does not survive a small batch

The paper's Fig. 21 shows bigger DiT → lower loss at fixed steps. Training from scratch (converted DINOv3 init), lr 1e-4, effective batch 16:

• A 3k-step sweep over S/B/L/xL (153M / 290M / 746M / 2016M, paper-exact sizes) gave losses of 0.16–0.22 with no size ordering at all.
• A 25k-step duel was blunter still: DiT-S at 0.107 train / 0.325 val-err vs DiT-xL at 0.121 / 0.390 — the 153M model matched or beat the 2B while training 1.7× faster.

Result 3 — reproducing ABC-VLA from the spec

With no official VLA code available, I rebuilt it from the paper description and trained the head with the backbone frozen in bf16.

• The full-scale dry run — frozen bf16 backbone + trainable head, B=4, k=8, 3×896px — peaked at 24.4 GB and ~1 s/iter. The paper's k-draws trick works exactly as advertised.
• The 10k real-only run landed in the same offline band as DiT-from-scratch (0.39 vs 0.390), matching the paper's small-batch finding that VLA ≤ DiT.
• Stage A confirmed the domain-gap hypothesis: adding sim episodes took val_sim from untrained to 0.0295, and the model visibly reaches for bottles in sim.
• Calibration insight: the "working band" is per-domain. Working DiTs score ~0.007 on sim validation (visuals are easier there), so 0.0295 is still ~4× short — the policy reaches but cannot grasp, hence 0/6. The offline↔closed-loop relationship holds once you calibrate it per-domain (confirmed five independent times across this work).
• The frozen backbone is now the binding constraint — exactly the paper's "freezing the VLM performs poorly" warning, now quantified.

Watching the policies run

Numbers only go so far; closed-loop video is where "reaches but cannot grasp" stops being a metric and becomes a thing you can see. Two rollouts, rendered from the sim evaluator, tell the two halves of the story — the finetuned DiT that mostly works, and the from-spec VLA that has learned to reach but not yet to close.

The eval noise floor

A caveat that shapes every number above: 50-world binary success only detects roughly 2× differences (about Δ≈7/50 for significance), and the per-world outcomes of similar checkpoints barely overlap. To rank close checkpoints you need mean_reward (300 bottle-outcomes per run) and a pinned physics path — vanilla and MuJoCo-Warp physics do not agree. Treating a 16% and an 18% run as "different" without that is reading noise.

What the paper reports that I cannot reach

• Headline success rates (e.g. bottles at 73.5% strict) require 200k steps at batch 9216 over 7,000 h of data — roughly 2–3 GPU-years on an A100 — and those weights were never released.
• Real-robot evaluation and sim↔real correlation — no hardware.
• Full-VLA backbone finetuning — the 4B AdamW state alone is ≥ 50 GB, over the card.

Infrastructure gotchas worth knowing

• torchcodec needs system ffmpeg (libavutil); every training/eval import fails silently without it.
• The data server does not support HTTP range requests: no resumable downloads, and prepare.py renames truncated downloads as complete. The fix is a size-verified whole-file curl loop with --http1.1 — expect several attempts for the 34 GB tar.
• Single-world eval: default MuJoCo-Warp physics is ~3.2 s/chunk; --vanilla-physics (CPU mj_step) is ~50× faster. Rendering (0.147 s/chunk) dominates inference (0.064 s).
• Step checkpoints are 23 GB (fp32 optimizer state); last.pt is 7.6 GB (model + norm stats).
• transformers ≥5: Gemma3 get_image_features returns an output object — the projected features live in .pooler_output.
• The released FastInferenceGraph is batch-1 hardcoded — the natural extension point for best-of-N sampling.

Where this goes next

Immediate, scoped and ready to build:

• Stage B — LoRA on the Gemma backbone (rank 16, attention + MLP, injected after weight load). Unfreezes the conditioning capacity that is currently the binding constraint; targets val_sim 0.0295→~0.007. ~1 day of code, 1–2 days of GPU.
• Best-of-N chunk selection — a BID-style verifier-free coherence score, or a SARM progress model as the scorer. Batch-N noise sampling is native to the sampler, and this is the fastest path to beating 18% on the DiT side.
• VLA action-prefix (RTC) training for parity with the paper's recipe.

When the full release lands: diff the from-spec VLA against the official implementation, and re-run the lr=1e-5 finetune from the 200k base checkpoint — likely a large jump over 18%.

Bigger swings: RL finetuning in MuJoCo-Warp (ReinFlow-style flow-RL; no published MJWarp RL exists yet), reward-model data curation, and cross-architecture distillation with a working DiT as teacher for the VLA head.

The takeaway

Shrinking the compute by two orders of magnitude does not just make the paper's numbers smaller — it changes which claims are even testable. The recipe findings (lr=1e-5 finetuning, offline metrics predicting collapse) reproduced cleanly and cheaply. The scaling findings (bigger DiT wins, VLA beats DiT) inverted, because they were always properties of throughput I didn't have. Knowing which is which is most of what an individual-scale reproduction is for.