Applied Intuition · Technical Report

TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations

A demonstration-free recipe for end-to-end driving: learn to drive by self-play in a vectorized simulator, then align a vision policy to the frozen teacher — no expert supervision at any stage.

Zikang Xiong¹, Weixin Li¹, Zhouchonghao Wu¹, Akshay Rangesh¹, Saarth Bonde¹, Grantland Hall¹, Chen Tang², Yihan Hu¹, Wei Zhan^1,3

¹ Applied Intuition · ² UCLA · ³ UC Berkeley

arXiv

The Idea

Decouple learning to drive
from learning to see

Master driving first by self-play in a fast vectorized simulator — then let the agent see, by aligning a vision frontend to the skills it already has.

0expert demonstrations, at any stage

≈2,535 yrsof self-play driving experience (8×10¹¹ steps)

0.490HD-Score — beats every end-to-end baseline

Full abstract

End-to-end autonomous driving has achieved state-of-the-art performance on benchmarks and real-world deployments. Its standard training recipe, however, is expensive across all stages: collecting and labeling millions of driving frames is costly, and closed-loop RL on images is bottlenecked by the per-step cost of photorealistic rendering plus a forward pass through a large vision backbone. Self-play in vectorized simulators changes the economics: millions of rollout steps per second, and a state distribution naturally rich in collisions, near-misses, and recoveries that no driving log contains.

Our approach exploits this asymmetry by decoupling learning to drive from learning to see. We pretrain a single policy by self-play, then align its latent space with a pretrained vision backbone, through the action KL divergence and a batch-relational low-rank structural loss. The action target comes from the self-play policy, so alignment never supervises against a logged trajectory: a paired dataset of (image, scene-state) frames suffices, with no need for the curated expert demonstrations that imitation pretraining is built on. On photorealistic 3D Gaussian splatting closed-loop scenarios, the resulting end-to-end policy matches or exceeds prior end-to-end methods.

Method · Phase 1

Learn to drive — by self-play

One shared policy controls every agent in TerraZero, our vectorized simulator. Trained with PPO, it learns from the long tail of its own behavior — collisions, near-misses, recoveries — never imitating a human.

8×10¹¹simulated steps

≈23Mdriving hours

≈2,535 yrsof driving experience

0demonstrations

Method · Phase 2

Learn to see — align a vision student to the frozen teacher

The self-play policy is frozen and becomes the teacher. A DINOv3 encoder with a trained adapter maps camera images into the teacher’s feature space — supervised by a feature alignment loss and an action KL loss on paired (image, scene-state) frames. No logged trajectory is ever a target.

Results

Matches or beats every end-to-end baseline

Closed-loop HD-Score on HUGSim — 88 photorealistic 3D-Gaussian-splatting scenarios. Aligned on nuPlan, evaluated on nuScenes: the scores also reflect cross-dataset generalization.

UniAD0.224

VAD0.239

LTF0.360

ECO (best variant)0.452

TerraTransfer (ours)0.490

Self-play teacher (privileged state)0.520

Within 0.03 of its own privileged teacher — with no logged-trajectory supervision. The one exception is the strongly out-of-distribution Extreme tier, where our policy’s conservative responses preserve safety but sacrifice route completion.

Full per-tier table

Closed-loop HD-Score on HUGSim (higher is better). **Bold** marks the best per column among comparison methods; the self-play teacher drives from privileged vectorized state and is excluded.
Method	Easy	Medium	Hard	Extreme	All
UniAD	0.367	0.198	0.249	0.109	0.224
VAD	0.400	0.228	0.242	0.095	0.239
LTF	0.634	0.391	0.289	0.098	0.360
ECO (Smoothing-only)	0.764	0.416	0.405	0.255	0.452
ECO (Smoothing + Re-time)	0.720	0.388	0.342	0.236	0.415
Self-play teacher (ref., privileged state)	0.780	0.497	0.639	0.185	0.520
TerraTransfer (Ours)	0.769	0.501	0.560	0.150	0.490

Qualitative Rollouts

Naturalistic driving across the long tail

Terra Series

Part of the Terra Series

Procedural Driving Simulation for Self-Play RL at Scale

TerraZero is the vectorized simulator and self-play recipe behind Phase 1 of TerraTransfer: a fast object-level traffic simulator, a procedural scenario generator, and a compute-efficient self-play recipe that trains driving policies from scratch — with zero human demonstrations — and transfers them across cities and datasets. Its frozen self-play planner is the teacher distilled into the end-to-end vision policy on this page.

Project Page →

BibTeX

Cite This Work

@article{xiong2026terratransfer,
  title   = {TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations},
  author  = {Xiong, Zikang and Li, Weixin and Wu, Zhouchonghao and Rangesh, Akshay
             and Bonde, Saarth and Hall, Grantland and Tang, Chen and Hu, Yihan and Zhan, Wei},
  journal = {arXiv preprint arXiv:2606.17386},
  year    = {2026}
}