Applied Intuition · Technical Report

TerraTransfer++:
Demonstration-Free End-to-End Planner

Learn to drive by self-play in a vectorized simulator, learn to plan by trajectory distillation from a flow-matching teacher, then learn to see by aligning a vision student to either frozen teacher — no human demonstrations at any stage.

Zikang Xiong¹, Weixin Li¹, Zhouchonghao Wu¹, Akshay Rangesh¹, Saarth Bonde¹, Grantland Hall¹, Chen Tang², Yihan Hu¹, Wei Zhan^1,3

¹ Applied Intuition · ² UCLA · ³ UC Berkeley

arXiv

Terra Series Origin

TerraZero

Learn to drive in self-play

Fast, feature-rich driving simulation via self-play
2.8M steps/s on an 8× server node
State‑of‑the‑art driving performance (94.19 on nuPlan val14, 70.87 on InterPlan)

Jun 2026

TerraTransfer

Transfer TerraZero to E2E driving by learning to see

First of its kind: brings a self-play policy into E2E driving by learning to see via image-to-vector encoder alignment
Driving policies learnt from 224M miles / 2,535 years of experience in self-play
State-of-the-art closed-loop E2E driving performance (0.490 HD‑Score on HUGSim)

Oct 2026 · This work

TerraTransfer++

Learn to plan in long horizon for TerraTransfer

Planning capability distilled from self-play rollout experience
State‑of‑the‑art open-loop and closed-loop E2E driving performance (59.4 EPDMS on navhard, 95.5 PDMS on navtest, 0.548 HD‑Score on HUGSim)

Philosophy

Decouple learning to drive and plan
from learning to see

Master driving first by self-play in a fast vectorized simulator, turn its rollouts into a trajectory planner — then let the agent see, by aligning a vision frontend to the skills it already has.

0human demonstrations, at any stage

#1ranking on NAVSIM navtest, navhard, and HUGSim

801 Bsimulated steps in self-play

Full abstract

End-to-end autonomous driving has achieved state-of-the-art performance on benchmarks and real-world deployments. Its standard training recipe, however, is expensive across all existing paradigms: imitation learning requires fleet-scale driving logs whose behavior distribution only thinly covers the safety-critical tail; closed-loop reinforcement learning in a photorealistic simulator, while offering diverse behaviors, is bottlenecked by sample-efficiency, per-step cost of rendering, and a forward pass through a large vision backbone. Self-play in vectorized simulators changes the economics: at millions of rollout steps per second, one can obtain a state distribution with natural diversity in collisions, near-misses, and recoveries that no driving log contains, and a highly reactive policy emerges from a basic driving-quality reward. Our approach exploits this asymmetry by decoupling learning to drive and learning to plan from learning to see. We pretrain a single policy by self-play, distill its rollouts into a flow-matching trajectory teacher that also plans, and train lightweight adapters on a frozen pretrained vision backbone to reproduce the latent space of either teacher through the action Kullback-Leibler divergence, a batch-relational low-rank structural loss, and, for the planner, a trajectory-matching loss. Every target traces back to the self-play policy, so alignment never supervises against a logged trajectory: a paired dataset of (image, scene-state) frames suffices, with no need for the curated expert demonstrations that imitation pretraining is built on. From this one self-play teacher we obtain two end-to-end students: a closed-loop policy that outperforms every published imitation-based end-to-end method on the photorealistic 3D Gaussian splatting closed-loop benchmark HUGSim, and a planner that reaches 59.4 EPDMS on NAVSIM navhard, outperforming every published method including DriveZero-Scale (57.1), and 95.5 PDMS on navtest, without human demonstrations.

Method · Phase 1

Learn to drive — by self-play

One shared policy controls every agent in TerraZero, our vectorized simulator. Each agent observes its ego state, nearby road polylines and traffic controls, and neighboring agents. Trained with PPO, it learns from the long tail of its own behavior — collisions, near-misses, recoveries — never imitating a human.

TerraZero · massively-parallel self-play224M miles

0human demonstrations

#1ranking on val14 and InterPlan

8.0 ×10¹¹steps in simulation

224M milesof simulated driving

469 ×Earth-to-Moon round trips in driving distance

2,535 yrsof self-play driving experience

Method · Phase 2

Learn to plan — distill self-play rollouts into a trajectory teacher

The self-play policy is frozen and its rollouts become a trajectory dataset: for every frame, the target is the policy’s own realized 8 s future, never a logged human trajectory. A flow-matching trajectory teacher learns these futures and actions from the same vector state, giving the self-play teacher a planning interface.

Phase 2: self-play rollouts of the frozen policy form a dataset of realized 8-second futures and actions at 10 Hz; a trajectory teacher with a vector-state encoder and flow-matching head is trained with flow-matching and action losses to output a planned trajectory.

0logged trajectories

#1ranking on NAVSIM navhard and navtest

1 hof rollouts on 64 A100 GPUs

Method · Phase 3

Learn to see — align a vision student to either frozen teacher

A frozen DINOv3 backbone with trained adapters maps camera images into the teacher’s scene features, while the teacher’s heads are copied and frozen. Three losses tie student to teacher on paired (image, scene-state) frames: a feature alignment loss, an action KL loss, and, for the planner, a trajectory-matching loss. The self-play teacher yields the closed-loop student; the trajectory teacher yields the planning student. No logged trajectory is ever a target.

Phase 3: a camera image goes through DINOv3 plus a trained adapter into scene features; the copied, frozen action and trajectory heads produce driving actions and a planned trajectory. The frozen teacher (self-play or trajectory teacher) reads 3D vector state. Feature alignment, trajectory match (L2), and action match (KL) losses tie the two.

0online rendering

#1ranking on HUGSim

10 hof alignment on 8 A100 GPUs

Results · HUGSim

Outperforms every published end-to-end baseline in closed loop

Closed-loop and native HD-Score on HUGSim — all 436 photorealistic 3D-Gaussian-splatting scenarios across four dataset splits (nuScenes 88, PandaSet 127, KITTI-360 113, Waymo 108), pooled over all scenarios.

Closed-loop HD-Score

VAD0.187

UniAD0.244

LTF0.266

ECO0.428

TerraTransfer++ (ours)0.548

Self-play teacher (privileged state)0.562

Native HD-Score

VAD0.183

UniAD0.247

LTF0.274

ECO0.344

TerraTransfer++ (ours)0.528

Self-play teacher (privileged state)0.542

Native HD-Score (earlier 345-scenario protocol*)

GigaPixel0.385

DriveZero0.466

TerraTransfer++ (ours)0.518

*GigaPixel and DriveZero report only the pre-challenge 345-scenario protocol and their checkpoints are not public, so we re-evaluate ours on the same 345 scenarios with HUGSim’s native scorer.

The student retains roughly 97.5% of its privileged teacher’s closed-loop score and leads every baseline on the Easy, Medium, and Hard tiers and the All aggregate of every split. The exception is the Extreme tier, whose partly implausible adversarial scenarios reward never reacting; like its teacher, the student yields as a human driver would and sacrifices route completion.

Full per-split table (closed-loop and native HD-Score)

Closed-loop and native HD-Score across all four HUGSim splits (higher is better). Both blocks score the same re-run rollouts; UniAD, VAD, and LTF are re-run from their released checkpoints, and ECO uses its released checkpoint. The student emits a single jerk action per step and is scored natively with a decel stand-in plan. **Bold** marks the best comparison entry per column within each split; the self-play teacher drives from privileged vectorized state and is excluded.
	Closed-loop HD-Score					Native HD-Score
Method	Easy	Medium	Hard	Extreme	All	Easy	Medium	Hard	Extreme	All
nuScenes (88)
UniAD	0.423	0.231	0.286	0.122	0.259	0.412	0.234	0.290	0.129	0.260
VAD	0.443	0.256	0.270	0.108	0.267	0.424	0.246	0.257	0.113	0.257
LTF	0.708	0.440	0.326	0.111	0.404	0.702	0.440	0.331	0.124	0.407
ECO	0.764	0.416	0.405	0.255	0.452	0.846	0.431	0.403	0.263	0.476
Self-play teacher (ref., privileged state)	0.906	0.684	0.685	0.279	0.647	0.875	0.657	0.653	0.250	0.618
TerraTransfer++ student (ours)	0.901	0.682	0.641	0.278	0.636	0.884	0.650	0.620	0.251	0.610
PandaSet (127)
UniAD	0.577	0.380	0.235	0.125	0.295	0.576	0.384	0.242	0.135	0.301
VAD	0.477	0.305	0.192	0.108	0.241	0.467	0.303	0.191	0.113	0.241
LTF	0.882	0.376	0.208	0.085	0.311	0.877	0.386	0.219	0.097	0.323
ECO	0.884	0.639	0.174	0.247	0.446	0.876	0.470	0.044	0.156	0.326
Self-play teacher (ref., privileged state)	0.890	0.883	0.302	0.224	0.554	0.876	0.872	0.275	0.186	0.532
TerraTransfer++ student (ours)	0.892	0.871	0.353	0.234	0.565	0.882	0.855	0.324	0.189	0.539
KITTI-360 (113)
UniAD	0.231	0.101	0.079	0.024	0.106	0.232	0.107	0.082	0.028	0.110
VAD	0.084	0.068	0.075	0.026	0.063	0.079	0.063	0.071	0.029	0.061
LTF	0.175	0.071	0.066	0.026	0.082	0.181	0.076	0.075	0.033	0.088
ECO	0.530	0.432	0.126	0.108	0.310	0.501	0.282	0.019	0.008	0.205
Self-play teacher (ref., privileged state)	0.659	0.497	0.244	0.019	0.363	0.650	0.488	0.205	0.015	0.348
TerraTransfer++ student (ours)	0.550	0.457	0.219	0.017	0.321	0.535	0.450	0.195	0.014	0.309
Waymo (108)
UniAD	0.591	0.275	0.334	0.095	0.315	0.585	0.278	0.340	0.101	0.317
VAD	0.237	0.197	0.208	0.109	0.189	0.226	0.189	0.199	0.114	0.183
LTF	0.520	0.312	0.218	0.100	0.292	0.530	0.325	0.225	0.111	0.303
ECO	0.801	0.581	0.339	0.254	0.510	0.757	0.463	0.213	0.124	0.403
Self-play teacher (ref., privileged state)	0.924	0.854	0.817	0.115	0.710	0.912	0.847	0.777	0.102	0.694
TerraTransfer++ student (ours)	0.941	0.836	0.758	0.116	0.695	0.936	0.827	0.717	0.100	0.679

Results · NAVSIM

A camera-only planner outperforming every published method

PDMS on NAVSIMv1 navtest and two-stage EPDMS on NAVSIMv2 navhard. The trajectory student plans a 4 s trajectory from cameras and ego state, with no human trajectory supervision.

navtest · PDMS (axis starts at 80)

Privileged reference (ground-truth scene state)

Human driver94.8

PDM-Closed89.1

DriveRL (DriveZero teacher)95.8

Trajectory teacher (ours)95.9

Sensor input, human demonstrations

UniAD83.4

TransFuser84.0

VADv286.2

DiffusionDrive88.1

ReCogDrive (VLA)90.8

Hydra-MDP++91.0

Centaur92.6

DriveSuprim (camera + LiDAR)93.5

DrivoR-Scale94.6

Camera input, no human demonstrations

DriveZero94.8

DriveZero-Scale95.3

TerraTransfer++ (ours)95.5

navhard · two-stage EPDMS

Privileged reference (ground-truth scene state)

PDM-Closed56.6

DriveRL (DriveZero teacher)55.6

Trajectory teacher (ours)62.1

Trajectory teacher (ours, IDM selection)65.6

Camera input, human demonstrations

LTF25.1

NavFormer34.1

RAP39.6

GuideFlow51.5

SimScale53.2

DrivoR-Scale54.6

Camera input, no human demonstrations

ZTRS48.1

GigaPixel50.1

DriveZero51.5

DriveZero-Scale57.1

TerraTransfer++ (ours)59.4

Hatched bars read privileged ground-truth scene state at test time.

On navhard the student reaches 59.4 EPDMS, 2.3 points above DriveZero-Scale (57.1) and 2.7 below its privileged, selection-based trajectory teacher (62.1). On navtest it reaches 95.5 PDMS, outperforming every sensor-based entry including DriveZero-Scale (95.3).

Full NAVSIM table

Pseudo closed-loop planning on NAVSIM (higher is better). `navtest` reports PDMS (NAVSIMv1); `navhard` reports per-stage scores and the two-stage EPDMS (NAVSIMv2). Baseline rows are the published results compiled by DriveZero; – marks a score that compilation does not list. *Demo*: human trajectory demonstrations used as training supervision. *Priv.*: reads privileged ground-truth scene state at test time. Italic rows are privileged references. Both teacher rows keep the best of 16 sampled plans; the student scores a single sampled plan with no selection. **Bold** marks the best score in each group.
			`navtest`	`navhard`
Method	Demo	Priv.	PDMS	Stage 1	Stage 2	EPDMS
Privileged reference (ground-truth scene state)
Human driver	–	–	94.8	–	–	–
PDM-Closed	✓	✓	89.1	–	–	56.6
DriveRL (DriveZero teacher)	✗	✓	95.8	84.1	65.4	55.6
TerraTransfer++ trajectory teacher (ours)	✗	✓	95.9	92.2	67.5	62.1
TerraTransfer++ trajectory teacher (ours, IDM selection)	✗	✓	95.9	94.6	69.5	65.6
Camera & LiDAR input, human demonstrations
TransFuser	✓	✗	84.0	–	–	–
DRAMA	✓	✗	85.5	–	–	–
VADv2	✓	✗	86.2	–	–	–
DiffusionDrive	✓	✗	88.1	–	–	–
Hydra-MDP++	✓	✗	91.0	–	–	–
Centaur	✓	✗	92.6	–	–	–
DriveSuprim	✓	✗	93.5	–	–	–
Camera input, human demonstrations
UniAD	✓	✗	83.4	–	–	–
PARA-Drive	✓	✗	84.0	–	–	–
Epona	✓	✗	86.2	–	–	–
OneVL	✓	✗	88.4	–	–	–
DriveLaW	✓	✗	89.1	–	–	–
AutoVLA	✓	✗	89.1	–	–	–
DriveVLA-W0	✓	✗	90.2	–	–	–
Qwen-Drive-1.0	✓	✗	90.7	–	–	–
ReCogDrive	✓	✗	90.8	–	–	–
R2SE	✓	✗	91.6	–	–	–
iPad	✓	✗	91.7	–	–	–
DriveFine	✓	✗	91.8	–	–	–
LTF	✓	✗	–	–	–	25.1
NavFormer	✓	✗	–	–	–	34.1
RAP	✓	✗	–	–	–	39.6
GuideFlow	✓	✗	–	–	–	51.5
SimScale	✓	✗	–	–	–	53.2
DrivoR-Scale	✓	✗	94.6	–	–	54.6
Camera input, no human demonstrations
ZTRS	✗	✗	–	–	–	48.1
GigaPixel	✗	✗	–	77.8	63.5	50.1
DriveZero	✗	✗	94.8	84.1	60.8	51.5
DriveZero-Scale	✗	✗	95.3	82.3	69.1	57.1
TerraTransfer++ trajectory student (ours)	✗	✗	95.5	89.1	66.5	59.4

Plan Visualization

What the camera-only planner plans

Plans of the camera-only trajectory student on NAVSIM, 16 scenes per panel. The planned trajectory is drawn in blue on the front camera, with a bird’s-eye view inset.

`navtest`

`navhard` · Stage 1

`navhard` · Stage 2

Qualitative Rollouts

Naturalistic driving across the long tail

Closed-loop HUGSim rollouts of the camera-only TerraTransfer++ student, two scenarios per dataset, shown through the three front cameras.

TerraTransfer++:
Demonstration-Free End-to-End Planner

Decouple learning to drive and plan
from learning to see

Learn to drive — by self-play

Learn to plan — distill self-play rollouts into a trajectory teacher

Learn to see — align a vision student to either frozen teacher

Outperforms every published end-to-end baseline in closed loop

A camera-only planner outperforming every published method

What the camera-only planner plans

`navtest`

`navhard` · Stage 1

`navhard` · Stage 2

Naturalistic driving across the long tail

nuScenes

PandaSet

KITTI-360

Waymo Open Dataset

Part of the Terra Series

TerraTransfer++:Demonstration-Free End-to-End Planner

Decouple learning to drive and planfrom learning to see

Learn to drive — by self-play

Learn to plan — distill self-play rollouts into a trajectory teacher

Learn to see — align a vision student to either frozen teacher

Outperforms every published end-to-end baseline in closed loop

A camera-only planner outperforming every published method

What the camera-only planner plans

navtest

navhard · Stage 1

navhard · Stage 2

Naturalistic driving across the long tail

nuScenes

PandaSet

KITTI-360

Waymo Open Dataset

Part of the Terra Series

TerraTransfer++:
Demonstration-Free End-to-End Planner

Decouple learning to drive and plan
from learning to see

`navtest`

`navhard` · Stage 1

`navhard` · Stage 2