Locating the cause of simulated driving failures

We injected faults into a driving planner, collected 468 at-fault failures, and simulated every counterfactual from every decision before each one. Search turned out to be cheap. Defining the failure, the cause and the simulator’s share of the blame did not.

October 2026

Findings

1. The point of no return is cheap and exact to find: checkpointed bisection takes 3.8 forks, about 1.25 s of CPU, and agrees with exhaustive search on 467 of 468 failures.
2. The last decision before a failure is the wrong thing to blame. Braking from it avoids the crash in 12% of failures; the last recoverable decision comes a median 0.9 s earlier.
3. Cause is usually a window. 41% of failures have no single decision whose replacement prevents them, and 42% have several.
4. A quarter of failures against simulated traffic are simulator artifacts. They vanish when the other vehicle replays its recorded path, and a displacement threshold cannot tell which ones.
5. The collision label matters. A rule that counts rear-end contacts made recoverability look non-monotone in 9.4% of failures; an at-fault rule brings that to 0.2%.
6. Collision checks miss traffic-law failures: 7.5% of collision-free runs of the fault-free planner run a red light or share a crosswalk with a pedestrian.

Setup

Twenty real scenes from the Waymo Open Motion Dataset, simulated in Waymax with the 32 nearest objects. Other vehicles follow Waymax’s IDM route policy; pedestrians and cyclists replay their logs.

The ego vehicle runs a reactive test planner. Every 0.5 s it chooses among accelerate, maintain, decelerate and brake, keeping the fastest option whose path stays clear of perceived agents extrapolated at constant velocity.

We ran 300 start states (20 scenes, five start times, three speed offsets) under 19 configurations. Faults came in three families: perception outages of 1.5 s, acceleration glitches of 1 s, and persistent faults in perception range, pedestrian detection, observation latency and planning horizon. That produced 468 at-fault failures.

For every decision up to each failure, we forked the run four ways from a saved checkpoint: emergency brake from that decision onward, the fault-free planner from that decision onward, the fault-free choice at that decision only, and a brake at that decision only. Forks reproduce the original run exactly. The table holds 525,000 simulated steps; the whole study ran about 1.1 million on a 2-vCPU machine without a GPU.

Four different “whens”

Each is computed exactly from the fork table. They are not the same moment.

The alarm comes after the point of no return in 5.6% of failures, so they are detected only once they can no longer be fixed. For time-localized faults, the point of no return falls inside the injected fault window in 89% of failures.

Finding the point of no return

Each method blames one decision per failure. “In fault window” is scored on the 298 failures with a trusted window; the other columns on all 468.

One full 6 s replay costs 0.41 s on this CPU, so bisection costs about three replays. Checkpointing saves 19% of wall-clock time against bisection that replays from the start.

The zero-simulation heuristic, the first alarm the planner ignores by accelerating or holding speed, lands in the fault window 97% of the time. But a single brake there avoids the crash only 61% of the time. It is a good proposal and a poor verdict: propose with it, verify with one fork.

Cause is a window, not a decision

In 41% of failures, no single decision replaced by the fault-free planner’s choice prevents the failure. For persistent faults it is 63%, because the fault reasserts itself at the next decision. In another 42%, several different single decisions each suffice.

The definition that stays well-defined is fix-forward: hand control to the fault-free planner from decision k onward, and find the latest k that still avoids the failure.

Was it the planner or the simulator?

We reran every failure against an IDM vehicle with only that vehicle switched back to its recorded path.

An artifact here means the vehicle had already left its recorded path in a run where the ego replayed its own log, so nothing the ego did caused the deviation. Only 2 of 722 failures depend on the vehicle genuinely reacting to the ego.

The common test, “the other vehicle is more than 1 m off its log”, flags 434 failures, and 58% of those persist against the recorded driver. Separating real failures from simulator artifacts takes a counterfactual on the world, not a threshold.

With the ego replaying exactly what the human driver did, IDM traffic still produces 15 at-fault collisions. Of the IDM vehicles that moved in the log, 90% end more than 1 m off their recorded path after 6 s.

The collision label matters

Waymax’s IDM follower does not stop after contact, so a fast rear-ender drives through the ego. A rule that counts any later overlap labels those as ego failures.

Under that rule, 9.4% of failures look non-monotone (braking earlier fails, braking later succeeds) and bisection returns the wrong decision for 7.1%. Under an at-fault rule that classifies each vehicle’s first contact and excludes rear hits and hits on a stopped ego, 0.2% are non-monotone and bisection matches exhaustive search in 99.8%.

The same correction moves earlier baselines: an always-brake policy goes from 3.1% unsafe to 0.2%, and the fault-free planner’s failure rate from 9.7% to 1.7%.

Collision-free is not rule-compliant

Where language models help

Physics and search stay in code. As a proposer in front of the simulator, Jev (TypeSafe’s decision model) saves 0.28 forks per failure, about 0.1–0.3 s of CPU, and costs a 0.23 s call: no net gain. Language models earn their place where the requirement is written in words.

Jev is the only model that recognizes a late reaction, where the planner tracked the other agent but kept going: it catches 65% of those cases. The hand-written rule was authored by the person who designed the fault taxonomy, which favours it.

Limits

• Twenty scenes with repeated start states, one toy planner, and decisions every 0.5 s.
• Faults are injected and failures follow them closely, so these are proximate causes. Long-latency causes, such as a lane choice 5 s earlier, are untested.
• Traffic is the Waymax IDM model. 467 of 468 failure partners are vehicles.
• Fourteen plain-language labels were adjudicated after seeing model outputs. Prompts were written once and not tuned.
• Timings are on 2 vCPU without a GPU. Model latencies were measured through OpenRouter at concurrency 12. The model calls reported here cost $0.53 in total.

Questions about the method: devin.r.liu@gmail.com