Nemesis Failure investigation for autonomy simulation

Find the last moment a crash could have been stopped.

Nemesis rewinds failed simulation runs, tests what would have saved them, and checks whether the simulator caused the failure in the first place.

Bring us a failed run Read the research
Timeline of a failed simulated run: a perception outage from 1.0 to 2.5 seconds, an alarm at 1.0 seconds, collision at 4.2 seconds. Braking works from any decision up to 2.5 seconds; a fork that brakes at 2.5 seconds ends with no collision. Perception outage Alarm fires Last moment braking works Collision Faulty planner Forked run Emergency brake at 2.5 s: no collision Decisions 0 s 1 s 2 s 3 s 4 s Braking from this decision avoids the collision Too late to brake Timeline of a failed simulated run: a perception outage from 1.0 to 2.5 seconds and collision at 4.2 seconds. Braking works from any decision up to 2.5 seconds; a fork that brakes at 2.5 seconds ends with no collision. Outage Alarm fires Last moment braking works Collision Brake at 2.5 s: no collision 0 s 1 s 2 s 3 s 4 s Braking from here avoids the collision Too late to brake
Simulated run, benchmark scene 1. A 1.5 s perception outage hides a merging car. Braking from any decision up to 2.5 s avoids the collision; from 3.0 s even an emergency brake fails. The failure persists when the other car replays its recorded path, so it is the planner’s, not the simulator’s.

The last decision before a crash is usually the wrong one to blame.

The last moment braking still worked came a median 0.9 s earlier. A crash log records what happened. It does not say when the outcome was decided.

12% of crashes were prevented by braking at the last decision before impact.

Three questions every failed run should answer.

Checkpoint the run. Fork from any decision. Compare outcomes.

  1. When did it become unavoidable?

    Nemesis bisects saved checkpoints to find the last decision from which an emergency brake still avoids the collision.

    467 of 468 searches matched exhaustive search, at about three replays of compute each.

  2. What would have saved it?

    It forks the run at each decision with alternate actions, from a single brake to a hand-over to a reference planner, and reports which ones avoid the crash.

    41% of failures had no single decision to blame, so Nemesis reports the window, not one culprit.

  3. Was it the planner or the simulator?

    It reruns the failure with the other car following its real recorded path instead of the traffic model.

    1 in 4 failures disappeared: 183 of 722 were simulator artifacts.

Every number here comes from a published benchmark.

468 planner failures across 20 scenes from the Waymo Open Motion Dataset, simulated in Waymax with a reactive test planner and injected faults. Every counterfactual was simulated, not estimated.

In development: exporting a failure as a regression test, and running Nemesis against your own simulator.

Read the research

Bring us your hardest failure.

Send one failed simulation run. We’ll show you when it became unavoidable, what would have saved it, and whether it was real. We’re working with a small number of design partners.

Bring us a failed run