Jev with Foresight

Safe Language-Guided Autonomous Driving via a Language World Model

Yansong QuJiancong ChenSamuel Labi

Lyles School of Civil and Construction Engineering, Purdue University

Closed-loop episodes from the experiments, in simulated highway merging and in replayed NGSIM traffic (1 min 33 s).

26.0%→2.4%
Collision rate in simulated highway merging: the same decision model without and with foresight (500 car-episodes)
48.2%→2.8%
Collision rate in replayed real-world NGSIM traffic (500 episodes)
93%
of free-form command trials achieve the passenger's intent, including negated requests and requests in Chinese
0.25 s
per decision including foresight, well within the 0.5 s decision period

Abstract

Large language models understand passengers' driving commands, but as driving agents they reason in generated text for seconds and must infer the consequences of each action implicitly from the scene. We present Jev-AD, which pairs Jev, a typed decision model that answers predefined questions with probabilities instead of generating text, with a language world model that predicts for every candidate action how nearby vehicles move over the next two seconds. Code turns these predictions into foreseen consequences, which Jev compares instead of inferring them from coordinates. In simulated highway merging (500 car-episodes), this foresight lowers the same decision model's collision rate from 26.0% to 2.4% at 0.25 seconds per decision, and Jev-AD achieves the passenger's intent in 93% of free-form command trials. In 500 replayed real-world NGSIM episodes, foresight lowers the collision rate from 48.2% to 2.8%, and the world model, learned from this traffic, foresees 93% of dangerous actions.

Why foresight?

An LLM driving agent reads the scene as positions and speeds, and has to work out for itself what each action will lead to. In dense traffic this can fail.

Three panels of the same NGSIM case on I-80: an LLM-based agent turns left and collides, a world-model-based agent stays in its lane, and Jev-AD waits 1.5 s and then moves left safely.
A real NGSIM case on I-80: the passenger asks to move to the left lane while a car approaches in that lane at 18 m/s. (a) An LLM-based agent, here Jev with KoMA's scene text, turns left at once and collides 0.95 s later. (b) A world-model-based agent stays safe but cannot take the request. (c) Jev-AD's world model answers “COLLISION within 1.5 s” for turning left, so Jev-AD waits 1.5 s and then moves left.

How Jev-AD works

Jev is a typed decision model: each question fixes the type of its answer (a choice among named options, a score on a scale, or yes/no), and the model returns a probability for every possible answer instead of generating text, in about 0.13 s per call.

  1. Once per commandCommand grounding. Jev answers four typed questions about the passenger's words: is the request unsafe, is it off-topic, how aggressive is it, and which maneuver does it ask for. Unsafe requests are refused.
  2. Every 0.5 sForesight. A small language world model reads the last 2 s as text and predicts, for each candidate action, how up to nine nearby vehicles move over the next 2 s, including how they react to the AV. Code turns the trajectories into consequences stated in words: whether and when the AV would collide, the minimum time to collision, and its final lane and speed.
  3. Every 0.5 sDecision. One Jev call reads the command and the foreseen consequences and chooses the safe action that best matches the passenger's request.
Framework diagram: (a) command grounding with four typed questions, (b) the language world model predicting each candidate action's outcome, (c) one Jev call choosing the action.
Jev-AD on an example scene (inputs and outputs abbreviated). (a) Jev grounds the command once with four typed questions. (b) The language world model predicts, for each candidate action, the controls of up to nine vehicles over 2 s; a bicycle model integrates them into trajectories, and code turns these into consequences. (c) One Jev call chooses Idle under a hurry command, because turning left is foreseen to collide and accelerating to leave 0.8 s to the ramp end.

Results

Foresight is what removes most collisions. With the same decision model and the same command grounding, KoMA's scene text leads to collisions in 26.0% of the car-episodes and foresight from the language world model to 2.4%. Even foresight from constant-velocity prediction lowers the rate to 2.8%, so the gain comes from the foresight itself. The world model makes the foresight more accurate: it raises fewer false alarms, and because it foresees that the car behind brakes, the car stops promptly when the passenger asks.

Scatter plot of time per decision against collision rate on the merge scenario; Jev-AD has few collisions at about 0.25 s per decision.
Speed and collisions. Time per decision and collision rate on the merge scenario. Only Jev-AD both collides rarely and decides well within the 0.5 s decision period.
Line chart of mean speed under seven graded commands, rising from stop to drive as fast as you possibly can.
Graded commands. Mean speed under seven commands, from “Stop the car” to “Drive as fast as you possibly can” (merge, 30 car-episodes each; bars: 95% bootstrap confidence intervals).

Replayed real-world traffic (NGSIM)

We replay recorded freeway traffic from NGSIM (US-101 and I-80). One recorded vehicle becomes the AV; every other vehicle follows its recording until the AV, or a vehicle that already reacts, is directly ahead of it and too close, and from then on IDM drives it. On average, 97% of the vehicles around the AV follow their recordings at a decision. The world model is trained on this traffic and foresees 93% of the dangerous actions, with false alarms for 10% of the safe ones.

Top: Jev-AD's path and the recorded driver's path drawn on an aerial photo of I-80. Bottom: time-space diagram of recorded trajectories coloured by speed, showing stop-and-go waves.
One test on I-80 in stop-and-go traffic. (a) Jev-AD's path (blue) and the path of the driver recorded in the AV's place (dashed), drawn on an aerial photo of the site; the photo comes from a separate survey, so its cars are not part of the test. (b) Recorded trajectories in the AV's starting lane, colored by speed; the box marks the part shown in (a).
Three NGSIM test states in which the world model's predicted trajectories (dashed) closely follow the replayed trajectories (solid) over 2 s.
What the world model foresees. Predictions for every tracked vehicle in three typical NGSIM test states: replay (solid) and prediction (dashed) over 2 s, with positions after 1 s (dots) and 2 s (boxes).

All results come from simulation and from replayed recorded traffic; no test was run on a vehicle. The paper lists the limitations.

Citation

@article{qu2026jevforesight,
  title   = {Jev with Foresight: Safe Language-Guided Autonomous
             Driving via a Language World Model},
  author  = {Qu, Yansong and Chen, Jiancong and Labi, Samuel},
  journal = {arXiv preprint},
  year    = {2026}
}