VL-SAFE: Vision-Language Guided Safety-Aware Reinforcement Learning with World Models for Autonomous Driving

(a) Offline RL, removes the need for online exploration entirely by learning from pre-collected driving data, thus avoiding the safety concerns of real-time trial-and-error; but it still fails occasionally in online testing due to no safety constraints.

(b) World models can reduce interactions by enabling policy learning through imagined rollouts, this can largely reduce risky actions and improve sample efficiency; but world models can still make risky actions at the initial training stage.

(c) Safe RL methods explicitly incorporate cost constraints or risk-sensitive objectives into the learning process to restrict unsafe behaviors and ensure long-term safety; safe RL might learn a too conservative policy and unable to generalize due to the lack of semantic understanding of the whole context.

(d) It is intuitive to combine these methods together to complement each other's strengths and weaknesses. However, more importantly, at the core of these challenges lies a fundamental question: how can we identify risky states, semantically understand “safety”, and guide policy learning accordingly?

We introduce our motivation starting from an intuitive life example: imagine an autonomous vehicle approaching a stopped school bus on a multi-lane road, where several children are walking across the street toward the sidewalk. The school bus has its STOP sign extended, indicating that all surrounding vehicles must come to a complete stop.

A classical RL-based or offline-trained agent may continue driving or only slightly slow down, as no collision has occurred and the reward function still favors progress. A world model-based agent may also ignore the school bus and its STOP sign during imagined rollouts, failing to distinguish the risk embedded in this visual context. A traditional safe RL policy may behave unpredictably: it may stop in some cases, but often fails to recognize the raised STOP sign and the presence of children, especially if such visual cues were not explicitly encoded in the cost function or training data. This can lead to unsafe decisions or over-generalized conservativeness, such as treating harmless objects like parked vehicles or cones as equivalent threats.

In contrast, a VLM-guided policy can semantically understand the scene: the raised STOP sign on the school bus and the crossing children clearly indicate a high-risk situation. Even in the absence of a collision, the VLM assigns a low safety score based on visual semantics, guiding the agent to stop proactively and yield to the pedestrians. This enables the agent to maintain appropriate caution in dangerous scenarios without falling into the trap of indiscriminate conservativeness.

This example highlights the advantage of semantic safety reasoning: it enables agents to make early, context-aware, and proportionate decisions, ultimately allowing them to act safely across varied scenarios.

The proposed method VL-SAFE has two phases, where the first phase generates ground truth safety estimations using CLIPs for each state in offline dataset collected by expert agent, and the second stage will learn a safety-aware world model to generate imagined rollouts along with predicted safety estimations for actor-critic learning.

In our approach, we similarly assign weights to each state-action pair in the imagined rollouts. The weight term combines both reward and cost advantages, and is modulated by a VLM-derived safety probability, which dynamically balances the influence of reward-seeking and cost-avoidance. The intuition is that the agent should prioritize reward maximization in safe conditions, and focus on cost minimization when encountering potentially risky situations, where the timing and degree of this tradeoff is guided by the semantic understanding provided by the VLM.