Safety-Critical-Scenario-Benchmark-for-AD

🆕 Updates

• 2025-01-06: We've launched the safety-critical scenario benchmark for autonomous driving! 🚗

---

Table of Contents:

• Table of Contents:
• 🌍Introduction of the Safety2Drive
• 🍃 Running the Autonomous Driving Scenario
• 🔧 Intelligent Perception Tasks
  • 1. Camera-based Object Recognition
  • 2. Lidar-based Object Recognition
  • 3. Depth Estimation
  • 4. Lane Line Recognition
• ⚡Adversarial Attack Scenarios
  • 1. Pixel-based Digital Attacks
  • 2. Patch-based Digital/Physical Attacks
  • 3. Camouflage-based Physical Attacks
  • 4. Backdoor Attacks
• 🏁 Leaderboard of Driving Agents
  • 1. Autopilot
  • 2. Garage
  • 3. Interfuser
• 📌 Roadmap
• 🔍 Safety-Critical Scenario Generation algorithms (Stay Tuned)
• [Acknowledgments]
• [📝 License]
• [🔖 Citation]

🌍 Introduction of the Safety2Drive

• The benchmark are in the standard OpenSCENARIO format, including 70 carefully designed standard regulatory scenarios for functional test, and support for 30 adversarial attack algorithms. Each of these 70 functional test items can be generalized to multiple scenarios. Theoretically, the benchmark contains an infinite number of scenarios.

Subset	Number	File List
Functional Test	70	xosc file

Note that the full scenario file is in Scene_leaderboard.

🍃 Running the Autonomous Driving Scenario

Download and setup CARLA

• CARLA 0.9.15

      mkdir carla
      cd carla
      wget https://carla-releases.s3.us-east-005.backblazeb2.com/Linux/CARLA_0.9.15.tar.gz
      tar -xvf CARLA_0.9.15.tar.gz
      cd Import && wget https://carla-releases.s3.us-east-005.backblazeb2.com/Linux/AdditionalMaps_0.9.15.tar.gz
      cd .. && bash ImportAssets.sh
      export CARLA_ROOT=YOUR_CARLA_PATH
      echo "$CARLA_ROOT/PythonAPI/carla/dist/carla-0.9.15-py3.7-linux-x86_64.egg" >> YOUR_CONDA_PATH/envs/YOUR_CONDA_ENV_NAME/lib/python3.7/site-packages/carla.pth # python 3.8 also works well, please set YOUR_CONDA_PATH and YOUR_CONDA_ENV_NAME

Use the ScenarioRunner

---

Please take a look at the Getting started documentation.

Here are 5 driving scenarios

---

Decelerating	Cut In	Cut Out Front	Pedestrian Crossing	Two Wheeler Riding
BEV	BEV	BEV	BEV	BEV

16 types of natural environmental corruption: Snow, Rain, Fog, Strong Sunlight, Gaussian Noise, Uniform Noise, Impulse Noise, Density Decrease, Cutout,LiDAR Crosstalk, Motion Blur, Local Density Decrease, Local Cutout, Local Gaussian Noise, Local Uniform Noise and Local Impluse Noise.

Weather-level corruptions

Sensor-level corruptions

Object-level corruptions

Here are 4 safety-critical scenarios

---

Sudden Pedestrian Crossing	Opposing Lane Pass	Lane Change With Cone	Two wheeler Retrograde

🔧 Intelligent Perception Tasks

Camera-based Object Recognition	Lidar-based Object Recognition

Depth Estimation	Lane Line Recognition
depth.mp4	line.mp4

⚡ Adversarial Attack Scenarios

1.Pixel-based Digital Attack

Without Digital Attack vs. With Digital Attack
Without Digital Attack
PGD Digital Attack

2.Patch-based Digital/Physical Attack

Without Patch Attack vs. With Patch Attack

Without Lane Line Attack vs. With Lane Line Attack

3.Camouflage-based Physical Attack

Without Camouflage Attack vs. With Camouflage Attack

4.Backdoor Attack

Backdoor Attack
backdoor.mp4

🏁 Leaderboard of Driving Agents

Closed-loop Evaluation Leaderboard

Build your agent

• Add your agent to leaderboard/team_code/your_agent.py & Link your model folder under the Safety2Drive directory.

      Safety2Drive\ 
        scenes\
        leaderboard\
          team_code\
            --> Please add your agent HEAR
        scenario_runner\
        tools\
        --> Please link your model folder HEAR

Autopilot

Lane Change With Cone	Sudden Pedestrian Crossing

Driving Agent	Scenarios	Collision	Complete Route	Driving Score
Autopilot	Lane Change With Cone	True	False	0.2746
Autopilot	Sudden Pedestrian Crossing	False	True	1.0

Garage

Lane Change With Cone	Sudden Pedestrian Crossing

Driving Agent	Scenarios	Collision	Complete Route	Driving Score
Garage	Lane Change With Cone	True	False	0.1785
Garage	Sudden Pedestrian Crossing	False	True	1.0

TransFuser

Lane Change With Cone	Sudden Pedestrian Crossing

Driving Agent	Scenarios	Collision	Complete Route	Driving Score
TransFuser	Lane Change With Cone	True	False	0.4225
TransFuser	Sudden Pedestrian Crossing	False	True	1.0

## 📌 Roadmap

• Demo Website Release

• V1.0 Release

  • Benchmark
  • Perception Task
  • Driving Agent Support

• V1.1 Release

  • Safety-Critical Scenario Generation algorithms