Deepen AI

Deepen AI Introduces Physical AI Safety Assessment Program (PASAP) to Support AV Evaluation with NVIDIA Halos SAFE

By Deepen AI Team

BERLIN, DE — October 21, 2026 at GTC Berlin: Deepen AI, a provider of data infrastructure for Physical AI, today introduced the Physical AI Safety Assessment Program (PASAP), bringing data characterization and safety benchmarks to autonomous vehicle evaluation workflows using NVIDIA Halos SAFE (Safety Assurance Framework for Evaluation).PASAP helps developers assess the quality and coverage of their driving data, evaluate model behavior across challenging scenarios and trace results to specific safety requirements. Its initial driving benchmarks draw on Safety PoolTM, the scenario library Deepen AI co-developed with WMG at the University of Warwick, UK.
NVIDIA Halos is a full-stack safety system for building, evaluating and deploying safer autonomous vehicles and robots. Within this system, Halos SAFE supports repeatable AV safety evaluation and assessment of how closely simulation matches real-world testing.
PASAP contributes data characterization, safety benchmarks, Safety PoolTM scenarios, and configurable evaluators to these workflows. Developers can compare model versions under consistent test conditions, identify coverage gaps and evaluate performance against their own safety requirements. The resulting evidence supports safety assessments and helps teams prioritize further testing and model improvements.

Data Characterization

Autonomous driving policy and its associated safety metrics are only as valid, reliable, and accurate as the data used to train, test and measure performance. A calibration error can misplace LiDAR points by about 50 cm at a range 35 m[3], and the label errors in widely used public benchmarks and datasets average at least 3.3%, enough to change which model ranks and drastically skew safety performance[4]. PASAP characterizes four areas, each traceable to the source dataset:

  • Calibration Integrity scores multi-sensor extrinsic calibration without ground-truth parameters and flags drift across sessions.

  • Label Quality measures precision and recall by object class and distance for human labels and autolabels, reported in ASAM OpenLABEL, the standard Deepen AI helped author.

  • ODD Coverage compares collected data against the target environment and operating conditions for the system known as Operational Design Domain (ODD), using the ISO 34503 taxonomy Deepen AI co-authored, and records gaps, provenance and privacy processing.

  • Real-world Data Sourcing fills those gaps through Deepen Refinery partner fleets in six geographies, delivering data already characterized.

Safety Benchmarks and Scenarios

PASAP evaluates driving policies, including vision-language-action (VLA) world models, and Foundation models in closed loop simulation scenarios, where the model's own decisions change how each scenario unfolds. NVIDIA Halos SAFE technical brief calls for closed-loop simulation whenever the system's actions influence surrounding traffic[5]. Following ISO 34502, each scenario is defined once with parameter ranges for speeds, gaps and timing, to generate variation perturbations.Each closed loop simulation scenario run is scored by modular evaluators with a standard interface, so developers can plug in their own success criteria on top of the base safety metrics. PASAP safety evaluations include:

  • Time-to-collision (TTC) and near misses: an industry-wide standard safety measure, with 1.5 seconds set as a common safe operating threshold[6].

  • Collision avoidance: compared with a reference model of an attentive, unimpaired human driver[7].

  • Rule and regulatory compliance: for example, UN Regulation 157 lane-keeping requirements.

  • Simulation realism: Consistency of reported metrics and driving behaviour between simulated scenarios and real world driving[8, 9].

Flagged events go to expert review. Each run is stored as a verified record with two use-cases:

  • PASAP Benchmarks apply the same scenarios and evaluators to every model, so scores are directly comparable across models and versions.

  • Evaluations with NVIDIA Halos SAFE use the evaluators that match a developer's own safety requirements, for example no more than one false emergency brake per 10,000 hours, and test the results against those targets.

Failures feed denser scenario sampling and corrective data for post-training.

"A safety case is only as credible as the data and evaluations behind it," said Mohammad Musa, Founder and CEO of Deepen AI. "PASAP provides developers and regulators with a standardized framework to generate verifiable safety evidence for NVIDIA Halos SAFE, enabling clear, apples-to-apples performance comparisons across autonomous systems."

“Safety of AI powered autonomous vehicles should be evidenced and measured objectively. While autonomous vehicles are getting mainstream, a global safety benchmark still evades the industry beyond philosophical aspirations. Through our partnership with Deepen AI and NVIDIA, leveraging the Safety PoolTM platform, we are addressing this gap in our ecosystem.” - Professor Siddartha Khastgir, Head of Safe Autonomy, WMG at the University of Warwick, UK.

“Developers need to understand both how an autonomous vehicle performs and how credible the evidence is,” said Rishi Dhall, vice president of automotive at NVIDIA. “By bringing data characterization, safety benchmarks and Safety PoolTM scenarios to NVIDIA Halos SAFE workflows, Deepen AI helps developers expand evaluation coverage and generate traceable evidence to support their safety cases.”

PASAP will extend the same approach to robotics, assessing how robot policies, VLA models and world foundation models perceive, act and recover from failures. Details are at benchmark.deepen.ai.
About Deepen AIFounded in 2017 in Sunnyvale, California, Deepen AI provides data infrastructure for Physical AI, from multi-sensor calibration and annotation to data curation and real-world data through Deepen Refinery, to more than 100 automakers, suppliers, autonomous vehicle developers and robotics companies. Deepen is an author of ASAM OpenLABEL, a contributor to ASAM OpenSCENARIO and co-developed Safety PoolTM with WMG at the University of Warwick. Learn more at deepen.ai.Media Contact: info@deepen.ai

References

  1. N. Kalra and S. M. Paddock, Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?, RAND Corporation, 2016. https://www.rand.org/pubs/research_reports/RR1478.html

  • C. G. Northcutt, A. Athalye and J. Mueller, Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, NeurIPS 2021. https://arxiv.org/abs/2103.14749

Z. Yang et al., UniSim: A Neural Closed-Loop Sensor Simulator, CVPR 2023. https://waabi.ai/introducing-unisim-one-of-the-core-groundbreaking-technologies-powering-waabi-world/