From Black Box to Certifiable Maneuver: A Risk-Based Approval Approach for AI-Supported Autonomous Navigation using Explainable AI and Uncertainty Quantification
Nico Jungbauer, M.Sc., TKMS ATLAS ELEKTRONIK GmbH, Bremen
The transition towards Maritime Autonomous Surface Ships (MASS) poses significant challenges to the traditional certification landscape. Current regulatory frameworks, such as SOLAS, are predicated on human cognition and presence, creating a gap for fully autonomous operations where digital systems must guarantee equivalent safety in situational awareness and collision avoidance. A critical hurdle is the approval of stochastic Artificial Intelligence (AI) systems, particularly Deep Learning models used for computer vision and object classification. The established standard for system safety, MIL-STD-882E, is designed for deterministic software and does not yet provide a sufficient framework for the probabilistic nature of machine learning. Comprehensive extensions are only expected in future revisions (e.g., MIL-STD-882F). Consequently, there is an urgent need for a method to map the non-deterministic behavior of AI components onto the strict, deterministic requirements of maritime safety cases.
This presentation proposes a methodological approach for a risk-based approval process at the vessel level, specifically addressing the safety gaps of "Black Box" AI models in the context of autonomous maneuvering.
First, the approach addresses Hazard Identification starting with a system-level Fault Tree Analysis (FTA). However, conventional FTA faces a critical limitation when applied to deep neural networks: the "Black Box" nature of the model prevents the definition of specific basic events or root causes for perception failures. To resolve this opacity, Explainable AI (XAI) is introduced as a necessary investigative layer. By employing XAI methods, the decision-making basis of the perception algorithm becomes transparent, revealing whether the AI classifies a ship based on the hull or merely water texture. Consequently, XAI enables the decomposition of a generic "AI Error" into concrete, systematic weaknesses, thereby allowing for a granular and meaningful FTA.
The second aspect, critical for operational safety, is the runtime monitoring of the Operational Design Domain (ODD) aligned with requirements such as DNV-CG-0264. An autonomous maneuvering system must recognize when it reaches the limits of its specified capabilities. For this purpose, the application of Uncertainty Quantification (UQ) is demonstrated. The focus lies on distinguishing between aleatoric uncertainty (inherent sensor noise) and epistemic uncertainty (model ignorance). The paper demonstrates how UQ metrics can robustly detect Out-of-Distribution (OOD) situations, meaning, scenarios not included in the training dataset. In the context of maneuvering, this UQ metric acts as a dynamic safety layer: if uncertainty regarding a detected object rises, the path planning algorithm can automatically adjust safety margins or trigger "Minimum Risk Conditions" (MRC), ensuring the vessel does not execute unsafe maneuvers based on low-confidence predictions.
The presentation demonstrates how combining XAI for design validation and UQ for operational monitoring paves a technical path for the approval of autonomous maneuvering systems, bridging the gap between data-driven innovation and the rigorous safety standards of the maritime industry.
