Blog post: Situation hyperspace - using a simulated world to obtain situation coverage for AV safety assurance
The societal benefits of autonomous vehicles (AVs) — those that operate fully without a driver — have never been clearer than they are right now. The delivery of essential items, such as medical supplies and food, with limited human contact, would serve us well in the current Coronavirus climate. While the development of AV performance is key, we must not neglect the assurance of their safety.
Why safety standards are not enough
Current standards for the safety of road vehicles (e.g. ISO/PAS 21448 and ISO 26262) do not map well to AVs (SAE level 3 and above) because they assume the presence of a human driver to take over in an emergency.
There are emerging standards that don’t assume human drivers, with perhaps the most prominent being UL 4600. That standard focuses on developing a valid safety case for an autonomous product. But it is not, and cannot be, fully prescriptive — we still need to do bespoke work to build the safety case for the AV ourselves.
Our approach to the safety assurance of AVs
As UL-4600 outlines, a valid safety case for an AV must be based on evidence from a combination of the following techniques:
- Formal analysis
- Functional testing (black box testing)
- Structural testing (white box testing)
- Closed course testing
- Public road testing
Our current research as part of the EU project Safer Autonomous Systems (SAS) focuses on the simulation-based testing for verification and validation (V&V) of AVs.
We are testing the AV in simulation, considering the AV as a black-box or in some cases as a grey-box. We will test the AV with challenging situations (edge cases included) with a feedback mechanism/metric called “situation coverage”.
Situation coverage is a coverage criterion that will let our automated test generation algorithm know which situations have been covered in our testing and which have yet to be tested. It will also highlight those situations that would cause the AV to fail: these will be given a higher score so that similar situations are searched more by our automated test generation algorithm. We are also looking into using methods such as fault injection in conjunction with situation coverage-based testing.
Challenges in our proposed situation coverage approach
One of the most challenging tasks with this approach is to design the situation hyperspace i.e. the simulated world around the AV. It is from this hyperspace that our automated test generation algorithm systematically selects challenging situations that a real AV would face.
For this task, we are examining existing AV world ontologies closely. These include the works of Krzysztof Czarnecki in designing operational world models for defining the Operational Design Domain (ODD) and the National Highway Traffic Safety Administration ODD for AV.
Our situation hyperspace is different from these ontologies as it has to be structured in a way that our automated test generation algorithm can easily understand and navigate to find and implement valid real-world situations.
These situations challenge the AV, possibly causing it to fail. The algorithm gives a situation coverage metric and a pass or fail score. The algorithm continues to navigate the situation hyperspace, finding more challenging situations for the AV to test it thoroughly.
In the end, some confidence measure of our testing can be provided to the public: a situation coverage metric along with the pass/fail rate of the AV under test.
Practical guidance
The safety assurance of AVs is the key to unlocking their benefits. To develop the safety case that a standard such as UL-4600 requires, we must be confident in every element of assurance, from road tests and simulation-based V&V to hardware verification and micro-chip level verification.
Our proposed situation coverage-based testing for V&V of AVs is a part of this safety assurance. As our research continues and we obtain confidence measures that can be used with the public, regulators, and others, we’ll provide practical guidance in the AAIP Body of Knowledge for all to access and use.
Zaid Tahir
AV safety researcher, AAIP
Connect on LinkedIn
Acknowledgements
This research has been funded by the European Union’s EU Framework Programme for Research and Innovation Horizon 2020 under Grant Agreement №812.788.