2.2.1.2 Defining understanding requirements

Maritime guidance is highlighted

Authors: ALADDIN demonstrator project

Overview of approach

Understanding is the key to enable RAS to interpret the data received from sensors about the system’s states and operational environments and make informed decisions on actions to be taken to reach desired goals. Therefore, when defining the understanding requirements for an autonomous system, we need to consider:

1. Consideration of goals and associated system states and features
2. Understanding the state of system
3. Understanding the operating environment
4. Associated metadata

In this Body of Knowledge entry, we consider the understanding requirements of autonomous fault diagnostics systems and in particular for MAS as the domain-specific example studied in the ALADDIN programme, with this content highlighted in blue throughout. However, the general considerations presented here share much in common with other robotic domains and thus the key concepts should be transferable.

1. Consideration of goals and associated system states and features

The objectives of RAS operations form the success criteria and the RAS understanding requirement to complete its mission. Furthermore, the goals also influence the system’s capabilities and sensing abilities (see Section 2.2.1.1), thus affecting its associated system states and available features that can effectively represent such states. Additionally, real-world environments (e.g. maritime) can be highly dynamic and can pose various influences on a RAS such as an autonomous underwater vehicle (AUV). In many cases, the complex environment can lead to observations that vary significantly throughout a standard operation and such variations can be challenging to interpret and understand. Therefore, the understanding of system states needs to be thorough to capture normal variances imposed by the RAS itself and its dynamic operating environment to be able to distinguish anomalous behaviours and faulty states. We can also deduce indirect features from direct sensing information using techniques such as system identification, to better suggest the system operating state. From the perspective of fault diagnostics, a well-established understanding of normal operating state can minimise the rate of false positives.

AUV example: As the AUV example discussed in Section 2.2.1.1, an AUV is deployed to undertake a bathymetric survey of the seafloor, operating at a depth of 6000 m below the sea-surface. The understanding of this deliverable should be considered first, for example:
• What system parameters are to be understood? The system state parameters can include signals measured by AUV’s onboard system sensors as well as provided by external data sources.
• Note that the system states/features can either be scalar values or multi-dimensional matrices, depending on the sensor types. How to interpret complex sensing information provided by systems such as cameras?
• What thresholds are to be set for these system state parameters? For instance, how much deviation of AUV depth is permitted for the AUV to achieve a satisfactory survey?
• Are the data resolutions good enough to establish an understanding of the AUV’s states and its environments?
• Are there any constraints from the environment (e.g. bathymetric features or currents affecting AUV localisation) and minimum requirement of water clarity that may influence the quality of the survey data and impose safety hazards to the AUV and its operating environment?
• Does the AUV and/or the operator have accurate estimations of power consumptions and remaining battery state of charge, such that the mission can be accomplished with the AUV recovered safely?
• What are the features that can be deduced to better represent the vehicle’s status? For instance, drag and lift coefficients can be determined for the AUV using its direct sensing, to indicate potential biofouling.

2. Understanding the state of the system itself

The robotic system requires an understanding of its own system state to operate safely and as intended to achieve the defined goals. In our AUV example, such an understanding can be established from appropriate interpretations of the signals from the AUV’s onboard sensors and virtual features deduced from direct sensing. As in Section 2.2.1.1, the understanding of the RAS state can be further developed in the following aspects:

• critical items list and associated safety functions,
• run-time monitoring to react to events or detect developing faults, and
• long-term reliability analyses.

Approaches established in reliability engineering, such as Failure Mode and Effect Analysis (FMEA) [1], can be applied to outline the elements of the RAS whose failure modes may lead to a degraded system performance and/or safety hazards to the RAS and its operating environment. It is critical to understand the potential consequences associated with different failure modes and to define appropriate mitigation actions to minimise risks for each element of the critical items list. The thresholds of determining critical failures should be clearly indicated to the RAS designers/operators and integrated with RAS internal safety functions.

The state sensing data is also used for monitoring the system during an operation to react to events or developing faults (see the ALADDIN project). It is essential to understand the underlying characteristics presented in the sensing data which can reflect the RAS operating status (i.e. whether it is in normal operation or has developed or is developing a certain type of fault - either known or unknown). Different approaches including model-based and data-driven methods can be applied to infer the RAS operating state. Model-based methods are more suitable for systems where available data is limited and an accurate model exists for the RAS. For complex systems, data-driven approaches such as deep learning may show significant improvements in understanding the RAS states where significant prior data exists [2].

Additionally, understanding derived from the sensing and perception of a RAS system can be applied to studying the long-term reliability of a RAS. One example for doing this is by monitoring the shift in performance of a system (both software and hardware) as well as deviation in accuracy of the models or algorithms used for the RAS perception system over time. Long-term reliability analyses can also inform future designs and safety assurance of RAS.

AUV example: For AUV applications, we need to define the critical items list with clearly defined safety functions, based on the comprehensive understanding of the failure modes and their consequences. For instance, immediate recovery measures need to be taken when the AUV’s battery state of charge is critically low, where continuing the operation would put the AUV at an increased risk of being lost at sea. For run-time monitoring, the changes in drag and lift coefficients may suggested biofouling that can increase the AUV’s power consumptions, and consequently lead to reduced deployment time. For long-term reliability analyses, the gradually shifting battery open-circuit voltage vs state of charge curve over the AUV’s lifetime can suggest when the AUV battery has reached its end-of-life.

3. Understanding the operating environment

In addition to understanding the system’s own state, understanding of its operating environment is also critical. Understanding the operating environment can be established through data from sensors onboard the RAS and external data sources (e.g. satellite observations, data from nearby platforms, either provided before the deployment or sent to the RAS during operation).

It is important to extract the key information and features contained in the raw measurements of the operating environment. A reliable interpretation or understanding of this information by the RAS or its human operator helps in informed decision-making for the next steps. For instance, terrain-aided navigation and Simultaneous Localization and Mapping (SLAM) sensing can locate the RAS within its environment, which can be regarded as a basis for the adjustments of the AUV’s actuators.

As indicated in Section 2.2.1.1, some payload sensors installed onboard a RAS can also indicate the states of the RAS and its environment. For instance, the temperature and salinity measurements by an AUV can not only be used to calibrate the AUV’s other sensors, but to suggest the status of its operating environment. Therefore, it is important to identify the areas of overlaps where better understanding of its operating environment can be established.

From the perspective of fault-diagnostics, understanding the operating environment can potentially improve the fault-diagnostics accuracy on different faults. For instance, a high seawater temperature and increasing drag coefficient could suggest biofouling. However, it should be noted it can be challenging for human operators to identify some fault types with the abnormal patterns distributed over multiple sensors readings in low magnitudes. Deep learning fault diagnostics approaches can potentially provide an improved understanding of the operating environment where complex measurements and features exist, hence, to achieve enhanced assurance of safety for the RAS.

AUV example: An AUV was deployed to measure oceanographic parameters such as chlorophyll levels, temperature, and salinity. However, during the deployment the operator had noticed that a few signals, including the battery voltage, state of charge and rudder angle, showed unusual patterns. Further investigation suggested that the AUV had experienced strong disturbances due to transverse ocean currents. The survey quality could have been compromised due the disturbances from the operating environment of the AUV. If the operator had never seen such an anomalous situation, it could be difficult to promptly identify the root cause.

4. Associated metadata

In addition to metadata detailed in Section 2.2.1.1, the following datasets are useful for development of fault-diagnostics tools for the safety assurance of RAS:

• ALFA: A Dataset for Unmanned Aerial Vehicle (UAV) Fault and Anomaly Detection [3].
• A Fault Detection Data Set for Performance Bugs in Component-Based Robotic Systems [4].
• Robot Execution Failures Data Set [5].
• UTIAS Multi-Robot Cooperative Localization and Mapping Dataset [6].
• The Zurich Urban Micro Aerial Vehicle Dataset [7].
• AUV datasets: the SOCIB datasets of AUVs [8].

Summary

In this Body of Knowledge entry, we define the understanding requirements for RAS from two perspectives (i.e. understanding the system state and the operating environment). We hypothesise that deep learning fault-diagnostics methods can be a solution for a RAS to understand the normal states and faults. Our work [2] conceptually verifies that, with accurate labelling of the training dataset, supervised learning can achieve high fault diagnostics accuracy for AUV. However, in practice, complete labelling can be prohibitively expensive or time-consuming. It is crucial to develop methods such as semi-supervised learning that can detect the fault types for RAS with partially labelled training datasets.

References

[1] IEC, IEC 60812:2018 Failure modes and effects analysis (FMEA and FMECA), International Electrotechnical Commission, 2021.
[2] P. Wu, C. A. Harris, G. Salavasidis, I. Kamarudzaman, A. B. Phillips, G. Thomas and E. Anderlini, “Anomaly Detection and Fault Diagnostics for Underwater Gliders Using Deep Learning,” in IEEE/OES OCEANS 2021, accepted, San Diego, 2021.
[3] A. Keipour, M. Mousaei and S. Scherer, “ALFA: A Dataset for UAV Fault and Anomaly Detection,” Carnegie Mellon University, 2020. [Online]. Available: http://theairlab.org/alfa-dataset/. [Accessed 10 September 2021].
[4] J. Wienke and S. Wrede, “A Fault Detection Data Set for Performance Bugs in Component-Based Robotic Systems,” 2016. [Online]. Available: https://doi.org/10.4119/unibi/2900912. [Accessed 10 September 2021].
[5] L. S. Lopes and L. M. Camarinha-Matos, “Robot Execution Failures Data Set,” Universidade Nova de Lisboa, 1999. [Online]. Available: https://archive.ics.uci.edu/ml/datasets/Robot+Execution+Failures. [Accessed 9 September 2021].
[6] K. Y. K. Leung, Y. Halpern, T. D. Barfoot and H. H. T. Liu, “UTIAS Multi-Robot Cooperative Localization and Mapping Dataset,” 2011. [Online]. Available: http://asrl.utias.utoronto.ca/datasets/mrclam/. [Accessed 8 September 2021].
[7] A. Majdik, C. Till and D. Scaramuzza, “The Zurich Urban Micro Aerial Vehicle Dataset,” 2017. [Online]. Available: http://rpg.ifi.uzh.ch/zurichmavdataset.html. [Accessed 30 August 2021].
[8] SOCIB, “SOCIB Balearic Islands Coastal Observing and Forecasting System,” 2021. [Online]. Available: https://thredds.socib.es/thredds/catalog/auv/glider/catalog.html. [Accessed 1 August 2021].