1.2.1 Considering human/machine interaction

Air Traffic Control (ATC) and Air Traffic Management (ATM) take place within a complex socio-technical environment which is expected to get busier in the future as UK airspace will need to cater for new airspace users whilst continuing to keep the skies safe. It is expected  that a step change would be required from current decision support tools to also having tools with decision-making capabilities which can facilitate human-machine collaboration. The  introduction of automated decision-making tools has the potential to improve the efficiency and effectiveness and the overall management of the airspace. However, existing approaches to design, develop, test, validate concepts for such tools and assure their safety for use in real-time operations often require years of development, real-time simulation in dedicated physical environments with skilled personnel, specialised facilities, and equipment which are resource and time intensive to set up and deliver. Current approaches also limit the opportunities for stakeholders and end users to be involved at an early stage of the concept development of future tools.

Prototyping techniques such as “Wizard of Oz” provide an opportunity to gain insight into human-machine interaction much earlier in the development process. Wizard of Oz (WoOZ) is a prototyping method that has been used within interaction design to test the interaction between a human user and a machine. In WoOZ prototyping, the functionality of the machine is entirely or partially controlled by another human (typically referred to as the “wizard”). By using WoOZ prototypes and experiments, it is possible to gain feedback from users and stakeholders without having to invest significant resources, time and money into developing a fully functional product or service. By adopting this approach, data could be collected from users before having to develop the actual technologies that underlie automated decision-making. Thus, there is the potential to spot problems early on, including the identification of possible hazards, and to then use this information to feed into the next development iteration. 

Considerations for human/machine interaction

The WIZARD project involved interviews with air-traffic controllers (ATCOs) and ATC stakeholders, the development of multiple prototypes at various levels of fidelity, informal open feedback sessions during the prototyping journey, and a validation activity involving two interactive digital prototypes where a wizard was used to simulate how an automated decision-making tool would work during interactions with users. Based on the project findings, we highlight the following areas to consider in relation to supporting human/machine interaction within the context of automated decision-making and ATC:

1. The need for explicit roles and responsibilities: When designing for human machine teaming and different levels of automation, it is vital that the division of responsibilities between the human controller and automated decision-making tool are clear. For this to happen, ATCOs need to be able to be confident that the tool will operate safely so they can focus on the information they need to carry out their own tasks. There is also an open question here that needs to be considered in terms of who is liable if the automated tool is involved in an incident. 
2. Supporting and maintaining situational awareness: Situational awareness is particularly important to consider in relation to any handover between a human controller and automated tool as well as for the overall human-machine team to have a shared mental model. While ATCOs want to understand how the tool is making decisions, the timing and amount of information provided is key. Too much information at the wrong time could be distracting or increase cognitive load, thus increasing potential risk to safety. There could also be situations where it is not necessary for ATCOs to be informed about the actions of the tool e.g., if this information is not required to maintain their situational awareness.
3. Training and acceptance: The concept of automated decision-making tools is still very new in the context of ATC. Thus, while the level of automation in ATC is likely to increase over time, training will be a necessary part of supporting new ways of working. In addition, training will also be important for helping ATCOs understand and become familiar with automated decision-making technologies. Introducing automated decision making concepts early and in small, incremental steps may help to increase acceptance and build trust.

Guidance for using Wizard of OZ to explore human/machine interaction

Overall, the feedback regarding the use of WoOZ prototyping was very positive, with participants recognising benefits such as speeding up the development process (saving both time and money) and the way in which the prototypes were able to support communication with a range of diverse stakeholders. They also saw the potential of the approach to be applied across different projects, use cases and as a way to explore safety aspects e.g., by comparing and contrasting how users behave in different scenarios and asking them to explain their actions. It was also suggested the WoOZ techniques could be used to inform a preliminary safety assessment and early identification of potential key hazards.

Based on our experience of employing WoOZ, we suggest the following guidance for how to approach this technique and get the most out of it:

1. Planning and project management: Some time needs to be allocated to selecting an appropriate use case that will allow for exploration of human/machine interaction. An agile management approach which utilised sprint cycles allowed for development work to be managed in a more dynamic way and provided support for pivoting, where necessary, as clarity and understanding evolved.
2. Iterative prototyping: As part of a rapid prototyping approach, we would recommend starting with storyboarding exercises before building low fidelity prototypes (e.g., cardboard, and whiteboards) and then working up to higher levels of fidelity (involving digital tools, e.g., Microsoft Visio and other suitable digital prototyping//interaction design tools). We found it particularly valuable to concurrently develop more than one prototype as this allowed us to explore different approaches and to consider different ways in which automation could be implemented. As part of an iterative approach, it is important to solicit feedback from a range of stakeholders during development (e.g., through informal open feedback sessions, and more formal evaluations).
3. Setting up WoOZ studies: It is important to decide who the wizards will be (i.e., to consider what expertise they will require), and to establish what actions they will need to carry out to simulate automated decision-making in different scenarios as part of developing a WoOZ prototype. As part of the validation activity, users need to be briefed and given clear instructions regarding what they are being asked to do and what their role is. 
4. User awareness of the Wizard: During validation sessions, wizards do not necessarily need to be in a different room to users, but should be ‘hidden’ during the interaction (e.g., through use of screens) so that the wizards don’t distract the users. Although all participants were given an introduction to the project at the start of the session, including an explanation of what WoOZ prototyping involved, many participants appeared to be unaware of the wizards (e.g., they forgot about them) or assumed that the tool did include actual (rather than simulated) automated decision-making. Ethically, it is important to debrief all participants after the session to explain the simulated aspect of the interaction and ensure they do not leave the session with any misunderstandings.
5. Dealing with validation data: The WoOz approach can yield a large amount of data including observation notes about how users interacted with the tools, session recordings (audio and/or video), and further interview data where users discuss their reactions in more depth. Thought needs to be given both to how to handle data collection and in terms of timescales for data analysis. The data can be used to feed into defining and refining user and safety requirements as part of ongoing development. There may also be opportunities to explore complementary methods for gathering feedback and input in more dynamic ways from users in real-time e.g., through survey/polling tools, or using in-built analytics (though care needs to be taken not to disrupt the tasks they are engaging in).