Studying at York
Sound Interactions in the Metaverse - ELE00178M
- Department: Electronic Engineering
- Credit value: 20 credits
- Credit level: M
- Academic year of delivery: 2023-24
- See module specification for other years: 2024-25
Module summary
This module covers audio for virtual and augmented reality. You will develop an understanding of how immersive and interactive audio can be implemented into multiple applications for the Metaverse, including entertainment, healthcare, education/training and research. You will gain a knowledge of developing immersive and interactive digital environments by using 3D game development engines which can be experienced utilising virtual and augmented reality technology. Furthermore, you will learn specialised research techniques to record user behaviour and responses to spatial sound within extended realities that can be utilised in multidisciplinary research. The module also covers current research in this field within the AudioLab and will prepare students who undertake their final MSc project in this domain to the state of the art in audio for the metaverse.
Module will run
| Occurrence | Teaching period |
|---|---|
| A | Semester 2 2023-24 |
Module aims
Subject content aims:
- To give students an understanding of how to develop interactive and immersive audio and visual environments using industry standard game engine software and technology
- To give students an understanding of object-oriented programming techniques used within industry and research
- To extend students’ understanding of spatial audio production within immersive audio and visual environments using industry standard audio engine software
- To give students working technical knowledge of using audio effects processing to support immersive and interactive environments
- To give students and understanding of how spatial audio can be used to support and enhance software applications within the entertainment, healthcare and training / simulation industries
- To extend students’ knowledge base of audio research techniques. Targeting mixed reality applications and multi-disciplinary experiments in the perception and/or effect of spatial sound
Graduate skills aims:
- To give students the understanding of the requirements of creating an interactive piece of software based upon a detailed design document.
- To develop students' understanding of mixed reality technologies in order to produce novel research software applications.
- To apply critical research thinking to multidisciplinary research.
- Give a professional spoken presentation of research outcomes and data.
Module learning outcomes
Subject content learning outcomes
After successful completion of this module, students will be able to:
- Create immersive and interactive audio and visual environments for mixed reality technologies.
- Demonstrate a fundamental understanding of development within the Unity3D game engine and Visual Studio
- Show a working knowledge of object-oriented programming in C#
- Incorporate third party audio software into Unity3D to utilise higher order Ambisonics
- Deploy interactive environments to a range of mixed reality hardware using various software development kits
- Capture and export research data from mixed reality applications and hardware
- Demonstrate their understanding of the use of spatial audio in the gaming, healthcare, and training industries
- Analyse the psychological impact of spatial audio in mixed reality applications
Graduate skills learning outcomes
After successful completion of this module, students will be able to:
- Create pieces of software based upon a detailed design brief
- Present clear technical documents to support software development
- Demonstrate problem solving techniques
- Critically evaluate current research and their own work
Module content
Software development:
- Unity3D game engine development
- C# scripting language
- Object-oriented programming (OOP)
- Visual Studio integrated development environment
- Hardware specific software development kits (SDK)
- Audio engine integration
- Audio production and sound design techniques within audio engine software
- Multiple mixed reality hardware deployment
- Application of Google Resonance Ambisonic development SDK
- Research data capture using immersive technology hardware
Lecture and workshop plan:
- Lecture 1: Introduction to Immersive Audio in the Metaverse
- Lecture 2: The psychological response to immersive auditory environments
- Lecture 3: Spatial audio production techniques for immersive environments
- Lecture 4: Immersive Audio in Healthcare and Wellbeing
- Lecture 5: Immersive Audio in Training and Simulations
- Lecture 6: Object-oriented programming for mixed reality development
- Lecture 7: Mixed reality software design and architecture
- Lecture 8: Mixed reality research capture techniques
- Lecture 9: Mixed reality research analysis techniques
- Workshop 1: Introduction to Unity3D and Visual Studio – Hello world
- Workshop 2: Introduction to C# development
- Workshop 3: Building an interactive virtual environment
- Workshop 4: Utilising hardware SDK and deployment to hardware
- Workshop 5: Integration of external audio software
- Workshop 6: Audio production techniques in the external audio software
- Workshop 7: Extended Object-oriented programming
- Workshop 8: Software design techniques
- Workshop 9: Capturing and exporting data for research
- Workshop 10: Assignment app development
Indicative assessment
| Task | % of module mark |
|---|---|
| Essay/coursework | 75 |
| Oral presentation/seminar/exam | 25 |
Special assessment rules
None
Additional assessment information
There are 2 main parts to the assessment:
Extended Reality research software application and documentation - worth 75% of the
module mark
Students create a piece of software that can be used to assess audio within a multidisciplinary study. The application will be deployed to extended reality hardware on any available platform (e.g. PC, mobile). Assessment of the application is based upon its functionality, research data capturing and exporting techniques, application of spatial audio production with the audio engine, system architecture and application of C# programming techniques.
In support of the application, also write a detailed design document which resembles those used within the software development industry. The document must include the system functionality, timeline, goals, hardware deployment, target audience / industry, software and technology dependencies.
Presentation of Extended Reality application - worth 25% of the module mark
Students create a 15-minute spoken presentation which communicates the intended research study for the developed software application. This will include an outline of the system, research goals and data captured from the device in a dummy pilot study. Students are assessed on their research presentation skills and the quality of data captured during the dummy study.
Indicative reassessment
| Task | % of module mark |
|---|---|
| Essay/coursework | 75 |
| Oral presentation/seminar/exam | 25 |
Module feedback
'Feedback’ at a university level can be understood as any part of the learning process which is designed to guide your progress through your degree programme. We aim to help you reflect on your own learning and help you feel more clear about your progress through clarifying what is expected of you in both formative and summative assessments. A comprehensive guide to feedback and to forms of feedback is available in the Guide to Assessment Standards, Marking and Feedback.
The School of PET aims to provide some form of feedback on all formative and summative assessments that are carried out during the degree programme. In general, feedback on any written work/assignments undertaken will be sufficient so as to indicate the nature of the changes needed in order to improve the work. The School will endeavour to return all exam feedback within the timescale set out in the University's Policy on Assessment Feedback Turnaround Time. The School would normally expect to adhere to the times given, however, it is possible that exceptional circumstances may delay feedback. The School will endeavour to keep such delays to a minimum. Please note that any marks released are subject to ratification by the Board of Examiners and Senate. Meetings at the start/end of each term provide you with an opportunity to discuss and reflect with your supervisor on your overall performance to date.
Formative Feedback
- Regular workshops allow you to engage with the programming material and receive verbal help and feedback on your coding and design.
- Weekly quizzes on the module Wiki page help you to gain feedback on your understanding of the key module material covered in the lectures.
- Emails to the Module Coordinator with Questions / Comments will be answered as soon as possible.
- Questions can also be submitted at any time via the Question Box on the module Wiki page.
- A draft version of your assignment can be submitted to the module coordinator by the end of term, who will confirm whether this is all in the correct format, along with some general written feedback comments, and a series of recommendations for improvement.
Summative Feedback
You will receive a customised feedback sheet, detailing the mark breakdown against each section of the assessment criteria. Key areas of assessment will be - Software Application and Documentation: Programming Quality; Software Design; Report quality. Presentation of Extended Reality Application: Presentation Skills, quality of data, analysis of research data. In addition, you will receive feedback highlighting areas of improvement and future development.
Indicative reading
Acoustics and psychoacoustics / David M. Howard and James Angus. Howard, David M. (David Martin), 1956- Angus, Jamie
As the technology is changing rapidly, along with the most relevant documentation, a website will be provided with links to the latest code documentation and 3rd party tutorials.