Today, GameSoundCon features a guest writer on our blog: Meet Scott Looney from Game Audio Institute. Scott is a passionate artist, soundsmith, educator, and curriculum developer who has been helping students understand the basic concepts and practices behind the creation of content for interactive media and games for over fifteen years. Having taught interactive online audio courses at the Academy Of Art University, Ex’pression College, Cogswell College, Pyramind Training, UC SantaCruz, City College SF, and SF State University, Scott has now developed a new online course about Audio for XR.
Being a game audio professional means keeping up with the latest technologies.
You’ve likely heard lots of fancy new acronyms like VR, AR, MR and most recently ‘XR’ describing the fields of Virtual, Augmented, Mixed and Extended or Expanded Reality (nobody agrees on what ‘XR’ means currently). Let’s let Scott from the Game Audio Institute de-mystify how audio for XR works and how it’s both similar and different to audio you’ve already experienced in games.
YouTube Video link:
Let’s get some basic facts out about how XR based audio works:
1. Audio for XR is still non-linear audio, at least if you’re talking about interactive experiences or games.
2. 3D games already have a very robust (though somewhat less realistic) form of spatial audio for well over two decades.
3. The usage of these new sophisticated technologies is going to depend on whether the game itself will need them.
4. Spatialized XR audio uses binaural encoding for headphones/earphones because XR devices are personally oriented.
5. In XR based games head rotation in particular is kept track of, and spatial audio relates in one of two ways to this action:
Head Tracked audio is audio that changes its apparent orientation relative to the head rotation. So, this behavior is similar to the head turning, and making the sound appear to come from the right side.
Head Locked audio is audio that does not change its orientation relative to the head rotation. This means that the audio will not change and in effect, acts like 2D audio, unaffected by distance or direction.
Now let’s cover the two new spatial audio technologies emerging in XR.
Ambisonic Audio
● It was originally adapted from the M/S (Mid Side) miking technique with other mics covering other dimensions and eventually developed into an electronic encoding procedure.
● It achieves better accuracy especially in height than stereo audio does. The level of detail is governed by the order number, with first order ambisonic files at the lowest level, and progressing upwards to higher orders, offering greater detail, although at greater computation cost.
● Strengths For Game Audio
○ Great for 360 Film or for audio in which the player’s head position does not change.
○ Non-local ambiences can also benefit from Ambisonic audio
○ Music can benefit by offering a larger sound stage.
● Weaknesses For Game Audio
○ Cannot track position relative to the player. Which means that if an object in a game is playing an Ambisonic file from a sound source, moving further away or closer will make no difference to the sound - only rotation of your head will affect it.
○ Ambisonic files of higher orders are not supported by the Unity and Unreal game engines
HRTF Based Audio
● Head Related Transfer Function - is a form of binaural audio filtering based on subject tests over the entire frequency range at different angles (azimuth) and at different heights (elevation).
● The result of these tests is compiled into an Impulse Response filter and then real-time convolution is performed on the signal to make it seem as if the sound is arriving right at your eardrum
● Strengths for Game Audio
○ HRTF is definitely an excellent audio technology that can take any 3D audio source and make it seem like the sound is coming from a 180 hemisphere surrounding the player
○ Not subject to the limitations of Ambisonic audio in that head position (distance) is also tracked in addition to rotation (direction).
● Weaknesses For Game Audio
○ You are limited to some extent by the platform’s processing ability to have a large amount of HRTF based audio sources present
○ It may not be the best sounding solution for all kinds of 3D games
Although both of these new spatial audio technologies can help sound designers in particular achieve more immersive results in their game’s audio, they should not be thought of as replacements for current game audio, but rather considered as an extension to the current audio palette. Instead professional designers should always be focusing on what serves the game best, and what can also be managed in terms of pipeline production for the developer or publisher.
Going Further
If you’re interested in taking a deep dive into XR based audio, check out the course in audio for XR at Game Audio Institute. This will be a hybrid course of Zoom meetings and lecture material. You’ll learn everything you need to know about this field and be working with Steam Audio and Unity, implementing sounds and music for a simple VR based GAI Game Lesson that you can then use for portfolio work. Seats are limited to 10 students, in order to give maximum attention to each attendee. And you’ll get 20% off on the course if you use the code ‘gai-vr2022-20’ at checkout. Go here for more information: