HOW CAN VOICE-CONTROLLED TECHNOLOGY BE USED IN

INTERACTIVE GAMES FOR INDIVIDUALS WITH MOTOR SKILL

IMPAIRMENTS?
Research Report (2706 Words)

https://summalinguae.com/language-technology/voice-controlled-games/

Student Cody Bowyer

Student Number 403243
Course FdSc Games Design & Development 2021/2022
Module Emergent Tech GA5002

0
Contents

Acknowledgements................................................................................................................................. 2
Abstract ................................................................................................................................................... 3
Background and Rationale ...................................................................................................................... 4
Research Methodology ........................................................................................................................... 6
Analysis of Data....................................................................................................................................... 7
Summary ................................................................................................................................................. 9
Conclusion ............................................................................................................................................. 10
Appendix ............................................................................................................................................... 11
Bibliography .......................................................................................................................................... 19

1
Acknowledgements

I would like to acknowledge these software’s and people for allowing me to be able to complete this
study:

• Epic Games – UE4 for allowing me to create the game.

• Shane Colbert – Speech Recognition Plugin (Sphinx UE4.20) for UE4.
• Microsoft Office Forms for the survey.
• Ten anonymous participants for taking part within the primary research.
• Steve Barton – programme lead and recording videos of some participants for my research
and to be posted on social media.
• OBS studio for the recording of some participants.
• UCWise staff for their support on this report and allowing me to use the provided
equipment.

2
Abstract

Voice control in video games requires a microphone to work; words spoken into the microphone by
the user will then be converted into text using a speech recognition software. If one of the converted
speech recognition phrases matches up with a set function, then the game will perform an action.
For example, if the player says, ‘turn left’, the character in the game may move 45 degrees to the
left. Other inputs seen to be used in video games is the loudness of the players microphone input to
perform an action, such as the louder the players microphone input the higher the character jumps.

Voice control is not present in most of the games we play today as a majority of players do not suffer
with motor skill impairments. In this context, motor skill impairments refer to a person that has
partial or total loss of body part functionality (Garcia, N., 2012.). For those that suffer with this issue,
the difficulty of playing games with voice control is greatly increased.

My research into this study aims to show game developers that integrating voice-controlled
functions into games they create would have a positive impact on the community and aim to show
how voice-controlled technology can be used in interactive games for individuals with motor skill
impairments.

The primary research that was conducted during this study demonstrated that games with voice-
controlled functions could become widely available in games, if larger companies in the games
industry experimented or researched about this subject. An example of this is Apple’s Siri, which is a
well-established function for Apple products. The primary limitation I found when creating the
practical project was finding a high-quality voice-recognition software to use.

3
Background and Rationale

Background information

In 1977 a British company called Maltron was founded to make fully ergonomic keyboards to battle
Repetitive Strain Injury (RSI) as their keyboard designs were made to provide less strain on the user’s
hands, due to the keyboard having a better fit to the shape of a hand. A one-handed ergonomic
keyboard was created by Maltron (see appendix A) with the purpose of aiding those with cerebral
palsy and various other disabilities where the user could only function with one hand. The problem
with this was that users with motor impairments in both hands could not use these keyboards.
(Maltron Ergonomic Keyboards. n.d.)

Mouth sticks and head wands were invented (see appendix B) and unlike the ergonomic keyboards,
these could be used by those with motor disabilities in both hands. They were also simple and low-
cost designs making them accessible to a wider variety of users. Both the mouth stick and head
wand designs were placed on the user’s face allowing them to then interact with either a keyboard
or a trackball mouse. These designs worked but caused fatigue when a lot of keystrokes were
required.

In 1952, the first voice recognition device was created by Bell Laboratories which could recognise
spoken digits. Voice recognition became more advanced over the years and battled the issue of both
hand impairments and fatigue when using a lot of keystrokes, as the user did not have to use any
movements apart from their mouth and was only required to speak keywords to be able to have an
output.

Current voice recognition uses

Voice control in games has been used in the past six years in order to create some games such as
One Hand Clapping game (Clapping, O., 2021.) (See appendix C) where the character is controlled by
the loudness of the player’s voice. Vocal Warrior (YouTube. 2016.) (See appendix D) the character
performs fighting actions based on what the player says through their microphone. The Voice Inside
(Inside, T., n.d.) (see appendix E) the player controls the character’s movements through different
keywords spoken into the microphone. From my research, it was quite hard to find games that only
required the players voice to be used.

Other forms of voice control can be found in technology such as iPhones that use a virtual assistant
named Siri (see appendix F). This is helpful for users with motor impairments as they can control a
phone with mostly just their voice. Smart speakers such as the Amazon Echo (see appendix G) are
controlled by your voice using a virtual assistant, it can perform many automated tasks and control
compatible devices. Software’s can be installed on computers to aid those with motor skill
impairments, such as Dragon made by Nuance (Nuance. n.d.), which outputs the user’s speech into
text on a chosen window. These types of software’s are particularly useful when a user with a motor
skill impairment would need to perform simple actions on a computer.

4
How it relates to the question

During my research into video games using voice recognition, I found most indie/studio companies
focus on the majority of players without motor skill impairments and create games that are based
around that audience. I also found that the software and technology for voice recognition are
seeming to become more powerful and inexpensive, these could be used more to aid those with
motor skill impairments within the games industry.

5
Research Methodology

Primary research

For my primary research I will be creating a simple platformer game where the participant will have
the ability to control the character through spoken movements. For example, if the participant was
to say “turn right” into their microphone, the character will consequently turn right. Before playing
the voice recognition game, the participant will be asked to read and agree to the description of the
survey and then asked a few questions about their previous voice recognition software knowledge.
After the game has been played, the participant will then be asked various questions based on the
game and their experience.

To teach the player the correct spoken inputs, I will be making a tutorial level that will take the
player through various obstacles with displayed text that will tell the player which input to use. The
main game will then consist of a range of different collectables for the player to pick up around an
obstacle course map. The collectables will be a range of difficulties, some may be easy to get to with
a small or easy obstacle path and some may be hard to get to with lots of obstacles and of a greater
difficulty. The participants will be asked to keep a record of the collectables they managed to collect
for an answer within the survey, to give me an idea of the difficulty of voice recognition.

The project will be using the ‘Sphinx’ UE4.20 plugin (Google Docs. n.d.) for the voice recognition and
will allow me to input my own controls, in order to allow the participant to move the character
around. However, due to a lack of other voice recognition software and the UE4.26 version of the
plugin not working as intended, this will be created in UE4.20.

To protect the participants’ data, they will be asked before playing the voice recognition game to
read and agree to the description of the survey. The terms and conditions in the description will
state that the participant will not be asked any personal questions, apart from their past experience
with voice-controlled games, and that withdrawal from the survey can be done at any time as
participation is voluntary. (GOV.UK. 2018.) (See appendix O)

This research will show the game developers a way they can create games using voice control
recognition, as well as showing its impact on the audience.

Secondary research – articles, case studies

Secondary research for this study will be conducted into different studies.

A study on the rise of voice-controlled games was conducted by Summa Lingue Technologies
(Summa Linguae. 2021.). This study goes into the challenges of developing voice-controlled games,
as well as some of the early examples of voice-controlled games.

A thesis was written on the history of voice interaction in video games conducted by Kiiski Teemu
(Theseus. 2020.). This thesis goes through the history of voice-controlled games from when it was
first founded up to what it has come to within the recent years.

An article was written on how voice provides a more immersive gaming experience conducted by
Leonard Klie (Speech Technology Magazine. 2021.) and talks about some of the games made in more
recent years, text-to-speech and other forms of voice-controlled technology.

6
Analysis of Data

I received ten anonymous responses from my primary research study.

60% of the ten participants had not played any voice recognition games in the past and the other
40% had. For those that had played these games in the past, they were asked for their experience
with those games, in which the average number out of four individuals was a score of 6.25 out of 10
(see appendix H). This backs up my argument with there not being enough voice recognition games
on the market and for the games that are available, the participants did not have the most positive
experience with them.

100% of ten participants had used other forms of voice recognition, to which they all gave an
average score of 7.6 out of 10 for their experience with voice recognition software (see appendix H).
From these responses compared to the voice recognition games, a much higher percentage of
participants had used voice recognition in this area. The average experience score was also much
higher and more positive, reinforcing my belief that if more focus was put into the gaming aspect of
voice recognition, it would be beneficial to create games that include voice recognition as this would
create a positive user experience.

From the study, 60% of the ten participants found the game had a medium level of difficulty
controlling the character and the other 40% had a hard level of difficulty controlling the character
(see appendix I). This may be down to the voice recognition software that was used; as some spoken
phrases, such as “turn right”, were not recognised as well as the other phrases. However, this also
demonstrated that voice recognition software in gaming is not as developed compared to other
areas of voice recognition.

Only 10% of the ten participants experienced background noise whilst playing, which affected their
gameplay experience. This appears to be one of the main issues faced when developing voice
recognition games, as 90% of the participants played the game within a controlled environment with
little to no background noise (see appendix J). Background noise can affect the gameplay for the
player as a phrase or word could be misinterpreted and end up not doing what the player intended.

The immersion level for all participants during their gameplay was asked in the survey. Out of ten,
10% didn’t feel immersed in the game, 70% felt neither immersed or not immersed and the other
20% felt very immersed when playing (see appendix K). The immersion level for this game leans
more towards the immersive side for the participants. As this is a simple game, it shows that if
games companies could put their effort into creating a voice recognition game it could have a very
immersive effect on their players.

Participants were then asked for their positive opinions on the voice recognition movements. Some
responses stated the voice recognition movements allowed for; “complete hands-free gameplay”,
“made me feel more connected with experience and that my voice actions took effect” and “made
the whole experience different from anything else I have played” (see appendix L).

Participants were then asked for their negative opinions on the voice recognition movements. Some
responses stated, “sometimes it would register me saying ‘one eighty’ when I said ‘forward’ or when
I wasn't saying anything at all” and that they “didn't find it very responsive” (see appendix M).

My programme lead (Steve Barton) asked for some of the participants to record the gameplay (see
appendix N), with a microphone and webcam facing the participant for the university to display on

7
their social media accounts. This was a positive experience in terms of developing my study further,
as I was able to watch this recorded gameplay footage multiple times. As a result of this, I learnt
more about the various gameplay experiences using the voice recognition software.

8
Summary

As voice control is not present in most of the games we play today, I wanted to find out how voice-
controlled technology could be used in interactive games for individuals with motor skill
impairments.

Voice control has advanced over the years as it started off as software that could receive spoken
digits and have advanced in the past six years into some games like; One Hand Clapping game
(2021), Vocal Warrior (2016), The Voice Inside (2022). As well as some other technology such as
Apples Siri, and Amazons Alexa.

To find out how this could be done I conducted my own primary research by creating a voice-
controlled game in UE4 and having different participants play and answer questions within the
survey. Secondary research was also done into various written pieces on the subject of voice-
controlled games.

The secondary research went into a study on the rise of speech technology, a thesis on the history of
voice interaction in video games and an article on how voice provides a more immersive gaming
experience.

From this study I found that majority of participants had not played a voice recognition game in the
past, but a majority of the participants had used other forms, like Apples Siri. I also found that the
immersion level for this game leans more towards the immersive side for the participants.
Participants were then asked for their positive opinions on the voice recognition movements. Some
responses stated the voice recognition movements allowed for; “complete hands-free gameplay”,
“made me feel more connected with experience and that my voice actions took effect” and “made
the whole experience different from anything else I have played”. One of my main concerns with
voice recognition is that it is controlled by a microphone and may pick up background sounds. This
was not present in my study as 90% of the participants played the game within a controlled
environment. Participants were then asked for their negative opinions on the voice recognition
movements. Some responses stated, “sometimes it would register me saying ‘one eighty’ when I
said ‘forward’ or when I wasn't saying anything at all” and that they “didn't find it very responsive”.

9
Conclusion

I felt that my study into ‘how can voice-controlled technology be used in interactive games for
individuals with motor skill impairments?’ went better than expected, as I was able to create a fully
functional practical project and survey for the primary research and received a high number of
participants for this. Looking back on the study, it gave me more insight into the world of voice
recognition games and motor skill impairments. It has also led me to believe that if larger companies
were to experiment or create fully functioning games with voice recognition, there would then be
more of a chance for the small developers to do the same, increasing the chance of people with
motor skill impairments to have more access to gaming.

If I was to conduct this study again, I would like to create a control variable for the primary research.
This could be done by creating another version of the project for players to play with keyboard and
mouse, as this could then be compared to the voice recognition software for a clearer understanding
on the difficulty.

I would also like to ask more questions to the participants during the survey within the motor skill
impairments area, as I felt like I could have gathered some more useful information. I managed to
gather a fair number of participants for this study, but a larger quantity of participants would have
greatly benefitted the end results for this study. To acquire more participants, I could have reached
out to people outside of the university.

Further experimentation or research into voice recognition software would have aided not only my
knowledge but the participants’ experience when playing the game. This could have been achieved
by reaching out to people with a greater experience in voice recognition or asking around on forums
for help, however due to the timeframe given this would not have been possible. External voice
recognition software may have also given me a better result as these are more advanced than the
Sphinx plugin I was using but were harder to work with UE4.

10
Appendix

A – Maltron One Handed keyboard (FAAST, Inc. n.d.)

B – Maltron Mouth Stick (Maltron Ergonomic Keyboards. n.d.)

C – One Hand Clapping (Hauter, E., 2020.)

11
D – Vocal Warriors (YouTube. 2016.)

E – The Voice Inside (Inside, T., n.d.)

12
F – Apples Siri (Greenberg, T., 2020.)

G – Amazons Alexa (Hahn, J., 2021.)

13
H – Participant data for past experiences

14
I – Participant data for difficulty

15
J – Participant data for background noise

K – Participant data for immersion

16
L – Participant data for positive opinions

M – Participant data for negative opinions

17
N – Videos of participants

2022-04-06 2022-04-06 2022-04-06 2022-04-06 2022-04-06

15-32-16.mp4 13-49-43.mp4 13-27-10.mp4 13-22-56.mp4 13-16-43.mp4

O – Primary research

https://forms.office.com/Pages/DesignPageV2.aspx?subpage=design&FormId=1v2NW4ONVUegOTJ
wddtRXD268j6OBeFHiUa9jj1DHXhUQUFZTVFVODZaMVVDV0pWSzhOWFdJMUZVRS4u&Token=6fa4f
16f4a174245a60d459dc77e1f3f

18
Bibliography

Berkeley. 2021. The Creator of Siri: “Make your prototype magic”. [online] Available at:
<https://scet.berkeley.edu/the-creator-of-siri/> [Accessed 15 December 2021].

Clapping, O., 2021. One Hand Clapping. [online] Steam. Available at:
<https://store.steampowered.com/app/893720/One_Hand_Clapping/> [Accessed 1 December
2021].

Ergomania. 2020. how does it make sense to develop a voice-based game?. [online] Available at:
<https://ergomania.eu/ill-give-it-to-you-straight-how-does-it-make-sense-to-develop-a-voice-based-
game/> [Accessed 10 February 2022].

Epic Wiki. n.d. Speech Recognition Plugin. [online] Available at:

<https://ue4wiki.imzlp.com/mediawikiv2-website-
prod05.ol.epicgames.net/indexe0f9.html?title=Speech_Recognition_Plugin> [Accessed 17
November 2021].

FAAST, Inc. n.d. Maltron One Handed Keyboard. [online] Available at:
<https://faast.org/product/maltron-one-handed-keyboard-2/> [Accessed 24 May 2022].

Garcia, N., 2012. Motor Impairment. [online] Neuromodulation. Available at:

<https://www.neuromodulation.com/motor-
impairment#:~:text=Motor%20impairment%20is%20the%20partial,muscle%20control%2C%20or%2
0total%20paralysis.> [Accessed 9 March 2022].

GitHub. 2021. Sphinx for 4.26. [online] Available at: <https://github.com/CHADALAK1/sphinx-ue4>

[Accessed 17 November 2021].

GitHub Wiki. n.d. How to use in your Project. [online] Available at: <https://github-wiki-
see.page/m/shanecolb/sphinx-ue4/wiki/How-to-use-in-your-Project> [Accessed 17 November
2021].

Google Docs. n.d. Sphinx_UE4_Demo [4.20].zip. [online] Available at:

<https://drive.google.com/file/d/1FEXPVhO67oIFWaKoVwsV3F9jcBwJy2rA/view> [Accessed 18
November 2021].

19
GOV.UK. 2018. Data protection. [online] Available at: <https://www.gov.uk/data-protection>
[Accessed 29 April 2022].

Greenberg, T., 2020. We need a full investigation into Siri's secret surveillance campaign. [online]
The Guardian. Available at: <https://www.theguardian.com/commentisfree/2020/aug/14/apple-siri-
secret-surveillance-campaign-investigation> [Accessed 24 May 2022].

Hahn, J., 2021. Amazon's Alexa. [online] People. Available at: <https://people.com/human-
interest/alexa-tells-10-year-old-girl-to-put-penny-in-electrical-outlet-mom-says/> [Accessed 24 May
2022].

Hauter, E., 2020. One Hand Clapping Interview. [online] Gaming Nexus. Available at:
<https://www.gamingnexus.com/Article/6141/One-Hand-Clapping-Interview> [Accessed 24 May
2022].

Inside, T., n.d. The Voice Inside. [online] Steam Discovery. Available at:
<https://www.steamdiscovery.com/1032280/steamgame/the-voice-inside/> [Accessed 24 May
2022].

itch.io. n.d. Albedo by ApeStudio, arnauruiz. [online] Available at: <https://apestudio.itch.io/albedo>

[Accessed 1 December 2021].

Maltron Ergonomic Keyboards. n.d. Maltron Head/Mouth Stick Keyboard. [online] Available at:
<https://www.maltron.com/store/p59/Maltron_Head%2FMouth_Stick_keyboard_for_Denmark.htm
l> [Accessed 24 May 2022].

Maltron Ergonomic Keyboards. n.d. Maltron History. [online] Available at:

<https://www.maltron.com/maltron-
history.html#:~:text=The%20first%20keyboard%2C%20(top%20image,of%20interest%20from%20po
tential%20clients.> [Accessed 31 March 2022].

Medeiros, J., 2019. 6 Voice-Controlled Games That Prove Voice Belongs in Gaming. [online]
Voicesummit.ai. Available at: <https://www.voicesummit.ai/blog/6-voice-controlled-games-that-
prove-voice-belongs-in-gaming> [Accessed 10 November 2021].

Microsoft. 2011. Single Handed Ergonomic Keyboard. [online] Available at:

<https://www.microsoft.com/buxtoncollection/detail.aspx?id=83> [Accessed 31 March 2022].

20
Nuance. n.d. Dragon FAQ. [online] Available at:
<https://shop.nuance.com/store/nuanceus/en_US/Content/pbPage.dragon-faq> [Accessed 14 May
2022].

Progress, W. and →, n., n.d. Sphinx-UE4: Speech Recognition Plugin. [online] Unreal Engine Forums.
Available at: <https://forums.unrealengine.com/t/sphinx-ue4-speech-recognition-plugin-release-v1-
0/69405> [Accessed 17 November 2021].

Scribbr. 2019. How to write an abstract. [online] Available at: <https://www.scribbr.co.uk/thesis-

dissertation/abstract/> [Accessed 9 March 2022].

Speech Technology Magazine. 2021. Voice Provides a More Immersive Gaming Experience. [online]
Available at: <https://www.speechtechmag.com/Articles/ReadArticle.aspx?ArticleID=146964>
[Accessed 10 February 2022].

Summa Linguae. 2021. Voice Controlled Games: The Rise of Speech Technology in Gaming. [online]
Available at: <https://summalinguae.com/language-technology/voice-controlled-games/> [Accessed
10 February 2022].

Tecla. 2017. Built-In Fine Motor Accessibility Features for Mobile Technologies. [online] Available at:
<https://gettecla.com/blogs/news/built-in-fine-motor-accessibility-features-for-mobile-
technologies> [Accessed 15 December 2021].

Theseus. 2020. Voice Games: The History of Voice Interaction in Digital Games. [online] Available at:
<https://www.theseus.fi/bitstream/handle/10024/336352/Voice-Games-The-History-of-Voice-
Interaction-in-Digital-Games-Teemu-Kiiski-final.pdf?sequence=2&isAllowed=y> [Accessed 10
February 2022].

WebAIM. n.d. Types of Motor Disabilities. [online] Available at:

<https://webaim.org/articles/motor/motordisabilities> [Accessed 31 March 2022].

Wiki.unrealengine.com. n.d. Speech Recognition Plugin. [online] Available at:

<https://michaeljcole.github.io/wiki.unrealengine.com/Speech_Recognition_Plugin/> [Accessed 12
October 2021].

21
Youtube. 2012. Introducing the Leap Motion. [online] Available at:
<https://www.youtube.com/watch?v=_d6KuiuteIA> [Accessed 29 September 2021].

Youtube. 2016. Speech Recognition Test Video. [online] Available at:

<https://www.youtube.com/watch?v=wtXIiFTre_w&t=2s> [Accessed 3 November 2021].

Youtube.com. 2016. Sphinx UE4 Speech Recognition Demo - Version 1.0 Demo. [online] Available at:
<https://www.youtube.com/watch?v=b6vd0pWSGKU> [Accessed 18 November 2021].

YouTube. 2016. Vocal Warrior: World's First Voice-Controlled Fighting Game. [online] Available at:
<https://www.youtube.com/watch?v=9R_tSt_sn_E> [Accessed 1 December 2021].

YouTube. 2017. Intense Voice Controlled Game. [online] Available at:

<https://www.youtube.com/watch?v=ykKor95hcCs> [Accessed 10 November 2021].

Youtube. 2017. Pokemon Go's new augmented reality mode. [online] Available at:
<https://www.youtube.com/watch?v=yUNtfM1CIPU> [Accessed 29 September 2021].

Youtube. 2017. Say Hej to IKEA Place. [online] Available at:

<https://www.youtube.com/watch?v=UudV1VdFtuQ> [Accessed 29 September 2021].

Youtube. 2019. Emergent Gameplay - Voice Recognition Prototype. [online] Available at:
<https://www.youtube.com/watch?v=MGsJA18VwyI> [Accessed 29 September 2021].

Youtube. 2021. Hand Physics Lab: Launch Trailer. [online] Available at:
<https://www.youtube.com/watch?v=w-w7omxJuOk> [Accessed 29 September 2021].

Youtube. 2021. Speech Recognition Test Video. [online] Available at:

<https://www.youtube.com/watch?v=wtXIiFTre_w> [Accessed 17 November 2021].

YouTube. 2021. THE VOICE INSIDE | Voice-Controlled Game |Gameplay Trailer 2021. [online]
Available at: <https://www.youtube.com/watch?v=kf4cpPdNfv0> [Accessed 1 December 2021].

22