Globally, an estimated 1.5 billion people develop mild (<20-34 dB; approximately 1.1 billion people) to complete (>95 dB; approximately 12.6 million people) hearing loss [1,2]. Older adults (aged ≥50 years) are much more affected [1]. If traditional external sound-amplifying hearing aids cannot treat severe or profound (≥70 dB [3]) hearing loss anymore, a cochlear implant is a viable solution for specific individuals [4]. Implantation depends on the individual clinical picture—for example, for one (ie, single sided) or both (ie, bilateral) ears or together with an additional hearing aid (ie, bimodal) [5]—and other prerequisites, such as access to and financial support for aural rehabilitation [1,6,7]. The underlying idea of cochlear implants is to stimulate the auditory nerve from within the cochlea with electrical signals generated by an externally carried processor [1,4]. Hence, the damaged areas of the auditory system in the inner ear are bypassed through a cochlear implant, resulting in a coarser signal resolution than a normal hearing sensation [1,8]. Therefore, patients must learn and practice interpreting this new stimulus of the auditory nerve, especially for language comprehension [7]. Supporting this learning process, postsurgery auditory training is crucial for cochlear implant recipients [6,9,10].

Auditory training can be divided into 2 main categories: face-to-face auditory training guided by a therapist and self-directed, home-based auditory training by the cochlear implant recipients themselves [10]. For most cochlear implant recipients in many countries, ongoing face-to-face auditory training is unattainable owing to financial limitations and the unavailability of therapists [11]. Hence, cochlear implant recipients are usually supported with self-directed, home-based auditory training materials like reading aloud tasks; having someone else read specific content to the cochlear implant recipient; listening to audiobooks, the radio, or television; or computer-based auditory training [10].

Unlike other auditory training materials, computer-based auditory training provides benefits such as automated testing and scoring, progress monitoring, real-time corrective feedback, or customized training [6,10]. Therefore, computer-based auditory training (with a particular focus on cochlear implant recipients [6,10-14]) gained much attention as an inexpensive, low-threshold, and successful rehabilitation form [6,9,10]. However, Völter et al [14] noted that a crucial determinant for the success of self-directed auditory training lies in the intrinsic motivation of the patient to adhere to a given training. However, as the authors noted further, patients with chronic illnesses are often driven by external motivation [14]. The patients are aware that the learning and training will be exhausting or painful but observe it as a necessary step to reach a desirable and enjoyable outcome [15].

Drummond et al [15] argue that in the context of eHealth, serious games can specifically address extrinsically motivated learners if the serious game lets the learners experience the enjoyment of the future outcome while presenting the learning activities. While the game design aspects of serious games, such as motivational, ludic activity, or narrative elements, are relevant to create this joyful experience [16-19], it has been shown that the educational content of serious games, such as exercises, meta-cognitive, or meta-reflection support, must be adapted to the actual skill level of the learner to avoid a motivational decrease (eg, through frustration or boredom) [19-21]. To achieve these dynamic adaptations, intelligent tutoring systems have emerged to mimic distinct human tutoring interventions [22,23]. Therefore, such systems must provide two domain-specific functionalities: (1) a detailed learner analysis and, based on this analysis, (2) a recommender service for content and instructional adaptations [24] (see research question [RQ] 1 and RQ2).

A recent literature review revealed that intelligent tutoring systems combined with gamified or playful content (eg, serious games) are common in the fields of science, technology, engineering, mathematics, and language learning [22,25]. Hence, with this feasibility study, we presented a novel approach that combined an intelligent tutoring system with a serious game in the context of aural rehabilitation for adult German-speaking cochlear implant recipients (see RQ3). We wanted to encourage future researchers and developers to build more advanced computer-based auditory training by answering the following RQs:

A digital, adaptive educational system tries to improve learning outcomes and raise engagement by altering the training application to a student’s or learner’s specific needs [21,26]. While the characteristics of these kinds of systems vary, researchers generally refer to them as computer-aided instruction, adaptive learning systems, or intelligent tutoring systems [22]. Usually, an intelligent tutoring system contains 4 conceptual models: the domain model (or content model), the learner model (or student model), the tutor model (or instructional model), and the interface model (or presentation model) [22,23,27]. The domain model contains all domain-related information pieces and their implicit and explicit structure and interdependencies [22,23,28]. The learner model captures what a person knows and does, for example, knowledge, preferred learning style, goals, or demographics [22,23,27]. The tutor model encompasses the didactic components and instructional strategy [22,23,27]. The interface model facilitates the interaction between learners and an intelligent tutoring system [22,23]. In a nutshell, the tutor model uses the learner model as the source for adaptation and the domain model and the interface model as targets for adaptation [22,23,27]. Therefore, examples of adaptation targets are specific content (eg, present feedback for specific errors), navigation (eg, the sequence of learning objects), presentation forms (eg, text vs video), and assessments (eg, difficulty level) [29].

Due to the variety of artificial intelligence systems used in educational systems in the past years [30,31], a definition of requirements for our use case is needed. First, due to the lack of available datasets about German-speaking cochlear implant recipients, a system design was needed to overcome a so-called “cold start problem” [31]. Second, due to ethical concerns, we were looking for an algorithm in the context of explainable artificial intelligence [25,30]. Regarding these preconditions, the evidence-centered design (ECD) framework seems to be a fitting methodology supporting the design and development process of the presented conceptual models.

Almond et al [32] summarized the ECD as “an approach for constructing educational assessments in terms of evidentiary arguments.” They argue that when learners fulfill tasks, they create some kind of result (work products) that incorporates (to some degree) the learner’s performance (compare with the study by Gnadlinger et al [33]). Thus, work products contain evidence about a learner’s latent competencies. Extracting evidence for competencies from performance aligns well with Forth’s [34] definition of competency as a “...set of skills and behaviors required in the performance of a task or activity within a specific context.” Hence, if a computer-based system collects this evidence, it can also model the learner’s competencies to some degree. While the ECD does not strictly depend on a specific statistical method to describe the learner model, Bayesian networks have often been used and suggested in the past [19,32]. Bayesian networks are probabilistic graphical models that hold a set of variables and their conditional dependencies as a directed acyclic graph [35]. The core idea of Bayesian networks is to use measurable variables (eg, exercise results) to infer directly immeasurable or latent variables (eg, the level of a complex competency) [35]. Hence, Bayesian networks can be used to model the learner’s competencies and continuously describe the current beliefs about these competencies by updating the measurable variables based on evidence from multiple tasks [32]. In the Conceptual Assessment Framework and Assessment Implementation section, we present how this can be achieved according to the use case of a learner model for language comprehension of cochlear implant recipients. Furthermore, we show how the other conceptual models that address these 2 prerequisites were designed and built.

Aural rehabilitation can be seen as a synonym for audiologic rehabilitation, auditory rehabilitation, hearing rehabilitation, and rehabilitative audiology and describes any intervention that addresses the communicative and psychosocial consequences of hearing loss [36,37]. Auditory training interventions vary in many aspects, such as training stimuli (eg, pure tones, phonemes, and complete sentences); frequency; duration of the training; and complexity [38]. It was shown that active, lexically oriented auditory training supported the learning process of adults far better than passive exercises [8]. In addition, a recent comparison of 16 studies of active auditory training provides evidence that intensive auditory and auditory-cognitive training supports the improvement of aural communication skills [38]. In a similar way this is also addressed by Deutsche Gesellschaft für Hals-Nasen-Ohren-Heilkunde, Kopf- und Hals-Chirurgie (German Society of Oto-Rhino-Laryngology, Head and Neck Surgery) [39].

Some very popular auditory training tools for English-speaking cochlear implant recipients are Angel Sound [40], the Listening Room [41], and MED-EL Academy [42]. A recent literature review of German and English computer-based auditory training shows that some auditory training systems support adaptivity and real-time feedback [9]. One prototype for a German-speaking hearing training platform was Train2hear [13,14]. It supports adaptive exercises to a certain extent and embeds different learning modules (filled with exercises) into a story about a journey through Europe. These systems generally analyze the quantity and kind of errors, the exercise duration, and the number of assistance requests to adapt exercises or training plans [13,14,40,41]. The supported adaptation of exercises can be categorized into (1) audio content—for example, differentiated by type (ie, syllables, words, sentences, and texts), similarity or complexity, and length (eg, word length); (2) hearing taxonomy—for example, differentiated into understanding, identification, discrimination, and detection; (3) exercise conditions—for example, difficulty adjustment via background noise, open or closed exercise sets, and the possibility for users to obtain assistance (eg, repetition) [13,14]. While some available platforms are well advanced according to the number of exercises and available content, none embed these exercises into a game-based learning environment. On the other hand, Garadat [43] successfully demonstrated the impact of a serious game to enhance the perceptual learning of speech by English-speaking cochlear recipients.

Hence, with this contribution, we presented a novel approach that combines an intelligent tutoring system that encountered the 2 major requirements (“cold start”—capable and explainable), with a serious game in the context of aural rehabilitation for German-speaking adult cochlear implant recipients.

The system design started from the perspective of assessments owing to the behavior of hearing therapists, who included exercises and tasks to assess hearing competencies and adapt the rehabilitation program. The ECD was followed because this methodology supported the design of assessments in educational systems and the inference of relevant competencies (Textbox 1).

In the beginning, an interdisciplinary team was gathered, including 2 language and speech therapists, a linguist, an audiologist, a cochlear implant surgeon, a game designer, 2 research associates with human-computer interaction backgrounds, and 2 software engineers. At the beginning of the design phase, the clinical staff gave insights into the aural rehabilitation process to the research associates. The research associates were also invited to participate in 2 face-to-face therapy sessions. Afterward, the team collaboratively designed a graphical model to describe the ability to hear and its subcompetencies in 4 distinct steps. First, the ability to hear was divided into general areas from “nonlexical language understanding” to “spoken language understanding” and additional sublayers from “nonlexical,” “lexical,” and “morphosyntactic.” Second, the experts agreed on major subcompetencies within the resulting areas, which the training environment should cover (visualized in blue in Figure 1). Third, the experts added the relationships between those subcompetencies. Finally, 7 different observable variables were defined: “sentence identification,” “word differentiation,” “word identification,” “consonant differentiation,” “vowel differentiation,” “sound categorization,” and “sound perception” (visualized as observable variables in Figure 1). The Conceptual Assessment Framework and Assessment Implementation section illustrates how these observable variables are used. The main result of this phase is a conceptional competency model pictured in Figure 1.

In the 2 ECD phases explained subsequently, the conceptional definition and implementation of the 4 main models of the intelligent tutoring system and serious game were performed.

The study design and evaluation approach addressed 3 main aspects: structure, process, and outcome [48]. In addition, system functionality, user perspective, and organizational context were considered [48]. Figure 3 illustrates the study timeline.

The 23 participants completed 2259 tasks (minimum: 3, RQ1: 32, median 70, mean 98.2, RQ3: 157.50, SD 79). In total, the 23 participants solved 2004 tasks correctly and 255 tasks incorrectly (median correct/median incorrect: 65/8, arithmetic mean correct/arithmetic mean incorrect: 87.1/11.1, min correct/min incorrect: 3/0, max correct/max incorrect: 229/40). The overall arithmetic mean of the incorrect answer proportion was 9.5%. Figure 4 presents the task type distributions of each participant and allows a comparison between them. All participants received a pseudonym from A to W. The total number of fulfilled tasks is stated in round brackets below this identifier.

The system also tracked the total playtime within the 4 weeks. This can be summarized as follows: average: 159, SD 106; range 7-376 minutes.

The result of the Chi-Square Goodness of Fit test (χ²₆=86.7; P<.001) allowed us to reject the null hypothesis that all task types were uniformly distributed. This confirmed a first assumption that each participant’s recommended task type proportion distribution was nonuniform, as described in the Tutor Model section. To further investigate the differences between the recommended task-type proportions, we performed a Kruskal-Wallis test because we met the assumption for it by checking a Q-Q plot and performing the Shapiro-Wilk test (P<.001), both indicating a nonnormal distribution. The null hypothesis of the Kruskal-Wallis test could be rejected (P<.001), meaning that the medians of the recommended task-type proportions were different. The result of the post hoc Dunn test identified 3 groups, A, B, and C, which shared a median from a similar distribution. A compact letter display format (Figure 5) was used to visualize this. The analysis shows that the training system recommended specific task types for the participating cochlear implant recipients more likely in the following grouped order—group A: “consonant differentiation,” “vowel differentiation,” and “word identification”; followed by group C: “sound categorization” and “sound perception” and group B: “sentence identification” and “word differentiation.” The resulting analysis also indicates that the system did not recommend the task types “sentence identification” and “word differentiation” for all cochlear implant recipients in the same way, compared with “sound categorization” or “vowel differentiation,” where the variance is higher.

Figure 6 gives an overview of the correct and incorrect answers from the recommended tasks. To further investigate the error proportions of each cochlear implant recipient based on the recommended task types, we performed a Kruskal-Wallis test because we met the assumption by checking a Q-Q plot and performing a Shapiro-Wilk test (P<.001) indicated a nonnormal distribution. With the result of the Kruskal-Wallis test (P<.001), we could reject the null hypothesis, which indicated that the medians of the error proportions from the task types were not equal. The result of the post hoc Dunn test identified 3 groups, A, B, and C, which shared a median from a similar distribution. To visualize this, a compact letter display format was used. Group C (“sound categorization”) showed the highest error proportion with a high variance between the cochlear implant recipients. Group A (“consonant differentiation,” “sound perception,” and “vowel differentiation”) showed a lower error proportion compared with group C but also showed a higher variance among the cochlear implant recipients, especially “vowel differentiation.”

Figure 7 shows the estimated competence level over time (ie, tasks) for each cochlear implant recipient for the overall competence hearing and all observable variables (compare with Figure 2). The starting position for each competence depends on the selected initial competence profile for the cochlear implant recipient. In Figure 7, this is visible because the starting points of each plot are grouped around initial starting values. A GAM analysis [52] for each competency was performed, and the resulting model (blue dotted line) and the CI (gray area) were plotted. These plots showed that the individually updated Bayesian networks converge over time by incorporating the produced task results (evidentiary arguments) from the 7 different task types for each cochlear implant recipient. The GAM analysis indicated that the system updates the competencies quicker in the beginning (strong gradient) compared with the end (flatted curve). You can reflect on this behavior by comparing competency based on the error proportions Figure 6 with the additive model analysis Figure 7. Suppose you compare, for example, the estimated competence “sentence identification,” which has a low error rate Figure 6, with the additive model analysis of the competency “sentence identification” Figure 7. The additive model analysis of “sentence identification” showed a strong gradient in the beginning and flattened over time. Doing so for the competence “sound categorization” revealed the opposite behavior.

The results were generated by 23 adult cochlear implant recipients participating in this study. The average age of the cohort was 54 years; approximately 60% of them activated their cochlear implant ≥4 years ago. Therefore, the cohort comprised experienced cochlear implant recipients. The GAM analysis of 2259 task results (including a maximum of the first 250 tasks from each participant) indicated that the system adapted to the estimated competency levels of the cochlear implant recipients more quickly in the beginning than at the end. Compared with a uniform distribution of all task types, the recommended task types differed, indicating that the system selected specific task types for each patient. This was underlined by the identified categories for the error proportion of the task types. The following discussion connects the design and development process of the game-based, adaptive auditory training with the results presented by the feasibility study to answer the 3 main RQs.

Following the suggestions of the ECD framework, a conceptual model that describes the language comprehension competencies and their interdependencies was designed. Our developed model aligns with previous findings like Erber’s hierarchy of listening [53] or the developed model used within the auditory training Train2hear [54].

We transformed our model into a machine- and human-interpretable probabilistic learner model (Bayesian network) to estimate the subcompetency levels of the competence language comprehension for each cochlear implant recipient. Seven single-choice task types (“sentence identification,” “word differentiation,” “word identification,” “consonant differentiation,” “vowel differentiation,” “sound categorization,” and “sound perception”) were developed that elicit a task result that incorporates evidence about the levels of the different language comprehension competencies. This evidence was used to update the learner model. This approach aligns with the suggestion from the ECD framework [32] and shows that this framework is applicable in the context of hearing rehabilitation. The developed task types empathize with already developed exercises from other similar computer-based auditory training software [14,40,54-56].

In Figure 7, we demonstrated how these Bayesian networks from the cochlear implant recipients converge over time by incorporating the produced evidence from different task types. A GAM analysis was used to visualize and reflect on the training system’s behavior. The calculated model (blue dotted line in Figure 7) indicates that the system updates the competencies quicker in the beginning (strong gradient) compared with the end (flatted curve) overall cochlear implant recipients in general. This matches with the error proportions in Figure 6 for the various task types, which also show low error proportions in general. On the one hand, this might indicate that their initial estimated competence level based on the “initial analysis” (see the Assessment Delivery section) did not fit their actual competence level. On the other hand, the generated tasks may not have met the correct difficulty level (compare further with RQ2). However, the visualization is evidence that the system adapted itself according to the behavior of the cochlear implant recipients and their competence level over time. Hence, compared with other adaptive computer-based auditory training systems [40,41,43,54,56], the illustrated use of a Bayesian network allowed us to estimate a cochlear implant recipient’s current level of language comprehension based on the required subcompetencies and their interdependencies.

The definition of task types serves as a blueprint to generate tasks. The task types were designed to use the individual estimated competence level to select the appropriate target key and distractor options from sound asset pools with a utility-based algorithm. To examine the system behavior, we analyzed each cochlear implant recipient’s first 250 fulfilled tasks and saw a strong variation between the fulfilled tasks (SD 79). However, the median of 70 indicates that 50% of all participants finished ≥70 tasks. These results and the total playtime (mean 159 min) already gave some evidence that cochlear implant recipients were willing to interact for a longer period with such an application. This finding is congruent with the results in the study by Völter et al [54], which indicated a high adherence rate for adaptive computer-based auditory training systems. We would like to examine this and the reason for their behavior in detail in an upcoming publication.

Our analysis of the recommendation system showed that the chosen task types were not uniformly distributed. Our deeper investigation revealed that “consonant differentiation,” “vowel differentiation,” and “word identification” were more likely chosen for the cochlear implant recipients (Figure 5). Furthermore, the examined error proportions associated with the 7 task types could be categorized into 3 groups (Figure 6). These identified categories do not align with the discovered categories from the recommended task-type proportions (Figure 5), which emphasizes the selection algorithm to consider different parameters rather than solely relying on correct or incorrect input. Figure 6 also indicates that the exercises were too easy even when they became more difficult because the error proportions were very low. Furthermore, there is a difference in the error proportion of the task types, which indicates that certain types (mainly “sound categorization”) seem more difficult to answer correctly. There might be various reasons for this result. For example, the task type “sound categorization” might have been misinterpreted by the target group, or the used sound assets did not meet the actual competence level. Therefore, further investigation of the task types and their difficulties would be a valid next research step.

We provided a detailed explanation of the conceptual and technological approach for building an intelligent tutoring system for aural rehabilitation, adhering to the recommendations for defining the 4 main models of an intelligent tutoring system: the domain model, learner model, tutor model, and interface model [22]. The intelligent tutoring system was designed to overcome two major requirements: (1) the so-called “cold start problem” due to the lack of available data in advance [31] and (2) the “explainable intelligence” due to ethical concerns in the context of eHealth [25]. The cold start problem was addressed using a Bayesian network as a learner model, designed based on expert knowledge, so initial learner data were unnecessary. In addition, the system can be explained by visualizing a Bayesian network and its local conditional probability distributions over time (Figure 7). This allows us to model each individual cochlear implant recipient based on their input, which stands in contrast to current common machine learning approaches that try to find one categorization model based on available data of a whole cohort (compare, eg, with the studies by Leduc-McNiven et al [57] and Goumopoulos et al [58]).

Furthermore, this description thoroughly shows how to embed this intelligent tutoring system via a task system into a serious game. The developed serious game in the form of a progressive web application supports 2 different game modes. The story mode embeds the tasks into a graphic novel–like game environment, while the quick training mode allows cochlear implant participants to practice 10 tasks in a row in a training-like environment. Hence, from a software architectural point of view, an autonomously functioning task system enables the generation of diverse scenarios by reusing identical task types. With the dialogue-driven story mode, we followed the initially stated argument that a serious game should allow learners to experience the enjoyment of the future outcome of their learning process [15]. Here, we allow the cochlear implant recipient to experience participation in conversations even in difficult situations (eg, at a restaurant, compare with parts A, B, C, and D in Figure 2). Arguably, the story mode might miss key elements that would classify it as a serious game. However, we emphasize the suggestion of the G, P, and S serious game classification to distinguish between serious games with a game-based gameplay component (strong goal oriented and rule based) and a play-based one (indirectly measurable goals) [59] and consider our serious game as a play-based one. It would be interesting to examine the motivation and flow experience of the cochlear implant recipients of serious games following a more play-based gameplay approach in further studies.

Unlike existing training platforms, which primarily analyze the quantity and type of errors [13,14,40,41,54] (as detailed in the Aural Rehabilitation and Existing Auditory Training section), our approach allows us to use the learning trajectory of each hearing competency and their interdependencies to recommend the next suitable training task for a cochlear implant recipient. By presenting a way to integrate such a training system into a serious game, we address the call from a recent literature review on the personalization of serious games to explore methods of incorporating learning progress into intelligent tutoring systems rather than relying solely on task results [60].

First, compared with studies in educational science where the ECD framework originated, the total number of participants in this study was relatively low because of the available funding for the conducted feasibility study. Hence, cautious generalization of the results is required. Nevertheless, the results presented in this publication are a valuable base for a formal sample size calculation (eg, Cochran’s sample size formula) for future studies. Second, only cochlear implant recipients from the Department of Otorhinolaryngology at the University Hospital Düsseldorf volunteered in this study, raising the possibility of an existing self-selection bias. This bias might also include side effects from different types of cochlear implant sound processors used by the cochlear implant recipients. Finally, we did not evaluate the competencies of the participating cochlear implant recipients with another measurement instrument due to the lack of standardized tests for the specific subcompetencies. The study was conducted to answer specific RQs and comply strictly with the EU GDPR. Hence, further post hoc analysis can only be applied to the specific tracked variables defined at the beginning of the project. Therefore, for example, questions regarding the preferred game mode of the cochlear implant recipients must be answered in future studies.

While the presented results allow a system-wise interpretation of the cochlear implant recipient’s input, future studies should focus on observing the cochlear implant recipient’s use behavior. This also includes the responses of cochlear implant recipients to specific game mechanics (eg, the preferred game mode) or interaction time. In addition, since the results indicated that the given tasks seemed to be easily solvable by the participating cochlear implant recipients, further evaluation is required to find out if the selected evidentiary arguments of the developed task types hold in general for cochlear implant recipients. In addition, examining the long-term effects of adaptive and game-based auditory training in the rehabilitation process of cochlear implant recipients versus traditional rehabilitation techniques would be very valuable.

For further research from a technological perspective, it might be interesting to investigate ways to use the potential of Bayesian networks to create synthetic data, as [25] suggested via agent-based simulations [58,60] for early development stages. Because the defined Bayesian network describes a competency ontology, future developments might consider coupling them with game mechanic ontologies illustrated, for example, in the study by Goumopoulos and Igoumenakis [61]. Such a coupling might be possible using generative artificial intelligence (eg, [62]) for immediate on-demand generation of training tasks that meet task difficulty and game mechanic requirements. Hence, a personalized recommendation approach similar to the one presented in this paper might lead to prompt generators that will reduce the cost of training content creation significantly.

Besides these technological perspectives, we already see a shortage of specialized therapists and trainers supporting adult cochlear implant recipients in Germany (compare with the study by Völter et al [54]). With the foreseeable increase in the number of adult individuals with cochlear implants, new ways of hearing rehabilitation are needed to address this growing gap between both groups.

This is the first attempt to map cochlear implant recipients’ language comprehension competencies using a Bayesian learner model for an intelligent tutoring system in cochlear implant rehabilitation. We integrated this system into an adaptive, game-based auditory rehabilitation training, addressing the need for an explainable design and solving the cold start problem owing to the lack of initial user data. A feasibility study with 23 cochlear implant recipients showed that the system adapts to the estimated users’ competency levels. The task system tailored tasks to each patient, as indicated by comparing error proportions and GAM analysis. With this work, we contribute and support the future game-based designs of computer-based, intelligent cochlear implant rehabilitation therapies; general hearing therapies; and similar fields in the eHealth context, where the personalized adaptation of a training environment is required to meet the needs of individuals.