The Kopstoring course aimed to prevent behavioral and psychological problems in offspring at risk and was offered to adolescents from ages 16 to 25 years. The Kopstoring course was an online 8-week group course with supervision by 2 trained psychologists or social workers from a participating mental health institution in the Netherlands. Every week a different theme was discussed and participants were expected to prepare for the weekly meetings by doing homework. The course had a preventive nature; therefore, adolescents were screened to ensure that they were not diagnosed with an illness as classified by the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) diagnoses. Screening was executed by the mental health institutions. In addition, participants needed to have access to a computer with an Internet connection and be able to participate weekly.

Alongside the process evaluation described in this paper, a RCT was conducted. The aim of the RCT was to examine the effectiveness and cost-effectiveness of the Kopstoring course. Participants were randomly allocated either to immediate enrollment in the Kopstoring course (intervention group) or enrollment after a 6-month waiting list (control group). Because the course was completely digitalized, the recruitment was done mainly, but not exclusively, through online recruitment, including banners, Facebook advertisements, links to the website, etc. In addition, articles were published in national and regional magazines and newspapers, school visits were performed, and an interview was broadcasted on a radio station.

In this process evaluation, 2 groups were included: participants (n=13) and providers (n=4) of the Kopstoring course.

The Kopstoring participants were selected from both the intervention and the control group of the RCT. Participants received an email in which they were invited to be interviewed. To select participants for the interviews from the trial pool, a maximum variation strategy was used to gather information from a sample with as much variation as possible to collect as many different perspectives [16]. This was done by looking at several characteristics (eg, trial arm, age, sex, online and written consent for the trial, dropout). See Table 1 for the characteristics of the Kopstoring participants who were interviewed. The interview sample is fairly comparable to the RCT sample.

At the start of the project, 9 Dutch mental health institutions participated and each institution trained 2 professionals for the provision of the Kopstoring course. All planned Kopstoring courses provided during this study were provided by 7 professionals from 4 different mental health institutions. These providers were invited to participate in interviews. In total, 4 providers agreed to participate in an interview.

The providers of the Kopstoring course were all female and approximately 30 years of age; all had a Master’s degree and had experience working in this field for 5 to 7 years.

Data were collected through semistructured individual interviews with a list of topics to be discussed (Textbox 1). Interviews were held between November 2014 and February 2015. The topic list was made by the research team in collaboration with the national coordinator of Kopstoring and the team of course providers.

Anonymity was very important for offspring of parents with mental illness or addiction problems. Therefore, interviews were conducted over the phone. Participants decided the time of the interview so they could be sure they were able to talk freely. Interviews with the providers were also conducted over the phone, but due to time constraints not because of anonymity. Interviews were held in Dutch. Textbox 2 displays the topic list for the providers.

The interviews were audiotaped and transcribed and identifiable information was removed to ensure anonymity. The interviews were analyzed by using inductive qualitative content analysis, specifically conventional content analyses [17]. This method helped provide an in-depth understanding about underlying perspectives and qualitative methods are inductive and reflexive and it allowed the use of quotes [18]. As a first step, the interviews were read by 2 researchers separately to identify emerging themes and subthemes and then labels were attached to the parts related to these themes. Secondly, new themes were added to existing themes and labeled accordingly. After the 2 researchers reached consensus, the interview data were clustered into themes and subthemes. Finally, citations of the interviewees were identified per theme and visualized in a data matrix. After approximately 13 interviews, no new information emerged from the interviews with the participants.

For the participants, the background mainly related to the motivation and reason for participation, the route to registration, and expectations of the online course. The providers’ background related to their experiences with the provision of similar face-to-face courses and online interventions.

There are some requirements when providing online interventions. Firstly, participants and providers needed a computer connected to the Internet, log-in codes, and some privacy. For the providers, a budget was necessary to provide the Kopstoring course. Some barriers were encountered at the organizational level and regarding the financial structure.

The technical component was found to be extremely important by both the participants and providers of the Kopstoring course. Not only were the technical aspects (eg, the website or the chat box) considered positive factors, the same technical aspects were mentioned as barriers for participating in or providing the course. Almost every interviewee mentioned technical problems of some degree (from having a slow system to being thrown out of the chat box) and providers also mentioned these disturbances interrupting the courses. However, there were no major incidents mentioned that fully hindered participation in the long run.

The technical problems described by both groups ranged from technical hiccups to some more prominent problems, such as not being able to log on to group sessions or being locked out by the system.

Minor technical problems involved a slow system, not being able to see when someone was typing, unable to see homework assignments on the screen, and double messaging occurring. These problems were mentioned, but were interpreted as minor problems and a consequence of an online working environment:

Providers also described experiencing the same minor problems. The technical platform and responsibility related to the technical aspects were more numerous for the providers. They were responsible for all requirements to be met even before groups started online sessions.

Participants were confronted with aspects such as a 6-month waiting list, randomization, extended follow-up, and questionnaires. It appeared that most participants understood there was a study linked to the Kopstoring course, but none of them could describe what the consequences were for them; regardless, patient information sheets were given to them by email, mail, and online:

Participants who were allocated to the waiting list believed they had to wait because the group was full. This explained why most participants expressed no strong negative experiences toward the research components. In some cases, the waiting list was experienced as problematic, although most participants accepted the waiting period:

Most of the participants indicated altruism as the main reason for participating in the study, although others participated because “it is part of the course”:

Participants provided feedback on the length of the questionnaires and some technical problems related to not being able to open links or links expiring due to waiting too long to fill out the questionnaire. From the interviewed Kopstoring participants, 2 persons had incomplete data; when asked why, there was no specific reason, but they said they forgot. In addition, some of the respondents mentioned the phrasing of some of the questions. They disliked the questions because they were too focused on the younger participants (students) living with their parent(s).

During the interviews, a couple reasons for the poor response rates were mentioned: laziness or forgetfulness and problems with parental informed consent in case of a minor.

Participants were asked to give online consent and written consent sent by post. Only one participant did not send back the informed consent papers and, therefore, was not a participant in the RCT. She explained that she forgot to send the forms back, whereas the other participants had clear motivations for participation in the study. Some minors sent back their informed consent papers, but not those of their parents, stating they did not wish their parents to know let alone sign a consent paper for participation in the study. Some minors found a way to let their parents sign.

To our knowledge, this study is the first evaluation of the experience of providers and participants with an online-delivered prevention course for offspring at risk. Therefore, this study differentiates itself from existing international literature and provides new information. The few process evaluations performed to assess experiences with online programs focus on online treatment, programs for somatic diseases, or and/or an adult patient population. The findings of this study give insight into the experiences of participants and providers of an online prevention course called Kopstoring. It sheds some light on experience with as well as barriers and facilitating factors of online delivery. It elaborates on the expectations and experiences of both participants and providers. Analyses showed similar experiences for the 2 groups despite their different perspectives.

The main lesson learned from participants lies in their assessment of the course content and the barriers and facilitating factors for participating and adhering to an online course. The online aspect and anonymity proved to be important as well as their autonomy to decide to participate without interference from anyone else. In the Netherlands and some other countries, minors (participants younger than 18 years) need to provide the research team with written consent and their parent’s consent for participating in a scientific study. This ignores the fact that minors can receive treatment (which is being assessed) of any kind from the age of 16 years without parental consent. Youth are considered capable of making an informed decision about treatment; however, for a scientific study we doubt their ability to make an independent and informed choice [19]. This subject was brought up even by participants older than 18 years who said that if they had to provide parental consent, they would probably not have participated. They explained that we cannot expect them to ask their parents for consent when they are the root of their problems. There is a strong need to rethink the policy concerning consent in the case of interventions for vulnerable populations and interventions with a high level of anonymity (mostly online interventions). This statement endorses the debate in the literature questioning when a minor should be considered capable to give informed consent and therefore protect his/her anonymity [20,21]. Despite the differences in consent procedure for minors, no substantial differences between minors and participants older than 18 years are reported in this study.

The lesson learned from the providers of the Kopstoring course lies partly in their professional assessment of the content of the course, but mainly in the experience with providing online courses and the barriers and facilitating factors to provide the Kopstoring course. Analyses showed that providers of online interventions in RCTs might feel ignored and may experience a gap between the research team and providers, even though the provider has many years of experience with providing online interventions in this target population. This implies that there is a need for closer collaboration with providers, and perhaps even with the target population, when designing such interventions and accompanying studies [22,23]. Collaborating with stakeholders could have led to other research questions, methods, and the use of other questionnaires more suitable for the target population.

The current situation in the Netherlands for children in need of mental health care is unsettling. This study could not have taken place in a more inconvenient time and political setting than it actually did. In the same period the RCT was running, political decisions forced youth mental health care out of the hands of mental health institutions and made it subsequently a part of the local municipalities. Even if the results of cost-effectiveness studies, such as the Kopstoring RCT, show positive results, online interventions may not be provided due to the complex financial structure and lack of responsible bodies to finance online interventions. This also shows that implementation and implementation research in the Netherlands, but very likely in other similar countries, is nearly impossible for these types of intervention.

There are several factors that could be considered to influence the findings of this study. The first is the number of interviews performed. One can question whether the small number of providers interviewed is sufficient to provide a complete overview of the ongoing issues. However, we do believe that providers who cooperated gave a lot of information about the Kopstoring course and the delivery of the course. We remained with only 9 providers who provided more than one course and were totally informed about every research detail. Therefore, we believe 4 providers were a good reflection of the 9 remaining providers and the group appeared to be homogeneous.

For the participants, there is a different reason for the low response rate (42 people were invited, 17 responded, and 13 were eventually interviewed; response rate: 13/42, 31%) for participating in this study. The target population appears to be extremely difficult to reach. As shown in the analyses, they wish their anonymity to be respected and feel “safe” in an online environment and not face-to-face or on the phone. In addition, a feeling of shame and guilt regarding their problems blocks them from sharing their experiences with a researcher. Despite this, the majority of the participants were enthusiastic about the online prevention course and potential bias might occur with this. It is possible that youth with negative experiences with the course or research were not willing to be interviewed. Additionally, due to the sensitivity of this problem and the fact that the parents are involved, youth might find it difficult to speak about this with a third party (ie, might feel like “airing their dirty laundry” in public). However, for both participants and providers, repetition in the interviews showed a level of saturation.

A second limitation relates to the generalizability of the findings. It is noticeable that an overwhelming majority of Kopstoring participants, participating in the underlying RCT and this process evaluation, were female. This is probably not a good reflection of an open population, assuming there are an almost equal number of boys who have a mentally ill parent as there are girls. This leaves questions about generalizability unanswered. In addition, questions have been raised such as “are the findings useful in a similar online context, but with a different underlying intervention?” and “are the findings the same when comparing the online course to a similar face-to-face group?” It appeared that several factors added up; the online aspect, age, anonymity, and sensitive problems and anonymity lead to barriers doing research within this vulnerable group. The results of this study focus on youth with parents with mental illness or addiction problems. Despite these factors, some general elements can be identified that are useful in other online settings, such as the aspect of anonymity, consent, and practical issues.

Online support for offspring of parents with mental illness or addiction problems is considered effective by the participants. There are not many RCTs performed to assess the effectiveness and cost-effectiveness of online prevention programs in the field of mental health care [24]. Consequently, there are not many process evaluations of these online prevention programs performed. This hampers comparison between online programs and process of delivery and expectations. In addition, a face-to-face group is set up differently in structure and has fewer participants; therefore, it is difficult to use it in comparison to an online program. In this respect, this study is unique and sheds some light on experiences and barriers for online provision of a prevention course in the field of mental health care.

The barriers for online provision of this health intervention are minimal, but the ones that exist lie in the technical sphere. Barriers for online research are multiple and touch on different aspects, such as informed consent, anonymity, lack of time, or just lack of interest. The findings of this study may explain partly why there are substantial dropout rates when delivering online interventions. The experiences of participants and providers of the Kopstoring course give valuable insights into the process of the online provision and study of Kopstoring.