Unmoderated User Tests: How and Why to Do Them

Many UX teams rely on remote usability testing to efficiently get design feedback from users. There are two types of remote user testing:

• Moderated remote testing involves a researcher meeting with a participant via remote screen-sharing software, which allows the researcher to provide instructions, observe the user’s interaction with the design in real time, and ask followup questions specific to that participant’s session.
• Unmoderated remote testing does not require a researcher to attend each test session; instead, a software application provides instructions to users, records their actions, and may ask them predetermined followup questions.

Is Unmoderated Testing Right for Your Project?

Unmoderated studies do not include any direct interaction between the researcher and the study participants, which is both their biggest benefit and their greatest drawback.

Because there’s no need to schedule an individual meeting with each participant, unmoderated testing is usually much faster than a moderated study. It may be possible to launch a study and receive results within just a few hours. Unmoderated studies also allow you to collect feedback from dozens or even hundreds of users simultaneously. And for international studies, you don’t have to get up at an ungodly hour to match users’ time zone.

However, there are important limitations of unmoderated usability testing:

• Early-prototype testing is difficult without a moderator to explain and help participants recover from errors or limitations of the prototype.
• Without a moderator, participants tend to be less engaged and behave less realistically in tasks that depend on imagination, decision making, or emotional responses.

To better understand this second limitation, think, for example, about the difference between shopping and purchasing. Shopping can include many different types of research and comparison — there’s no single ‘right’ way to do it. In order to shop realistically, participants must first imagine themselves needing that product, then pay attention to details and make comparisons. A participant who just pretends to shop and isn’t very motivated often will glance only at a few products and quickly select one that seems reasonable. But, in real life, consumers who are spending their own money on a product they actually need behave very differently. Of course, a moderated study is not totally realistic either, but because participants are aware that the moderator is observing them, they will be socially motivated to fully engage with the task. Thus, this social pressure compensates for the lack of personal motivation in moderated studies — and even more so in in-person studies.

Unmoderated studies work best for evaluating live websites and apps or highly functional prototypes. They are appropriate for studying activities that don’t require a lot of imagination or emotion from participants.

Unmoderated research requires even more meticulous planning that a moderated study, since you can’t rely on human judgment to adapt the study procedures on the fly. For an unmoderated usability study, you’ll need to go through all the steps below:

1. Define Study Goals and Participant-Recruitment Criteria

Choosing software should not be the first step in unmoderated research. Before you decide which testing software to use, you should get a clear idea of what you hope to accomplish by doing the study. Then you can select a tool which has the capabilities best suited for your research goals, rather than limiting your study to fit within the technical constraints of a particular tool. Clearly articulated study goals allow you to identify must-have requirements for the testing software.

Goals vary from study to study, and a tool that is well suited to one study might be not at all effective for another. Knowing the study objectives is essential in order to make a good tool choice.

In this stage, you will also need to think about the types of participants you want to include in your study. What should their demographic be? Where do they need to be located? Will they be users new to your system or experts? Will they match a particular persona or user group in your target audience? These questions usually are determined by your study objectives and can also inform your tool selection.

2. Select Testing Software

For unmoderated studies, the software that administers the test is absolutely crucial to getting useful results. The software must guide the participants through the session and record what happens. It may also control the selection of study participants.

Fortunately, there are many different unmoderated testing services available, with varying combinations of functionality at different price points. The plethora of choices means that you can now be more critical in selecting a tool that suits the requirements of your project. (Features and pricing change frequently for many of the unmoderated testing tools, so be sure to compare them to your needs at the time of your study.)

It’s definitely worth your time to thoroughly investigate and pilot test tools, because migrating a study to a different system due to a technical limitation discovered after you launch a study is not fun! (Neither is trying to integrate data collected by two different tools.)

3. Write Task Instructions and Followup Questions

Many unmoderated testing services include study templates with generic example tasks. Don’t blindly copy them. The tasks that you give participants to do on your site or application should be highly specific to your situation. Generic tasks, such as “What is the purpose of this site,” are unlikely to give you good insights: to really assess the usability of your system you will need to write your own tasks.

In our experience training other companies to run their own remote usability tests, writing tasks is where most researchers fail in getting the results they need from their studies.

To write good task instructions for an unmoderated study, first articulate what it is you want the user to accomplish (such as: use the help section to answer a question, upgrade an account, or save an article to read later). Then, describe that objective with instructions that are specific, realistic, and actionable — without including hints that make the task too easy. You’ll need different types of task instructions depending on whether you’re doing a qualitative or quantitative study.

In unmoderated studies, the activities that you want the participants to conduct have to be even more carefully written than the tasks for moderated sessions. Participants cannot ask for clarification if they don’t understand the instructions and you can’t ask them to try again if they do the wrong thing. If users misinterpret your instructions and perform the wrong task, your test is wasted. Unmoderated task instructions should also explicitly tell users when they should stop; remember, the moderator won’t be there to ask them to move to a different task.

You should also meticulously plan any followup questions. These can include quantitative questions, in which participants rate the subjective difficulty or satisfaction of an activity. Or you may ask open-ended questions which prompt users to describe specific parts of the experience. Carefully choose how you phrase your questions; broad wording such as “How would you describe this brand?” may lead unmoderated participants to talk about their past experiences instead of the system they just used.

4. Pilot Test

A pilot test is a trial session that you run before your actual study begins, in order to discover any problems with your study design or procedure. Pilot testing is a good idea for all user studies but it’s especially important for unmoderated studies, because there won’t be a moderator available to fix problems while the study is running. Even the most thorough pilot testing can’t catch everything, but you can often detect and fix problems with:

• Task instructions that people misinterpret because the wording is ambiguous

• Tasks that are missing, or presented in the wrong order (especially in complex studies with many tasks)
• Prototypes that are missing functionality or content necessary for the study
• Incompatibilities or technical limitations which prevent your testing software from capturing the data you want to record — especially if you are trying to test an intranet or native mobile application
  (If your testing software records data via a web-browser extension, then browser or website restrictions may prevent it from capturing entire pages or sites or lead to low-quality recordings that are difficult to analyze. If you’re using a testing service for the first time, do a quick test of the recording process before you even bother setting up your tasks.)

You can discover some of these problems by going through the study yourself as though you are a participant, but others will not become apparent until you have real participants using their own equipment. Make sure to analyze the data collected in your pilot study!

5. Recruit Participants

There’s no point in watching people use your system if they don’t match your target audience. Make sure you have some control over who participates in the test, either through screening questions, or by recruiting your own participants.

Some tools only offer unmoderated user testing with participants from their panel, while others will provide you with a URL to distribute to your own pool of testers.

Using a provided participant panel is fast and easy (especially if your product is something that is relevant to a broad consumer audience). Panel participants are familiar with the study software, too, and can participate in your test as soon as they have time. Almost every panel includes some basic demographic filtering, but most studies get better results if you screen participants using questions about their behavior, not just about their age or gender. If the experience you’re testing is relevant only to people who meet specific criteria such as driving a car or shopping online regularly, make sure to use a tool which allows you to write your own custom-screening questions.

The downside of using panel participants is that many do these studies so frequently that they’ve learned to focus on certain aspects of the design and look for things to critique. To compensate for possible “professional testers,” recruit extra participants and exclude data from people who didn’t seem honestly engaged with the activities.

Being online means that you can test users on the other side of the globe as easily as people on the other side of the street. Many unmoderated testing services now have panels which include participants from all around the world. If your site targets international customers, unmoderated testing is a great way to reach a wide variety of locations. Just remember that if you recruit participants to complete the test in other languages, you’ll also need a researcher or translator fluent in that language to interpret the results.

In summary:

• B2B sites, sites that target elite or rich customers, and other services with narrow target audiences usually can’t use panels and must recruit the test participants themselves.
• B2C sites and other services that target a broad audience usually get faster and cheaper results by using a panel.
• You don’t always need the biggest panel, but if you require users from a particular region some panels may take a very long time to fulfill your study. (If you really need a specific audience, discuss your requirements with the testing service or panel manager in advance, and if they’re at all vague about whether they can do it, consider using a different service.)

6. Analyze Results

Unmoderated studies can quickly accumulate a LOT of data, so you’ll need an organized, analytical approach to turn this data into actionable insights about your design.

If you collect qualitative data, such as video and audio recordings of participant actions and comments, you’ll need to review each session recording. Users’ verbal and written comments can be misleading, so you have to watch their behavior in order to understand what works or doesn’t. In a moderated study you can follow along with the participants as they conduct activities, but during unmoderated studies you need to be able to watch a recording afterwards.

Screen recordings are helpful, but, in the absence of an audio recording, it is easy to miss why certain behavior occurred. If nothing is happening on the screen, is it because the participant is reading, or is she thinking about where to click next? The audio recording of participants verbalizing their actions is essential. (Recordings of the participants’ webcam to capture their facial expressions are also nice to have, but not essential. It can be more difficult to recruit participants who have a webcam and are willing to be recorded, so make it a secondary requirement.)

In each recording you’ll want to identify problems, questions, and both positive and negative reactions to the design. This process can be relatively quick if you have only a few recordings to review, but for large studies with dozens of participants, video analysis becomes extremely time-consuming. If you expect to analyze large studies (or to carry out several different smaller studies), look for an unmoderated testing tool which offers robust features video analysis, particularly:

• Tagging videos with timestamped notes as you watch them
• Aggregating, exporting, sharing, and visualizing the notes you’ve added to your recordings
• Producing short clips or highlight compilations of important moments in your recordings

Of course, you can make notes and video clips even if this functionality is not built in to the unmoderated testing tool — but unmoderated testing services which include these features don’t necessarily cost much more than barebones tools that lack them. If you will analyze more than a few hours of recordings, it’s well worth it to pay slightly more for a tool that speeds up your data analysis.

For extremely large qualitative studies, consider tools which can collect some quantitative measures or which offer automatic transcription. These features don’t eliminate the need to carefully review your recordings, but they can certainly expedite the process by directing your attention to specific recordings that are likely to be significant (such as recordings where users had low satisfaction ratings or recordings where particular keywords were mentioned).

If your study is primarily quantitative, your analysis will be quite different. Metrics such as success rate, task time, and subjective ratings will be automatically collected by your study tool. But to ensure your conclusions are accurate you’ll need to review your data and:

• Clean the data by identifying and excluding inaccurate values. For example, if a few task times are much shorter or longer than the others, investigate why, and exclude the outliers from your analysis if the values are inaccurate because the participants didn’t fully complete the task, or did the wrong task.
• Perform statistical tests to assess the significance of your results (especially when your goal is to compare multiple designs or tasks).
• Generate data visualizations to help communicate your findings to others.

Some unmoderated testing tools can automate the process of excluding outlier data points and many have built-in data visualizations charts. But, since cookie-cutter charts don’t always show the most important results, if you plan to do quantitative analysis, make sure your tool includes the ability to export your data so you can perform your own analysis using Excel or specialized statistics software.

Summary

Unmoderated research requires less work than moderated testing during the session, but it requires meticulous advance planning before the study begins.