Conduct a Pilot Test First

Before beginning a user research study or usability test, one of the most important things you can do is to run a pilot test. What’s a pilot test? It’s a rehearsal of the study with someone standing in as the participant. You run the session as if it was with a real participant, but you throw away the data. The purpose is to see if there are any problems with your technique and also to get some practice with the procedures involved. Afterwards, you can make changes to your technique and discussion guide.

What Problems Do Pilot Tests Find?

Pilot tests help you find problems in your research plan and methods, including:

• Do the participants understand what you’re asking them to do?
• Does the phrasing of tasks give away the answer?
• Are your questions or tasks biased?
• Do your questions or tasks elicit the kind of information you need?
• Are there repetitive or unnecessary questions or tasks that can be eliminated?
• Can the tasks be completed in the prototype, or are certain screens, states, or interactions missing?
• How long does the session take?

Choosing the Pilot Test Participant

For some studies, you can use just about anyone as a pilot test participant, such as a coworker, friend, or relative. For example, if you’re doing a pilot usability test of a general, consumer-facing website, such as a travel booking site, you could use just about anyone. They don’t have to have any special knowledge or experience.

But for some tests, it’s best to get a pilot test participant who closely fits the profile of the actual participants you’ll be using. For example, if you’re conducting usability testing of software used by tax accountants, it’s ideal to get a pilot test participant with the domain knowledge that’s needed to use that application. Even for something that seems simple, such as a study of people using a car buying website, it’s best to have a pilot test participant that fits the user profile of people who are considering buying a car in the next three months. If your questions have a lot to do with the motivation of car buyers and how they use the site, you’ll get very different answers from someone who doesn’t fit the profile. In these situations, if you use a pilot participant who doesn’t fit the profile, you won’t get a good sense of whether the questions are relevant, produce interesting discussion, or how long the session will take.

Can’t Your First Session Be the Pilot Test?

If you don’t specifically conduct a pilot test, then your first session becomesa pilot test. That’s where you’ll find the problems with your method, the tasks, and the questions. The problem with waiting until your first participant to find the problems, is that you won’t have much time to make changes before the next session. You may be able to make minor changes in the brief time between the first and second session, but it will be too late to make major changes.

The Ideal Pilot Test

Ideally, conduct the pilot test at least a day or two before the first session. Recruit an extra participant, who matches all the screening criteria, to perform the pilot test. If everything works out well, you can keep that person’s data. If you find problems, you can correct those before the official research sessions begin.

Test New Research Activities

Pilot testing is always useful, but it’s especially important when you’re going to try a new technique. For example, I once conducted a group workshop in which we had the participants divide into small groups to illustrate their work processes on post-it notes. We found that it was too difficult for them to do this together as a group activity. They ran out of time, and it wasn’t the best experience. For subsequent workshops, we had each participant work individually to illustrate their own work process, which was much easier and more effective. In effect, our first workshop was successful as an unintended pilot test. However, it would have been much better if we had planned it as a pilot test with a smaller group of participants.

Why Pilot Tests Are Important

You never quite know how well a research study will go, until the first session. It’s much better to work out the kinks in a pilot test than in the first session. Sure, you can sometimes get away skipping the pilot test, and everything works out fine. But it’s those times when things go wrong, which could have been found earlier, that you’ll wish you had conducted a pilot test first.

 

Image courtesy of K2_UX by Creative Commons license.