Two Out of Five Failed. Now What? The Sense Behind Small-Sample Usability Results
This is a very fundamental question of UX Research. You ran a usability test with 5 participants. For a specific task, two failed while three completed. Now, the stakeholders ask “is it broken or not?”
There are two wrong answers for this question:
- This is a 40% failure rate, we need to fix it.
- Most succeeded, we are fine.
The issue is that two out of five is not a precise measurement. With 5 participants, the numbers tell you too little. It cannot be judged how common or rare this issue exists.
To interpret that finding, we need to understand the sense behind the usability test with small sample sizes.
Remember the 5 participants rule
The famous test with 5 users advice comes from Jakob Nielsen and Tom Landauer.
Their modal says, the number of usability problems found in a usability test with n users is;
1 - (1 - L)^n
L is the proportion of usability problems discovered while testing a single user. They averaged across large number of projects to find the typical L, which they found to be 31%. This lead to the famous curve that shows; with only five users you can find almost the 85% of the usability issues.
Five is optimal because the each new users starts to add less and less new issue. In other words, you start to see the same issues again and again.
The rule is for to find the issues, not to measure how common they are. The whole sense behind the rule is to detect the maximum number of usability issues with the minimum number of users.
In the very beginning, where you decide on conducting this test with 5 users, you actually should have accepted that an issue found in one user may not occur again for other four; and it will be noted down regardless.
Get Aysu Atasoy’s stories in your inbox
Join Medium for free to get updates from this writer.
So the right question is not “what is our success rate?” It is “what did two people just reveal about a problem?”
Find out why the two failed
The most valuable part of the usability test is not the finding that two out of five failed, it is having the recording when these users failed. Going back to these sessions and answering the following question is the next step.
- Name the screen, the element and the step. e.g. Looked for the share option under Settings, never opened the Publish tab.
- Find whether it is the same experience for both. This shows whether this is a one shared issue for both, or two different issues for each.
- Go back to the success sessions. Did they hesitated, confused, guessed at the same point? If not, it doesn’t mean that the two can be ignored, but if yes, they also suggests further evidence to the issue.
Now, you are at a point where you detect the issue, understand why that issue occurs, and probably have a good idea about how to solve it.
Rate the severity
You found the usability issues, but it is not enough. How can you decide whether it needs to be act on first or later than other issues?
The good new is that you do not need to know frequency to prioritize an issue. In fact, across nine studies, MeasuringU found almost no link between how often a problem appeared and how severe it was. An issue that was seen only once can still be a huge blocker. Therefore, deprioritizing an issue based on the frequency is not the good call here.
This is where rating severity comes in.
You can use Nielsen’s 0–4 severity scale. A problem that fully blocks the task usually sits at the top. Dumas and Redish add a useful extra question: is it “global” (in navigation or a pattern used everywhere) or “local” (one screen)? A global problem will hurt far more tasks than the one you tested.
Test the new solution
Let’s go back to the article that presents the 5 users rule. Nielsen’s actual advice is to run several small rounds, not one test. Retest the new solution with another five people. That gives three possible outcomes: the fix worked, the fix missed the real cause, or the fix revealed a problem that was hidden behind the old one.
Closing thought
“Two out of five failed” sounds like a statistic, but with five people it is really a story with two chapters. Once you understand the sense behind the 5 users rule, the finding becomes easy to interpret.
What counts is being honest about the limitations of a small sample, and still being clear about what to do next.
