Jeff Sauro • March 18, 2014
Most people are comfortable with the concept of an average or percentage as a measure of quality. An equally important component of measuring the user experience is to understand variability.Here are 10 things to know about measuring variability in the user experience.
- Variability is inherent to measuring human performance. People have different browsing patterns, speeds, inclinations and motivations when they use software or websites. Differences in prior experience and domain knowledge often play a primary role in how users solve problems and accomplish tasks in interfaces. This can lead to vastly different experiences—including encountering different interface problems and resulting in large differences in task performance times or perception metrics.
- Often the differences between users outweighs the differences between designs. One consequence of the high variability between users is that even major design changes aren’t detected in completion rates or perception metrics from observing samples of users. It might be that changes did make an impact, but the variability between users masks the findings.
- Two ways to manage high variability are increasing the sample size and using a within-subjects study. When making a comparison between designs, if each user attempts tasks on both designs (in a counterbalanced order), you effectively eliminate the between user variability. This is called a within-subjects study. Each user acts as their own control and differences between designs are easier to detect–even with small sample sizes.If you cannot use a within-subjects study or are not making a comparison, the next best alternative is to increase your sample size. Remember though that you need to roughly quadruple your sample size in order to cut your margin or error in half. [Article]