Sample representativeness in usability testing: the ramble in the woods example

Simone Borsci 17.03.2014

You might imagine that a practitioner invites a certain number of people for a particular kind of ramble in the woods. The practitioner assigns participants the following goal: “Go in the woods and follow the pathway indicated by the map in order to identify and report to me where in the woods you see mushrooms without picking them up from the woods. You can note on the map the position of each mushroom you see.” After each participant returns from the woods, the practitioner has a list of the mushrooms identified by each subject. It can so happen that some subjects will have identified the same mushrooms in the same positions; these mushrooms are probably the most visible. Other mushrooms, however—those in more hidden positions—are identified only by a small number of participants. When a mushroom is identified by a participant, the identification reported by the other participants adds nothing to the overall discovery behavior of the sample, while any new mushroom identified increases the overall effectiveness of the sample. The more participants to go into the woods the higher the probability that the sample will have identified a larger number of mushrooms, because a larger group has a greater scope for divergent behavior (i.e., more participants looking for mushrooms in hidden positions) as compared to a smaller group. Involving a large sample is quite expensive for the practitioner, however, and it is never the most efficient solution. It is in fact possible to identify the smallest group with the greatest quality of discovery behavior. In this context, the quality of the behavior is seen as the ability of a small sample to accurately represent the behavior of a larger sample. We can thus define the representativeness of a sample as the degree to which the mushrooms (i.e., the problems) accurately identified by the sample represent the mushrooms that can be identified by all possible participants in the woods ramble following the pathway indicated on the map (i.e., the task of the evaluation test). 

From:Computer Systems Experiences of Users with and Without Disabilities: An Evaluation Guide for Professionals, pp.184-185


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s