Journal of Usabilty Study – Bill Albert, May 2015
The idiom of “don’t let the fox guard the hen house” warns us about the potential danger of giving someone responsibility for overseeing something that he or she shouldn’t be involved with, particularly when there is an inherent interest in the outcome. Unfortunately, that idiom is all too fitting in the world of user experience (UX) research and practice, especially with respect to usability testing. I have seen all too often designers who evaluate their own design, or the design agency that is responsible for evaluating their own work. This is an inherent conflict of interest that results in poor quality research and, perhaps more importantly, undermines the credibility of our profession. The good news is there is an easy fix.
Let us start by stating that I do not mean to offend anyone, especially the very talented UX researchers and designers I work with every day or the design agencies that produce world class products across every industry. I have tremendous respect for their skill and professionalism. I know that they want to produce and deliver great user experiences to their clients. But even with the best of intentions, wrong decisions about how to test the usability of those products and services are sometimes made.
Even though there has already been a fair amount written about the risk of designers evaluating their own designs, this continues to be a problem, particularly with the lean UX approach. The general consensus is that this approach is not a good idea because designers have great difficulty in maintaining objectivity. It is widely understood that it is just too hard to maintain objectivity. But, what are the risks of design agencies evaluating their own design work?
In this editorial I focus on the inherent conflict of interest that design agencies have when they are responsible for evaluating their own design work, what can be done to mitigate this problem, and the implications for the UX community. I define a design agency as a consulting firm that is hired to design (from a visual and interactive perspective) digital products…