Asynchrony Solutions Blog

One Way To Conduct a Usability Study

by nikirae on March 3rd, 2011 at 2:54 pm

We knew usability was going to play a big role in the successful release of version 1.6 of an ongoing software development project. What we weren’t sure of was when we could schedule a formal usability study with the product end users. Fortunately, we were able to plan, conduct, and analyze the study prior to the product release.

The first thing we did was look over the feature list we had slated for the release to select which features we wanted our study to focus on. We ended up choosing five features that were prioritized high in the list

The usability study would consist of two formative usability tests on paper prototypes, a summative usability test on part of the application that was slated to get some major enhancements, and two user interviews on features that were completely new to the system. We had one hour to complete the study with each of our nine subjects over the course of three days, and only one week to prepare the paper prototypes, usability test scripts and interview questions.

When we arrived at the scheduled location, we had everything in place for a successful usability study. Our end users were each on one of the three teams that used the software application. To determine which users met the initial criteria for each part of the study, we conducted an introductory interview consisting of five yes or no questions. Based on their answers, we reordered our study protocols to match up with their team duties so we could focus on the features most relevant to them first, in case time didn’t allow for us to go through each of the protocols.

During each of the studies, two Certified Usability Analysts alternated between administering the study and taking notes. This allowed for us to both keep our energy up and momentum going over the next three days. During the usability studies, we took notes directly on the protocols we had printed out. After each one-hour study, we had time allotted to combine our notes and type them up, highlighting interesting things that our subjects said or did along the way.

It was a very long three days, but all of our end users showed up on time, and we ran most of the protocols with at least five subjects, so we had a good sample. When we returned back to St. Louis, we analyzed the test results and came up with a list of actionable items to deliver to the development team.

Since we already had the studies for all nine subjects transcribed, compiling the results wouldn’t take too long. During the test, we assigned each task a user attempted with a rating of 2 for completing that task without errors, a 1 for completing the task with some errors along the way, or a 0 for not completing the task. We took the average rating for each task and created a bar graph. From this graph, it was easy to see the tasks that had an average rating of 1 or lower, which we considered critical and should be addressed immediately. A written summary based on the rating, that stated what percent of users completed the task and with or without errors, was included with each task as well.

We looked through the transcripts for each of the users, pulling out those comments we flagged as interesting and added them to a master file that included the original tasks or questions for reference. These interesting comments were included a single time for each user, so sometimes there were duplicates which helped us see trends in how users were interacting with the application. The comments could imply solutions to making the task more usable or just provide insight into what our users were expecting.

After our results document was complete, we delivered it to the project manager and started creating the list of actionable items that would increase usability for the tasks our subjects struggled with the most. Regardless of how well the software works, we know the results of this study will improve how well the user works with the software.

Tags: agile, design, UCD, ui, usability