ACS User Experience

Back-to-back tests

Today was a killer day of tests--five in a row, and a sixth in a couple hours. After a few hours of going through the same tasks over and over, things start to blur just a bit. Finally, my years of being an obsessive note-taker are about to pay off.

I've been told by a number of people that taking notes as soon after the test as possible is the only way to go. Otherwise, hours of reviewing videos to recall who said what and exactly when will be in your future. And they are right. So many details come out in these tests that would otherwise be forgotten as soon as the participant walks out the door if it weren't for methodically typing every significant event. Having two sets of eyes seeing the same thing, or sometimes interpreting the same thing two different ways, brings great depth to the process.

But now instead of going through hours of video, we have pages of notes to go through. Lesser of two evils? Maybe.

A day in the lab

I thought I'd sketch what a typical day in the usability lab looks like for those of us down in the trenches.

Nathaniel and I usually get to the lab (Baltimore or Leesburg) an hour before the first session begins. This way we have time to set up the computers, arrange the paperwork, and make sure everything is in working order before our participant arrives.

When the participant walks in the door, I'm usually already tucked away in the observation room. Nathaniel ushers them into a conference room where they fill out the paperwork--a typical release telling them they will be video and audio taped, and a general description of what the test is for. Prior to this, they don't know the test is for ACS.

Once we have their John Hancock and they have helped themselves to snacks (a couple Famous Amos' go a long way), the participant is sent to the participant's room, where the computer is set up with the site(s) they will be using (chemistry.org, pubs.acs.org, or pubs.acs.org/cen). Nathaniel sits in there with them, reads the instructions, and we begin.

Instructions? Here's the gist: they are given a series of tasks on paper, one at a time; once they read and understand the task, they say "I'm ready", which is our cue to start the clock; after they begin a task, they are not allowed to ask questions (as if they are sitting in their office with no one to assist); once they are finished they say "I'm done", and we stop the clock. That's about all there is to the actual test. 

We also remind them that it is the site that is being tested--not them. Inevitably, they walk out of the lab feeling bad and apologizing that they didn't perform optimally. But the fact is, an abysmal performance on the site tells us as much as, if not more than a great one.

Most participants are also very concerned about how long it takes to compete the tasks--and well, so are we. That's the point. The cut off is 15 minutes, and only a few have actually taken that long (try going on chemistry.org and purchasing thermometers--yeah, we sell thermometers). Typically each task takes no more than 3-6 minutes, depending on the task and the participant's experience.

Once they are finished with all 4 tasks, they take a short survey on their general impressions of the site--what they thought of the site, ease of use, visual appeal, and comments and recommendations. They also have an opportunity to comment on the test itself and what they thought of the tasks. Then Nathaniel interviews the participant and asks the perception questions--how long do you think it took you, how long should it take, do you think you were successful in completing each task?

Once that's done, they are given their incentive (a nice little Visa gift card), and they are on their way. Repeat 71 more times, and you've got yourself a decent usability test on the ACS sites.

HS students fast on task

Week 4 of testing ended very well. Most of our test-takers showed up and we recorded a lot of interesting events (the no-show is an industry hazard). Nathaniel and I have been able to keep to our original plan of finding people from the desired demographics and getting as many members as non-members involved.

The demographics were based on the personas that were developed in part by the Information Architecture team. We created about 9 personas based on the types of people we consider our primary audience and the people we intend to appeal to most in the future. They include chemists, of course, but also product engineers, biochemists, professors, high school teachers, policy makers, journalists, librarians, graduate students and researchers.

We also included high school students involved in AP chemistry to get the younger generation's perspective. My impression is that it is this group that gets through the tasks the fastest. The several students that we have tested have had no problem finding the optimal path for each task in the best time, whether it be the quick find menu on the home page, the tabs at the top of chemistry.org or the search function, they know the key tools to look for (advanced Google search), the typical names for things (online store), and the logic behind Web design.

This week we are back in Leesburg, VA at the Noldus lab. We will be starting the analysis of the times for each task soon. This will involve reviewing all the video from each session (each test is video taped) and making sure the times are accurate. What a process!

Usability Test: Week 4 - Baltimore

Rachel Yang here. I have been on staff at ACS for about the past five years, and with chemistry.org/Web Presence for about a year. I was recently assigned to complete a usability project that will give us information about how people use the Web, and more importantly, how they use ACS sites.

Nathaniel from e.magination and I have been alternating weeks between Leesburg, VA and Baltimore to test participants. We chose to use the two locations to get as broad a sampling as possible of chemists and other constituents who are likely regulars of ACS sites. The demographics are based on personas that were made in the information architecture team and include chemists, biochemists, professors, students, government employees, and journalists, both members and non-members.

We are in the middle of week four and have done more than 40 tests so far. Each test is about an hour long and includes a pre-interview, the tasks themselves, a survey, and a post-interview. Each participant is given 4 tasks to complete for one of our sites--chemistry.org, ACS Publications, and C&EN. They have a maximum of 15 minutes to do each task, but most don't take that long.

Want to try? Here's a sample task:
Go to the chemistry.org homepage.

Imagine you are interested in learning more about gas chromatography.
First, use the site to check if there are any upcoming classes in gas chromatography.
Second, find the page with the online form to enroll in the class.

How long did it take you?

Welcome to the Usability Blog!

Web Presence is currently conducting a benchmark test to measure user satisfaction and the time it takes to accomplish common tasks on ACS Web sites. This will create a baseline against which future changes to the chemistry.org and Publications Division websites will be measured.

This blog is intended to describe the process of usability testing and log the progress of the benchmark usability test currently in progress. Feel free to check in and see how we are progressing on our tests!