-
Notifications
You must be signed in to change notification settings - Fork 14
Usability Tests
Recordings of each usability test are available at the links below.
The tests are designed to identify areas where we can make the portal more intuitive. The tests aren't really about about fitness-for-purpose (that comes later). The idea is to let the user muddle through unassisted so we can see where the portal is difficult to understand.
For those of you reviewing the usability tests I recommend considering the advice below (which is an excerpt taken from Don't Make Me Think by Steve Krug).
Here’s the best advice I can give you about deciding what to fix—and what not to.
Ignore “kayak” problems. In any test, you’re likely to see several cases where users will go astray momentarily but manage to get back on track almost immediately without any help. It’s kind of like rolling over in a kayak; as long as the kayak rights itself quickly enough, it’s all part of the so-called fun. In basketball terms, no harm, no foul.
As long as (a) everyone who has the problem notices that they’re no longer headed in the right direction quickly, and (b) they manage to recover without help, and (c) it doesn’t seem to faze them, you can ignore the problem. In general, if the user’s second guess about where to find things is always right, that’s good enough.
Of course, if there’s an easy and obvious fix that won’t break anything else, then by all means fix it. But kayak problems usually don’t come as a surprise to the development team. They’re usually there because of some ambiguity for which there is no simple resolution. For example, there are usually at least one or two oddball items that don’t fit perfectly into any of the top-level categories of a site. So half the users may look for movie listings in Lifestyles first, and the other half will look for them in Arts first. Whatever you do, half of them are going to be wrong on their first guess, but everyone will get it on their second guess, which is fine.
You may be thinking “Well, why not just put it in both categories?” In general, I think it’s best for things to “live” in only one place in a hierarchy, with a prominent “see also” crosslink in any other places where people are likely to look for them.
Resist the impulse to add things. When it’s obvious in testing that users aren’t getting something, most people’s first reaction is to add something, like an explanation or some instructions.
Very often, the right solution is to take something (or things) away that are obscuring the meaning, rather than adding yet another distraction.
Take “new feature” requests with a grain of salt. People will often say, “I’d like it better if it could do x.” It always pays to be suspicious of these requests for new features. If you probe deeper, it often turns out that they already have a perfectly fine source for x and wouldn’t be likely to switch; they’re just telling you what they like.
Grab the low-hanging fruit. The main thing you’re looking for in each round of testing is the big, cheap wins. These fall into two categories:
Head slappers. These are the surprises that show up during testing where the problem and the solution were obvious to everyone the moment they saw the first user try to muddle through. These are like found money, and you should fix them right away.
Cheap hits. Also try to implement any changes that (a) require almost no effort, or (b) require a little effort but are highly visible.
And finally, there’s one last piece of advice about “making changes” that deserves its own section:
Don’t throw the baby out with the dishes Like any good design, successful Web pages are usually a delicate balance, and it’s important to keep in mind that even a minor change can have a major impact. Sometimes the real challenge isn’t fixing the problems you find—it’s fixing them without breaking the parts that already work.
Whenever you’re making a change, think carefully about what else is going to be affected. In particular, when you’re making something more prominent than it was, consider what else might end up being de-emphasized as a result.
Kerrie hit a few technical problems but otherwise got what she wanted. There were a few interesting moments however:
- Kerrie said her first instinct was to draw a box on Step 2. This has happened on all tests so far (apart from Dan who had seen it before). She didn't get stuck here though.
- Poorly performing ncWMS layers were a big problem.
- Kerrie got a bit sidetracked when she clicked "more" on Step 1 and it took her to geonetwork. The "more" buttons should show the rest of the abstract (with a "less").
- It took Kerrie a while to figure out how to subset CPR by date, but she seemed happy with it after she found out that the CSV file contains multiple voyages. This seemed to be something she really wanted, and she said it was a big step forward. Pity there was a stack trace in the file rather than data :-(
- Kerrie thought the progress bar looked like buttons. We went to some trouble to make them NOT look like buttons - so this was disappointing. It didn't affect her though - so I think we can leave this for now.
- When she first arrived at Step 2 and was looking at the subset tab she looked closely at the opacity slider and the transect graphing. This was a distraction. The opacity slider would be better off in the styles tab, and the transect graphing would be better off somewhere else too. It's sort of a "kayak" issue but subsetting is the most difficult step so I think we should simplify this part of the UI (in particular) as much as possible.
This test went remarkably smoothly for someone who's never seen the 123 portal before. Kristina stumbled slightly at the same point as Pete Struton (the spatial extent), but, unlike Pete, she figured it out quite quickly. She even said something like "Oh .. the the map is the subset" :-)
We need more usability improvements yet, but we should feel pretty happy with ourselves that at least one user found it easy to use. Especially as it's only the first round of usability testing.
The obvious usability issue here was that Pete couldn't figure out how to change the spatial extent on step 2. He got there in the end - but it was quite a challenge!
Pete also got tripped up by the distinction between sea surface temperature and skin temperature. But he figured it out - with a small prompt. Perhaps this is just a "kayak" problem. We'll find out after more tests.
This is the first usability test. It's a dress rehearsal for the usability tests to come.
n.b. Something went wrong with the recording at the seven and a half minute mark (about 30 seconds were lost). It also truncated the last minute or so.
- Dan had trouble when he tried to download a file and it temporarily took him to a new tab.
- Dan struggled to find "oxygen" in the list of parameters - even though it was third in the list.
The tests roughly follow a script to keep them consistent.
Hi, ____. My name is ____, and I’m going to be walking you through this session.
You probably already know, but let me explain why we’ve asked you to come here today. We’re testing a web application that we’re working on so we can see what it’s like for actual people to use it.
I want to make it clear right away that we’re testing the software, not you. You can’t do anything wrong here. In fact, this is probably the one place today where you don’t have to worry about making mistakes.
We want to hear exactly what you think, so please don’t worry that you’re going to hurt our feelings. We want to improve it, so we need to know honestly what you think.
As we go along, I’m going to ask you to think out loud, to tell me what’s going through your mind. This will help us.
If you have questions, just ask. I may not be able to answer them right away, since we’re interested in how people do when they don’t have someone sitting next to them, but I will try to answer any questions you still have when we’re done.
We have a lot to do, and I’m going to try to keep us moving, but we’ll try to make sure that it’s fun, too.
You may have noticed the microphone. With your permission, we’re going to record the computer screen and what you have to say. The recording will be used only to help us figure out how to improve the site. It also helps me, because I don’t have to take as many notes. There are also some people who may be watching the screen in another room.
If you would, I’m going to ask you to sign something for us. It simply says that we have your permission to record you.
Do you have any questions before we begin?
Before we look at the web app, I’d like to ask you just a few quick questions. First, what’s your occupation?
Good. In a typical day, for instance, tell me what you do, at work and at home.
Have you ever downloaded scientific data via the Internet? How do you feel about downloading scientific data via the Internet?
Which data collections have you downloaded?
OK, great. We’re done with the questions, and we can start looking at things.
First, I’m just going to ask you to look at this web app and tell me what you think it is, what strikes you about it, and what you think you would click on first.
For now, don’t actually click on anything. Just tell me what you would click on.
And again, as much as possible, it will help us if you can try to think out loud so we know what you’re thinking about.
(Prompting question such as:) If you had to take a guess, what do you think it might be?
OK. What would you click on first?
Before you click on it, I have one more question. What about the steps near the top of the page—the ones with the numbers? What did you make of them?
(Follow up question such as:) Any reason why you didn’t pay much attention to them?
OK. Great.
OK, now we’re going to try something else.
Can you think of some data you might want to download from the AODN using this site? (If none - provide example based on real use cases)
So if you were going to download _____, what would you do first?
(Follow ups such as:) So what would you do?
Well, which one do you think you’d click on?
Why don’t you go ahead and do it?