) ISO 9241 standard, part 12, defines usability in terms of effectiveness, efficiency, and satisfaction.

• 0: (not in the tutorial - just my opinion) User abandons the site/product; or security hole
• 1: Severe. User cannot complete a task. User completes a task incorrectly without noticing (either could be a zero if the consequences are catastrophic)
• 2: Important. User is delayed or upset completing a task.
• 3: Cosmetic. (My personal opinion is that if it's cosmetic, it's a normal bug, not a usability bug)

Usability Tasks

• Don't use system-oriented terminology like register. Don't use slang. Don't use humor (it isn't funny when you can't do it). Don't use personal references ('send mail to your mother' - instead use 'a friend').
• Base tasks on user goals, not system features
• Get rid of hidden clues. Hidden clues include exact vocabulary matches between tasks and web site labels; the mere fact that a task can be achieved with the product at hand. Use indirect tests when necessary.
• To test something more realistic, use real loads: thousand-message inboxes, slow connections, etc.

Performing the test

• If a user has flailed for 8 minutes, they're not going to get it (this number is anecdotal, comes from apps testing, and may be too high)
• (I asked the group how long a typical test went). Rolf proposed 10 min for prep, 60 min for testing, 20 min for debrief and asked for a show of hands of who was doing something significantly different. Nobody raised their hand.
• Ask standard questions after each task for better results (people remember recent successes and discount the struggling once they succeed "I'm sure it would have been easier for someone else - it just took me a while to find it.")
• Q: what about tester in room versus tester in other room? A: I always do tester in room for social reasons - it's more humane.
• When doing a paper test, have a third person to manipulate the paper and "be the computer"
• Prototypes should be cross-functional efforts (dev, design, graphic, et al)
• some people take notes by computer instead of paper
• Nobody had any suggestions on how to better integrate help testing. Useful comments: "Testing whether or not somebody uses help is different from testing where or net help is useful." "Better to check logs to see if anyone uses help."
• Display tester limitations ("Sorry, I can't type that fast.") to make user more comfortable
• tell user in advance that some tasks may not be possible. Then, set an impossible task very early in the test. This helps reset the user's mindset back to something more realistic, namely that in the real world, you don't know in advance if the product you are using can do the task you need.
• Put a super-easy task first to make the user more comfortable
• avoid tasks that build on each other. If the user cannot complete one, the others are harder and (even if you have a fallback rigged) more stressful.
• Don't test your own baby as you won't be objective - very easy to insert your own bias at many points in the process.
• Good phrases
  • "Here's what the designer intended. How could we make that clearer?"
  • "I must be stupid." "Everything I've seen you do makes perfect sense to me. It's just that this system isn't designed that way."
  • "Don't tell me, show me." (Tip 4: avoid user opinions.)
• Bad phrases
  • "Yeah, we thought about that."
  • "Yeah, we tried that and it didn't work and here's why..."
  • "Lab," "Experiment," "Research," "Test Subject." Instead, "Usability Evaluation"
  • "I'll show you a good way to do that." Instead, "How should we change the interface to avoid the problems you had."
• When you hint, you will help more than you intend. Some labs have a strict "no hints" policy and just move people on to the next task.
• Discount Usability Testing Mr. Molich advocated this methodology. The idea is, "strive to optimize the cost/benefit ratio."
• Should you videotape? In general, no, because analyzing results takes much too long. Instead, take notes while it happens. Ask the user to stop for a minute if a lot of stuff happens at once. You will miss things, but it's unlikely you'll miss anything important (if it's important, it will happen more than once). Reasons to videotape include: proving your results to people who don't think a user would actually say X. Emphasizing the consequence of usability problems.

Testing with Advanced Users

• When testing with advanced users, the user should provide the task whenever possible.
• Have users bring typical tasks. "What are the last three things you did with this product?" "Can you defer your normal tasks for a few days and then do them here in the test?"

Reporting

• The users of a usability test are the developers. Make sure that the usability test and results are themselves usable.
• Usual result of a usability test is a report: 8-12 pages, no more than 40 defects. Doesn't seem to be much iterative usability testing.
• some people advocate tracking usability problems in the bug system
• include severity for each issue.
• Report should include: description of specific problem. Description of principle violated. Description of general problem, and notes on where dev/test should look for other instances of same problem.
• Reports should have plenty of positive items - perhaps one positive item per problem. (N.B. In my opinion, this is a structural problem. Should a usability test produce a list of problems to fix, or a validation that the site enables users to accomplish their goals? This is analagous to the same problem in quality assurance/control: you must start by testing all the things that a program is supposed to be able to do, and marking (as bugs) where it cannot; then, you can go digging for bugs. So, how can you report results in terms of validation - "10 basic users tasks; most users can accomplish N tasks with avg time X and error rate Y." Mr. Molich suggested a color-coded results grid.)
• The point of usability testing is to find and correct usability problems. Therefore, reporting is as important as testing.
• Having developers observe tests is crucial. It will help developers think like users and thereby code more usable functions. Testing should be located somewhere convenient to developers, not to users or the tester. (Although I agree on a pragmatic level, I'm a little uncomfortable endorsing gratuitously developer-centric thinking.)
• There was discussion of the KJ method and consensus-building among developers. I didn't take good notes because it isn't really relevant our continuous usability improvement process. (it's a way to sit down after a test and get everyone to agree what to fix)
• Should problem descriptions include proposed solutions? Yes, because it helps developers and helps show that the reporter understood the problem. No, it short-circuits other possible solutions.

Assorted notes

• eye-tracking isn't necessary, because there's still more than enough low-hanging fruit - problems that can be found without eye-tracking
• A typical test of a web site, from scratch, is ~US$5,000 - $10,000, 50-80 hours.
• Beware of habits that produce data you never use
• A useful error message always says what to do next.
• You should be able to repeat an error message to the user's face.
• videotape yourself to see how you do as a test manager
• Have yourself tested by a usability consultant (coach) every two years
• Controversial rule of thumb: "Five users can find 75% of all problems, and since you'll never be able to fix them all anyway, that's enough." Instead, "The first five users will probably find 75% of the problems for a particular task set for a particular version."
• Tip #27: "Keep the number of participants low so that you are just as interested in your final participant as in the first participant."
• Mr Molich's conclusion from CUE: Many of the problems are caused by lack of knowledge about elementary rules for usable design. More focus on prevention; less focus on testing.

 INDEX