Pluralistic walkthrough

Procedure

Walk-Through Team

• representative users
• * Users should be representative of the target audience, and are considered the primary participants in the usability evaluation.
• human factors engineers/professionals
• * Human factors professionals usually serve as the facilitators and are also there to provide feedback on the design as well as recommend design improvements.
• * The role of the facilitator is to guide users through tasks and facilitate collaboration between users and developers.
• product developers
• * Product developers answer questions about design and suggest solutions to interface problems users have encountered.
• * It is best to avoid having a product developer assume the role of facilitator, as they can get defensive to criticism of their product.

  Materials

• Room large enough to accommodate approximately 6-10 users, 6-10 developers and 2-3 usability engineers
• Printed screen-shots put together in packets in the same order that the screens would be displayed when users were carrying out the specific tasks. This includes hard copy panels of screens, dialog boxes, menus, etc. presented in order.
• Hard copy of the task scenario for each participant. There are several scenarios defined in this document complete with the data to be manipulated for the task. Each participant receives a package that enables him or her to write a response directly onto the page. The task descriptions for the participant are short direct statements.
• Writing utensils for marking up screen shots and filling out documentation and questionnaires.

• Assume the role of the user
• To write on the panels the actions they would take in pursuing the task at hand
• To write any additional comments about the task
• Not flip ahead to other panels until they are told to
• To hold discussion on each panel until the facilitator decides to move on

  Tasks

1. Participants are presented with the instructions and the ground rules mentioned above. The task description and scenario package are also distributed.
2. Next, a product expert gives a brief overview of key product concepts and interface features. This overview serves the purpose of stimulating the participants to envision the ultimate final product, so that the participants would gain the same knowledge and expectations of the ultimate product that product end users are assumed to have.
3. The usability testing then begins. The scenarios are presented to the panel of participants and they are asked to write down the sequence of actions they would take in attempting to complete the specified task. They do this individually without conferring amongst each other.
4. Once everyone has written down their actions independently, the participants discuss the actions that they suggested for that task. They also discuss potential usability problems. The order of communication is usually such that the representative users go first so that they are not influenced by the other panel members and are not deterred from speaking.
5. After the users have finished, the usability experts present their findings to the group. The developers often explain their rationale behind their design. It is imperative that the developers assume an attitude of welcoming comments that are intended to improve the usability of their product.
6. The walkthrough facilitator presents the correct answer if the discussion is off course and clarifies any unclear situations.
7. After each task, the participants are given a brief questionnaire regarding the usability of the interface they have just evaluated.
8. Then the panel moves on to the next task and round of screens. This process continues until all the scenarios have been evaluated.

Characteristics of Pluralistic Walkthrough

• The main modification, with respect to usability walkthroughs, was to include three types of participants: representative users, product developers, and human factors professionals.
• Hard-copy screens are presented in the same order in which they would appear online. A task scenario is defined, and participants confront the screens in a linear path, through a series of user interface panels, just as they would during the successful conduct of the specified task online, as the site/software is currently designed.
• Participants are all asked to assume the role of the user for whatever user population is being tested. Thus, the developers and the usability professionals are supposed to try to put themselves in the place of the users when making written responses.
• The participants write down the action they would take in pursuing the designated task online, before any further discussion is made. Participants are asked to write their responses in as much detail as possible down to the keystroke or other input action level. These written responses allow for some production of quantitative data on user actions that can be of value.
• It is only after all participants have written the actions they would take that discussion would begin. The representative users offer their discussion first and discuss each scenario step. Only after the users have exhausted their suggestions do the usability experts and product developers offer their opinions.

  Benefits and Limitations

Benefits

• Early systematic look at a new product, gaining early performance and satisfaction data from users about a product. Can provide early performance and satisfaction data before costly design strategies have been implemented.
• Strong focus on user centered design in task analysis, leading to more problems identified at an earlier point in development. This reduces the iterative test-redesign cycle by utilizing immediate feedback and discussion of design problems and possible solutions while users are present.
• Synergistic redesign because of the group process involving users, developers and usability engineers. The discussion of the identified problems in a multidisciplinary team will spawn creative, usable and quick solutions.
• Valuable quantitative and qualitative data is generated through users’ actions documented by written responses.
• Product developers at the session gain appreciation for common user problems, frustrations or concerns regarding the product design. Developers become more sensitive to users’ concerns.

  Limitations

• The walkthrough can only progress as quickly as the slowest person on each panel. The walkthrough is a group exercise and, therefore, in order to discuss a task/screen as a group, we must wait for all participants to have written down their responses to the scenario. The session can feel laborious if too slow.
• A fairly large group of users, developers and usability experts has to be assembled at the same time. Scheduling could be a problem.
• All the possible actions can't be simulated on hard copy. Only one viable path of interest is selected per scenario. This precludes participants from browsing and exploring, behaviors that often lead to additional learning about the user interface.
• Product developers might not feel comfortable hearing criticism about their designs.
• Only a limited number of scenarios can be explored due to time constraints.
• Only a limited amount of recommendations can be discussed due to time constraints.