Cognitive Walkthrough

Taylor just signed up for a new online project management tool to boost productivity when working with her team. Eager to get started, Taylor creates an account and is immediately met with different features, buttons, and menus. Almost instantly, a pop-up message suggests following a tutorial before navigating the tool alone. Thinking “How difficult can this be?” Taylor closes the pop-up. She believes it’s better to play around for a while than waste her time going through a tutorial and many instructions.

A cognitive walkthrough is a usability evaluation method used in human-computer interaction and user experience (UX) design. It involves a systematic analysis of a user's thought process while interacting with a product or system. The goal is to identify potential usability issues and to understand how easily users can accomplish their tasks. It’s conducted with people like Taylor in mind—those first-time users who feel that they learn better by actually navigating through a system than by reading a manual. 

A user like Taylor expects to be able to go through the system or product by following cues that guide them through the different tasks available. For example, if Taylor encounters a “Create New Project” button they anticipate a structured process, typically involving a series of steps or fields. This expectation is aligned with Jakob’s Law of Internet User Experience, which states that users bring with them expectations based on their previous interactions with other interfaces.

Cognitive walkthroughs aim to learn about the user's logic and ways of thinking so a business can facilitate intuitive navigation. They also intend to create tailored experiences that serve as reminders of how to perform different tasks.

Let's go through the steps of a cognitive walkthrough, using Taylor’s scenario as our guide.^1,2

1. Identify a goal for the test: Begin by defining a clear goal for a key section or feature of your interphase. For example, the objective might be for Taylor to successfully initiate a new project and add team members. Goals should be measurable (was the objective achieved? After how many tries? How long did it take?).

2. Assemble the team: You can use your own team of UX specialists, engineers, or domain experts as participants. But ideally, you should be choosing people who haven’t interacted with the product—to mimic Taylor’s experience—and avoid biases. It’s beneficial to include real users or participants who closely resemble the target demographic to gain insights into a wider range of user interactions and experiences.

3. Assign tasks/actions: Outline realistic tasks that range in difficulty, some that a new user might attempt and others that require a bit more knowledge (to see how experts interact). As the researcher, you should already know every single step that has to be taken to evaluate if the participant has gone through all of them. 

• Iterative testing, where the product is evaluated, refined, and tested again, is crucial in UX design as it allows for continuous improvements and the ability to catch new issues as changes are made.

4. Perform the walkthrough: For each task, you should ask the following four questions

• Will the user try and achieve the right outcome?: Determine if it’s clear to the user what to do. Typically, we give them goals rather than direct tasks. So, for example, ask: “How would you set up a new project with your team,” rather than: “Find the Create New Project button.”
• Will the user notice that the correct action is available to them?: Analyze if the system provides adequate cues. Is there only one route to complete the task? Are they faced with specific challenges?
• Will the user associate the correct action with the outcome they expect to achieve?: Consider if the system’s feedback and labeling are intuitive.
• If the correct action is performed, will the user see that progress is being made towards their intended outcome?: Evaluate if the system provides appropriate feedback to indicate that the user is on the right path.

5. Document findings: You should record if participants pass or fail and what made this result happen. Where did they struggle? What was easier for them?

• Gathering detailed feedback from participants after the walkthrough can offer additional insights that the structured test might not reveal. This could include their subjective satisfaction, perceived ease of use, and suggestions for improvement.

Once again, the whole point of cognitive walkthroughs is to evaluate learnability, especially for systems or products with new or unfamiliar workflows and functionalities.

User Experience (UX): UX describes the overall experience a user has when interacting with a product, system, or service. The goal of UX design is to create simple, effective, relevant, and overall enjoyable experiences for the user. UX design considers the why, what, and how of a product from the user's perspective.⁴

Usability: Refers to the extent to which a product or system can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction.

Usability Testing: Technique used to evaluate a product or system by testing it on users to observe and measure how simple and intuitive it is.

Learnability: How effortlessly can users complete a task upon their initial interaction with the interface, and how many repetitions are required for them to achieve proficiency in that task. 

Mental Models: How individuals process information and form expectations based on their understanding and experience. They guide our perception and behavior.⁵

In the early 1990s, personal computers started to become central to everyday life, including homes and workplaces. The need for intuitive digital products became an urgent matter for designers. It was during this time the cognitive walkthrough was developed. A team of researchers from the University of Colorado Boulder (Cathleen Wharton, John Rieman, Clayton Lewis, and Peter Polson), recognized the importance of usability and usability testing. With this in mind they came up with a method that would systematically evaluate a digital product’s usability without needing a significant amount of resources and training. Their goal was to design a technique that would allow designers and developers to anticipate and find potential usability issues before their release.

The process was actually developed with walk-up-and-use interfaces in mind (like ATMs), where users must understand and can use the interface without any prior training or experience. The science behind the method considers behavioral science and how people pick up interface knowledge through experimentation and problem-solving.⁶

As with most design methods, cognitive walkthroughs were modified and developed by the original authors as it was implemented in the real world. The questions made during these sessions were tweaked a couple of times through the papers Rieman, Lewis, Polson and Wharton published. Developments made the methodology less time-consuming and tedious. The most recent published update from the original authors is a 2002 paper titled “Cognitive Walkthrough for the Web” which includes the questions mentioned in the first part of this article.⁶

Cathleen Wharton: Doctor in Computer Science with certification in Cognitive Science known for developing and refining the cognitive walkthrough method. Her research focussed on human-computer interactions and ways to improve usability and develop practical strategies for businesses. ⁷

John Rieman: Computer Science PhD, and member of the team that developed the cognitive walkthrough.

Clayton Lewis: Computer Scientist known for his research on evaluation methods and interface design. Another creator of the cognitive walkthrough, he is affiliated with the Institute of Cognitive Science at the University of Colorado Boulder.⁸

Peter Polson: Professor of Psychology and Faculty Fellow at the Institute of Cognitive Science at the University of Colorado Boulder. His research and academic career focuses on the learning and transfer skills required to use computer-based systems.⁹

One of the advantages of cognitive walkthroughs is that they can be somewhat more resource-efficient compared to other usability testing methods. By using your internal team, you reduce the costs associated with recruiting external participants. Internal teams can be unbiased so long as they haven’t encountered or had a hand in developing the system or product. However, cognitive walkthroughs still require careful planning, and it may be more beneficial to perform the walkthrough with external participants.

This method is notably effective in educational settings, such as software design courses, for evaluating students' ability to create user-friendly interfaces, as it directly measures their usability knowledge and skills. It's also important to mention that although the original cognitive walkthrough often uses internal teams, recruiting external participants is permitted. Doing so can provide your team with more accurate and better findings. For example, you could run one session with your internal team, refine the interface, and then test it with external users. Their feedback could not only improve your product but also refine and add knowledge to subsequent cognitive walkthroughs. This approach, however, requires more planning and resources.

Additionally, the process and the questions that are asked during these sessions are quite simple and intuitive. This method requires minimal training as the questions remain the same over each situation, the individual conducting the interview needs minimal research skills. 

This method can integrate as much qualitative and quantitative data as the team is looking for. The different questions required for each task can provide qualitative data that includes asking for thoughts and feelings. Additionally, a cognitive walkthrough can integrate quantitative data by calculating statistics such as completion rates, error rates, and time spent on a task. Also, cognitive walkthroughs don’t have to take place once the product is finished, you can conduct them using prototypes.

Another great thing about this method is that you can easily combine it with other UX design tools or methods. Combining some tools can create a more comprehensive understanding of the interaction between your product and your user. For example, using cognitive walkthroughs with card sorting can provide valuable insights. Card sorting helps you understand how users categorize information, which can inform the intuitive structure of your interface. In general, regularly employing cognitive walkthroughs can contribute to a culture of continuous improvement and potentially reduce development costs.

Using your own team sounds great and cost effective but you have to be careful about possible biases. Maybe everyone in your business, regardless of them working on different products, uses specific jargon or abbreviations that aren’t common or easily understandable if you don’t work there. This familiarity could accidentally influence the assessment.  However, a potential solution is to create user personas or use other tools that might help you empathize with your users and imagine how they would interact with the interface.

Additionally, we have seen that this method is appropriate for new systems and that it’s useful for identifying surface-level usability issues. However, it may not be ideal for complex interfaces or fully capturing the emotional reactions of various users interacting with your product. Although the four questions included in the sessions could potentially open up a deeper conversation, this method doesn’t prioritize qualitative research. This might lead to an inaccurate portrayal of user experiences.

Finally, this method involves considering specific tasks and the sequence of actions required to complete them. In reality, some users might portray more exploratory actions or routes or interact with systems in unpredictable ways. A lot of users learn with trial-and-error situations which doesn’t have to be considered a bad thing. We’re often focused on productivity, which could potentially introduce a bias. This emphasis on efficiency may lead us to overlook the value of exploratory learning.

In conclusion, while the cognitive walkthrough is a powerful tool for improving usability with its systematic approach and efficiency, awareness of its limitations and potential biases is crucial. Balancing this method with other forms of user research and testing can help create a more comprehensive understanding of the user experience, ensuring that products are not only usable but truly resonate with their intended audience.

Edinburgh's Transit App for Newbies¹⁰

The city of Edinburgh has an extensive public transport system facilitated by Lothian Buses through a user-centric app. The whole point of the app is to provide real-time information on bus and tram schedules, stops and routes. It’s a crucial and great tool for newcomers who are being introduced to the city’s public transit network. Because of this, in 2018, a cognitive walkthrough was conducted to observe the app’s learnability for first-time users. The main focus was the task of finding the fastest public transport route. The walkthrough identified key steps a user would follow, from entering a destination to selecting the fastest route. Intuitiveness and efficiency were top criteria.

The findings of the study showed that the app was indeed intuitive and used effective visual cues that helped a new user understand the interface. However, challenges were identified in how to search results and route options were presented. For example, when you looked up a place in the search area, bus routes appeared in the results first. This confused users who were just trying to find a location, not necessarily a bus route. It also lacked separation between the total travel time and walking time between parts of the journey. This lack of clarity made it hard for users to figure out how long their trip would actually take.

The study suggested several improvements to the app’s usability such as enhancing the display of search results, improving the clarity of route selection feedback, and double-checking the way travel time was presented to better communicate the breakdown of transit and walking times. 

By adopting a user-centered design perspective, this study contributed to the ongoing development of accessible, intuitive public transit apps that cater to the diverse needs of a city's residents and visitors, reinforcing the importance of empathy and clarity in digital tool design for public services.

Empathy Mapping

Dive into a step-by-step guide that illuminates how understanding the needs, behaviors, and emotional experiences of users can lead to more inclusive, engaging, and successful products.

1. Interaction Design Foundation - IxDF. (2021). How to Conduct a Cognitive Walkthrough. Interaction Design Foundation. Retrieved March 29, 2024 from .https://www.interaction-design.org/literature/article/how-to-conduct-a-cognitive-walkthrough
2. Dovetail Editorial Team. (2023). Conduct a cognitive walkthrough in 9 steps. Retrieved March 29, 2024 from https://dovetail.com/ux/cognitive-walkthrough/
3. Arhipova, A. (n.d.). Don't Make Me Think: 20 Thoughts on Usability by Steve Krug. Retrieved March 29, 2024 from https://blog.tubikstudio.com/dont-make-me-think-20-thoughts-on-usability-by-steve-krug/
4. Interaction Design Foundation. (n.d.). UX Design. Retrieved March 12, 2024, from https://www.interaction-design.org/literature/topics/ux-designMolich, R. (n.d.). About Me – Rolf Molich. DialogDesign. Retrieved March 27, 2024, from https://www.dialogdesign.dk/about-rolf-molich/
5. Clear, J. (n.d.). Feynman Technique. Retrieved March 1, 2024, from https://jamesclear.com/feynman-mental-models
6. Salazar, K. (2022). Evaluate Interfase Learnability with Cognitive Walkthroughs. Retrieved March 12, 2024, from https://www.nngroup.com/articles/cognitive-walkthroughs/#:~:text=Cognitive%20walkthroughs%20are%20a%20proven,usability%20testing%20may%20be%20limited
7. Cathleen Wharton. (n.d) About [LinkedIn page]. LinkedIn. Retrieved March 12, 2024, from https://www.linkedin.com/in/wharton/
8.  University of Colorado Boulder. (n.d.). Clayton Lewis. Retrieved March 12, 2024, from https://www.colorado.edu/cmci/people/information-science/clayton-lewis
9. University of Colorado Boulder. (n.d.). Peter G. Polson. Retrieved March 12, 2024, from https://www.colorado.edu/psych-neuro/peter-g-polson
10. Velásquez, S. (2018). Evaluation of a user interfase using cognitive walkthrough (real case). Retrieved March 29, 2024 from https://medium.com/@sosegon/evaluation-of-a-user-interface-using-cognitive-walkthrough-real-case-ac94014003d8