User testing cheatsheet: before, during, and after
20+ pieces of practical advice that will help you conduct a successful user test.
In my previous post, I wrote about the importance of prototyping and how it can increase the chances of product success when applied timely. It led me to the idea of making a series of publications about digital product testing, which will be based on several keynotes I prepared for knowledge-sharing sessions. The topics are the following:
Writing a script and tasks for testing.
Tips for creating a prototype efficiently.
User testing cheatsheet: Before, during, and after.
Analyzing the results and delivering actionable insights.
And I will start, you guessed it, with the third one.
But first, a little side note on User testing vs. Usability user testing. Usually, designers and researchers use these terms interchangeably. But for me, they are a bit different:
User testing is testing users’ needs. The main question here is: ‘Do they need this solution?’.
Whereas, Usability testing is testing the interface. We try to understand: ‘Can they use my app?’.
Although this guide will be suitable for the user and usability testing, I use user testing for simplicity.
User testing cheatsheet
Pre-test communication
Contact your users telling them you want them to help you evaluate the effectiveness of a product you’re working on.
Send them just a short description of your product, ideally a piece of marketing description or ad.
Tell them that no preparation is needed.
It’s good practice to write to them 24 hours before the session to ensure everything is in place and they don’t forget about it.
User testing inventory
Consent form
A place to take notes during the script.
Your test script
Pre-test checklist
Post-test checklist
Introduction
Test tasks
Debrief questions
Outro
Test tasks for the user
Context or Short description of the product
Tasks (printed or displayed separately from each other)
Before the test (for the facilitator)
Go through the script by yourself one more time.
Set up everything needed for the tasks:
Devices
Browser with default settings and incognito mode
Test accounts and other credentials if needed
If it’s a remote test:
Prepare all the links with prototypes or websites.
Set up your working environment so you can comfortably see the script, send resources to the participant, and see the participant and a screen he is sharing.
Before the test (with the participant)
Consents and NDAs
Sort out things with formal consent and NDAs.Screen recording consent
Ask a content for screen recording. Explain that it’s only for the internal purposes of the product team.Testing product, not a participant
Explain that you are testing a product and not a participant and that all the problems you will find are not their fault but the product's flaws. And finding them will make the product better.Use the ‘product is not yours’ trick.
You can tell it’s not your product, and you are just a researcher, even if it’s a lie.Prototype limitations
Tell the participant if there are limitations in your prototype and something is not expected to work.Think aloud
Before you start the actual testing, explain that you want participants to think their thoughts aloud.‘Task finished’ rule
Explain that a participant should indicate that he has finished the task by saying ‘task finished’ or ‘task complete.’Intro questions
Ask intro questions about expectations, usage frequency, the product's role, etc.
During the user test
No reactions and emotions
Don’t assess, comment, or react to what the participant does. You should be silent and unemotional. Otherwise, the experience won’t be realistic and will confuse the participant.Don’t explain the design.
If you want to say “Yes, but…” instead, say, “Tell me what you’re doing right now.” You’ll then get into behavior without affecting the process too much. If you want to explain, then do it after the session.Don’t answer questions.
If you get “How do I get back to the beginning?” you respond: “How do you think you get back to the beginning?”.Don’t show how the task should be done.
If the participant is in a stalemate, you should silently guide the participant to the start of the next task or default point.Don’t ask questions
Interviews can be done before or after the testing.Don’t ask about likes
Don’t ask participants if they like the design.
After the test
Ask debriefing questions.
Ask the ‘Is there anything you wanted to discuss that we didn’t discuss or ask questions.
Ask about people who he can recommend for doing another session.
Thank the participant.
Analyze the session as soon as possible because you can lose essential details if you delay it.
That’s a pretty long list! As you get more experienced, you understand how to apply those rules efficiently, for example, you will be able to omit some parts of the introduction without affecting the quality of the test. But as you start, you should try to follow all the recommendations, it will be hard because this is not how people usually communicate, and you will feel the urge to communicate with participants naturally. Rewatch your recordings to see your mistakes, or ask your teammates to give you feedback.
Additional resources:
Think Like a UX Researcher [Book by David Travis and Philip Hodgson]
Usability Testing 101 [Norman Nielsen]
Your Guide to Impactful Digital Product Testing [People Nerds]