Write effective usability testing tasks with hypothesis
In the last post, I teased a series of articles on digital product testing. And while I was working on ‘Tips for creating a prototype efficiently’ and ‘Analyzing the results and delivering actionable insights,’ I realized that the hypothesis-driven design was at the core of the processes I described. And it inspired me to start writing about hypothesis-driven design and how it can help add structure to the design process. But the topic appeared so broad that two weeks was insufficient to finish it.
Nevertheless, I didn’t want to break the biweekly cadence of my blog, so I decided to post some short notes on writing usability testing tasks, where relying on the correct hypothesis plays an important role.
Anyways, let’s get right into business.
Always start with a hypothesis
I strongly support that every solution we build should have a hypothesis. It will guide the design and development process and help you understand if you have built the right thing.
When crafting usability testing tasks, you should always start with the user goal; the Lean UX hypothesis statement is a perfect tool.
We believe that [ doing this/building this feature/creating this experience ]
for [ these people/personas ]
will achieve [ this outcome ].
We will know this is true when we see
[ this market feedback, quantitative measure, or qualitative insight ].
Outcome = user goal. For every outcome, you should create user actions needed to achieve it. And then, for every user action or set of user actions, create a task. I usually make a 3-column table: hypothesis, actions, and tasks.
Example for a flights metasearch app:
Hypothesis:
We believe that [adding a prize alert feature]
For [budget tourists ]
Will [help them find a cheaper flight when they are available]
Actions:
Find a flight
Sign up / Sign in for the app
Click ‘Notify me if the price drops.’
Receive notification about prize drop
Buy a ticket
Test task:
You want to surprise your family with a visit over Christmas, but your budget is limited to $400.
Make it somehow so that website lets you know about a suitable prize for a trip from Detroit to Seattle from December 22, 2022, to December 26, 2022.
Tips for creating a test task
Set a scenario
You need to set the context for the task or even for the whole script. Usually, if you are testing with your target audience, you don’t need a lengthy introduction. But if you are doing a guerrilla test, a good scenario can help the participant to get into the role.
Make it realistic
A task should be realistic, something users would do in real life.
A task should have success criteria, a state, or an answer in which a task is considered completed.
Close-ended and open-ended
Close-ended tasks are tasks where you ask users to do something that has one clear, ideal answer.
Open-ended don’t have a clear success state, but sometimes this type of task can be helpful when you need to learn how users think about a task when unconstrained.
Don’t lead the witness
Tasks should not use the terminology of the interface
Not good: “Put 3 books in your shopping cart, then purchase them using Standard Shipping.”
Better: “Choose three books and buy them, ensuring they can arrive by next Wednesday.”
Task ordering
Progress from easier to more complex.
Cover a range of critical task types (e.g., browse, search, buy).
Can include open-ended and closed-ended tasks.
Should include up to eight tasks not to overwhelm the participant.
Be careful to avoid ordering effects. The previous task doesn’t give a hint about the next task.
Examples :
Task 1
Your manager asks you to help her plan a few trips for the company. She has heard of a website called YourWebSite that can help and encourages you to use it.
Plan a round trip from Detroit to Atlanta for under $280.00 from November 16, 2022 to November 19, 2022. Email the itinerary to yoursitetest@gmail.com
Success Criteria**: Ticket with appropriate price and dates found and emailed. On November 16, 2022 there were about 10 flights that met the criteria. If there are none, raise the price threshold so that about 10 flights are available.
Task 2
Your manager wants to join the Chicago team in New York (your office is in Detroit), but then she wants to go to London for a week and then return to Detroit. She plans to fly business class for the entire trip. What is the lowest price for her trip?
Success Criteria**: $5358 *Criteria created on November 3, 2022, valid during recorded test sessions.
Be creative with open-ended tasks
Typically our scripts are filled with close-ended tasks, which is a reliable way to test a hypothesis. But open-ended tasks can help discover more insights about your users and find unexpected or underserved goals.
David Travis in his article ‘Creating usability test tasks that really motivate users,’ gives an interesting categorization of test tasks, and the open-ended tasks described by him deserve the most attention:
Self-generated tasks.
Part self-generated tasks.
“Skin in the game” tasks.
Self-Generated Tasks
With this task, you ask participants what they expect to do with the product (before you show it to them), and then you test out that scenario. It can be used when you don’t know what people want or expect to do on your site.
If we take an example of our flight’s metasearch app:
You begin the session by interviewing participants and asking what they expect to be able to do with the app
You might hear something like ‘Build an itinerary with many flights’ or ‘being notified of a better ticket price.’
You then take each task in turn and ask the participant to be more specific.
Part Self-generated
These tasks work well when you have a good idea of the main things people want to do with the product, but you’re less sure of the detail. With this task, you define a primary goal (for example, ‘Book a flight ticket’) and then ask the participant to fill in the gaps. You ask what details are essential to the participant instead of priming them.
“Skin in the Game” Tasks
A problem with usability test tasks is that you want participants to carry out the tasks as realistically as possible. But there’s a big difference between pretending to do a task and doing a task.
The easiest way to do this with a commercial website is to give participants a redeemable voucher to spend during the test or reimburse their credit card after they have made a purchase.
All in all, it is essential to start with a hypothesis. It will guide you and make sure you are doing the right things to find answers to the right questions. Open-ended tasks can be a good fit when you need to discover new user needs and new ways they use the product.
Other resources:
Evaluating Designs with Users [MichiganX UX504x]