Measuring – Pavvy Designs

Usability is a user experience attribute that indicates how intuitive and easy to use a website or product is. Measuring usability allows you to learn if users can accomplish their goals with your product or if they encounter issues that you need to fix. Usability is as much related to the ease of use of a website as it is to how it looks and feels. In this context, aesthetics, useful content, and credibility matter too.

There are a few ways to measure the usability of your product. You can do a System Usability Survey (SUS), analyze Google Analytics data such as conversions or bounce rates on a live website, or run usability testing with prototypes and record metrics that help you improve the user experience.

The advantage of doing usability testing when your product or feature hasn’t yet been released is the ability to detect and fix issues early on. Spending time implementing a faulty design or losing customers over preventable errors can be avoided by running usability testing during the design process.

For a long time, measuring usability metrics had been a painful process, mostly done manually by aggregating the data from every task. If you’re a Maze user, you already know that we record and give you these metrics automatically once you finished testing your prototype.

In this article, our goal is to go through a list of essential usability metrics and learn why they’re important for the user experience.

The value of measuring usability

You may ask, ‘Why do I need to record usability metrics when I can just ask my users what they think about the design?’ Simply put, because of the Aesthetic-Usability Effect. This effect occurs when users perceive aesthetically pleasing design as more usable. On these occasions, measuring usability metrics allows you to objectively determine how your design performs as opposed to relying on a subjective understanding of what users say.

Additionally, measuring usability has benefits that go beyond what you’d initially expect. It can help you to:

Find usability issues that aren’t easy to detect. When you’re knee-deep into a design project, you’re too involved to identify issues that naturally slip when developing a product. Here’s where usability metrics come in. For example, when you run usability testing and record a low success rate for task completion or a high misclick rate on a particular screen—it’s evident that your users are finding it hard to complete their tasks. Such metrics can pinpoint what you need to improve and help you avoid a poor user experience.
Track progress between design iterations. Measuring the usability of your design allows you to set goals and track progress when you iterate on an existing feature or product. The metrics are easily comparable: you can see how the feature performed last time, so it’s easier to know if the new design is better or worse.
Get buy-in on design decisions. Sometimes arguing your way around your design choices doesn’t help. You want to include a navigation bar, whereas your client or manager doesn’t. When design choices are put to the test and measured with actual data—then arguments become less a matter of preference and more about what’s working or not.

3 categories of usability metrics you should track

1. Completion metrics

Completion metrics track if users can successfully complete a task. Each task can be deemed a success or a “fail,” and in some instances, an indirect success.

Note: when we refer to ‘fails’ in usability testing, we mean that the task failed, as we’re testing the design and not the users.

Direct Success

What it is: A metric that shows if the user completed the task successfully with the design. In Maze, you can set the expected flow(s) you think users will take through your prototype, and if it coincides with the one users take, the result is a direct success.

Why it’s important: If the majority of your users can complete their tasks using the flow you designed, then that’s a good measure of usability. When assessing if the completion rate for your tasks is good or bad—context matters. However, studies have shown that, on average, 78% is a good completion rate.

Indirect Success

What it is: When a user completes a task yet gets lost during the process or takes another viable path to completion, this is measured as an indirect success.

Why it’s important: Sometimes, the flow you designed might not be the one users take through the product. Usability testing can reveal if this happens and expose the most optimal flow of your design.

Fails

What it is: Fails are measured when a user doesn’t complete the task with the design. Maze records both Give ups (when users give up the task) and Bounces (when users leave the test altogether).

Why it’s important: Fails are an indicator that your design needs changing. Excluding reasons such as a user not understanding the usability task or out of control interruptions—measuring fails allows you to locate if there are any significant issues you should fix. In a live product, they may lead to a high bounce rate or exits from a page.

2. Duration metrics

This category of metrics relates to the time users to take to complete a task or the time they spend on a particular screen in the design. Time spent is an essential indicator of usability as more time might mean users are having a hard time finding what they need on a page, are lost in the flow, or don’t understand how to use the product.

Time on screen

What it is: Time on screen measures how long a user spends on a particular screen in your prototype.

Why it’s important: Except for pages such as blog posts—where the user might naturally spend more time, a long time spent on screen is an indicator that the user can’t find what they are looking for. When time on screen is correlated with metrics such as the misclick rate for that screen, it can reveal issues with the interface, labels, or layout of the page.

Time on task

What it is: Time on task is a metric that measures the duration users take to complete a task.

Why it’s important: As with time on screen, the longer the duration, the bigger the probability that users can’t find what they’re looking for or are lost trying to complete the task they were given. When you record a long time spent on a task, it’s important to identify the reason: Is it because users are getting lost through the design, and so, you might need to rework the navigation? Or perhaps it’s because they didn’t understand the task, in which case rephrasing the question might produce better results. The context in which the metric was recorded matters.

3. Errors

Errors are actions such as clicks or taps that users do on your website or app. Errors can easily be measured by tracking the misclick rate or analyzing heatmaps of your prototype screens.

Misclick rate

What it is: The misclick rate is the average number of misclicks outside hotspots or clickable areas in a prototype. They usually occur because users can’t find what they’re looking for, or expect something to act similarly to other websites.

Why it’s important: A high misclick rate indicates that users are confused, or that part of your design is misleading. This might lead to users getting lost or even bouncing on a live website. According to Jakob’s Law, users spend most of their time on other websites, so they expect yours to function similarly. Measuring the misclick rate allows you to detect if there’s any part of your product that needs an immediate redesign.

Measuring usability through satisfaction

Tracking usability metrics allows you to quickly determine how people are using your website, if there are issues you need to fix, and how your design performs over time.

Apart from metrics about what users do on your website, you can also measure what they think about your product by asking a task satisfaction question, such as the Single Ease Question (SEQ). The answers to these types of questions will help you better understand their experience when completing usability tasks. Plus, they can reveal some valuable insights about how to improve your design and explain the recorded metrics.

Task satisfaction metrics can be collected with a rating scale after each task. You can ask users: “On a scale of 1-7, how was your experience completing this task?” and measure user satisfaction for each task in your test.

Alternatively, you can do a System Usability Scale at the end of the test. This survey provides a quick way to measure usability by asking users to rate ten items on a scale from zero to 5, ranging from ‘Strongly Agree’ to ‘Strongly Disagree.’

There are a variety of questionnaires to measure the user experience of your product—thinking about which one best suits your user research needs will help you get the most out of usability testing.

Conclusion

Many times, usability gets side-tracked and becomes something that will be dealt with later on. Tracking the usability of your product with metrics allows you to have a clear understanding of the experience you’re providing to your users, and improve it over time.

With tools like Maze, usability metrics are measured and aggregated into actionable results, which allows you to act instantly on the data you record. That makes it painless to keep track of how your design’s usability progresses, detect issues, and improve your users’ experience.

One of the most important steps in developing any digital project is defining what success means is. Because we have the possibility to measure virtually everything, it’s easy to get caught in the trap of measuring all the wrong things. We often throw these numbers around and, of course, the bigger the number is, the more impressive it is. We tend to do this with almost everything nowadays! Page views, email subscribers, instagram followers, funding raised, revenue, and even numbers of employees. Big is better, right? Don’t get me wrong, in some cases big is better – like profit, but in some cases it’s just vanity metrics.

The only metrics that entrepreneurs should invest energy in collecting are those that help them make decisions. Unfortunately, the majority of data available in off-the-shelf analytics packages are what I call Vanity Metrics. They might make you feel good, but they don’t offer clear guidance for what to do.

When you hear companies doing PR about the billions of messages sent using their product, or the total GDP of their economy, think vanity metrics. But there are examples closer to home. Consider the most basic of all reports: the total number of “hits” to your website. Let’s say you have 10,000. Now what? Do you really know what actions you took in the past that drove those visitors to you, and do you really know which actions to take next? In most cases, I don’t think it’s very helpful.Eric Ries on Tim Ferriss

With all of the tools available showing us all the charts, how do we know which are useful and which are just vanity metrics? Hint: the answer is right there in the question — “u-s-e-f-u-l.” Metrics that help us make decisions are useful, metrics that can’t be used to make actionable steps are vanity metrics.

It all comes down to one thing: does the metric help you make decisions? When you see the metric, do you know what you need to do? If you don’t, you’re probably looking at a vanity metric. Vanity metrics are all those data points that make us feel good if they go up but don’t help us make decisions.Neil Patel – Metrics, Metrics On The Wall, Who’s The Vainest Of Them All?

Measuring UX

While other areas have it easier to measure their progress in numbers, UX is still tougher to tackle. Because UX is so difficult to measure, we still struggle to define our value to organizations who could benefit from our work.

Business metrics are often tied directly to dollars and cents, so it’s a metric almost everyone understands. Improved sales and reduced costs equal profit and everyone loves money. If it’s a business that relies on technology they already have tons of stuff they measure. Reduced server loads, faster software, and less latency are all things that while not everyone in the public truly understand them, most understand them as a good thing.

But UX, and design in general, is harder to tackle. One way to measure its effect is through the HEART framework, designed by Kerry Rodden, Hilary Hutchinson, and Xin Fu from Google’s research team.

To make this work in practice it’s important to use the right metrics. Basic traffic metrics (like overall page views or number of unique users) are easy to track and give a good baseline on how your site is doing, but they are often not very useful for evaluating the impact of UX changes. This is because they are very general, and usually don’t relate directly to either the quality of the user experience or the goals of your project — it’s hard to make them actionable.How to choose the right UX metrics for your product

The framework is a kind of UX metrics scorecard that’s broken down into 5 factors:

Happiness: How do users feel about your product? Happiness is typically measured by user satisfaction surveys, app ratings and reviews, and net promoter score.
Engagement: How often are people coming back to use the product? Level of user involvement, typically measured via behavioral proxies such as frequency, intensity, or depth of interaction over some time period. Examples might include the number of visits per user per week or the number of photos uploaded per user per day.
Adoption: How many people successfully complete the on-boarding process and become regular users? Adoption is measured by number of new users over a period of time or percentage of customers using a new feature.
Retention: The rate at which existing users are returning. You can measure how many of the active users from a given time period are still present in some later time period. As a product owner, you may be more interested in failure to retain, commonly known as “churn.”
Task success: Can your users achieve their goal or task quickly and easily? Task success is measured by factors like efficiency (how long it takes users to complete the task), effectiveness (percent of tasks completed), and error rate.

Using the HEART framework gives you a better understanding of the user and their relationship with the product. The nice perk of this is that it can be applied to a single feature in your app or to your whole product. It gives you the option to measure just what you need to at any moment.

Informed by data, driven by empathy

The other day, I published this quote from Data-Driven Design is Killing Our Instincts that’s stuck with me.

We’re told all design decisions must be validated by user feedback or business success metrics. Analytics are measuring the design effectiveness of every tweak and change we make. If it can’t be proven to work in a prototype, A/B test, or MVP, it’s not worth trying at all.

In this cutthroat world of data-driven design, we’re starting to lose sight of something we once cherished: the designer’s instinct. “Trusting your gut” now means “lazy, entitled designer.” When we can ask users what they want directly, there’s no room for instinct and guesswork.

Or is there?

As we have access to all of this data, and the general consensus is that data should lead the way, I’m confused by what the future role of a designer is. What I truly like about the HEART framework is that while the output of it is informed data, it’s all done through the eye and mind of a human.
Just like we need to be careful to not measure everything just because we can, we need to remember that in order to really understand human’s behaviors, we need to talk with them, not just look at charts. Charts are great for an overview, but human insight is the best way to understand the underlying reason for the chart’s direction.

“Data-driven” is all the rage at the moment, everyone wants a slice of the “big data” cake. Data scientists are the new rock stars, replacing the JavaScript and Front-end gurus and ninjas from a few years back. My problem with trends like these is that they cause the so called “tunnel-vision”. Thing x is a trend right now and we should do that too because,… you know… everyone’s doing it.
Some companies have already started to realise that data alone can’t answer all the questions they need answered. Booking.com put “Informed by data, driven by empathy” in their design guidelines.Measuring and Quantifying User Experience

How do you get from the HEART categories to metrics you can actually implement and track? Unfortunately, there’s no off-the-shelf “HEART dashboard” that will magically do this for you. The most useful metrics are likely to be specific to your particular product or project. Personally, I like to start bigger projects with a UX Strategy. This UX Strategy should define what the most important tasks and metrics for your product is, and consequentially, what failure looks like. How might success or failure in the goals actually manifest itself in user behavior or attitudes? For YouTube, an engagement signal might be the number of videos users watch, but an even better one could be the amount of time they spend watching those videos. A failure category for YouTube Search might be entering a query, but not clicking on any of the results.

Even fashion brands are using data to design products that will meet customers demands better:

“You never design by data, but the data provides a compass as you’re navigating a hunch.”Analytics are reshaping fashion’s old-school instincts

So while we as designers might not get the out-of-the-box dashboard with all the charts ready for us to act on, there are tools and techniques that we can use to get the information needed for us to act upon to get a confident start. Let me know if you need help getting started!

Measuring user experience with usability metrics