Use competitive usability tests to improve your own product.
You don’t have to copy your competitors, or ignore them instead. By using two user testing methods explained in this article, you can see what your competitors are doing right and wrong instantly to start learning from them.
When I started my career in UX, people were always telling me how they liked this function at service X that they had used or seen. Then they asked me to copy it. That made me feel bad, because it was basically like stealing work from other people.
Other people were telling me that I should never even consider copying functions from existing services, because the fact that it worked for them did not mean that it would work for me. But that also felt wrong, because if someone else had already spent time inventing the wheel, why would I try to do it all over again?
For a long time I didn’t know what to do with this. I copied some functions other people had already made, trying to see if that would work for me. I also tried to come up with new stuff that no one had ever done before, trying and see if that would work.
That was until I started to do competitive usability tests. And it worked really well. I had already done my fair share of regular usability tests. So I knew how to come up with a hypothesis, make a user scenario, set up a user test and collect results to analyze and report. Doing this with competitors was easy.
One thing that is really important to know when it comes to testing with your competitors is that you should be collecting comparitive data. You’re always collecting data when you are doing a usability test, but this time you need to be able to compare the data from a competitor with the data of your own service.
When I started testing competitors, I had the luck that my colleagues pointed me in the right direction. We had mapped main functions in the website of our client and two important competitors as a benchmark, and we were wondering what these functions were worth. But we also wanted to have some statistics that we could show to our client in order to convince them.
So in the usability test, I asked users if they could try and find these functions in the websites of our client and their competitors. For every user that could find a function on a website I would add a point to that website. After five tests, I could calculate the exact scores for every website, and compare them to each other.
This technique is called a traffic light usability test. There are a number of ways to use these kind of usability tests. You can also look at how long it takes a user to perform an action for example, to measure and compare how hard it is for a user.
So far I have tried two kinds of competitive usability tests, and they both worked really well for me. I will explain for each of them how they work and when you would use them to do research on your product or service.
The first test is used for when you need to find out how your product is performing compared to its competitors. You want to know how far ahead or behind you are compared to other products, or you want to know where your focus should be. I had the following hypothesis in my case.
My marketing website about my kitchen store has a lot of functionality compared to the websites of my competitors, but a lot of users still choose their services over mine. I think their functions are easier to find compared to the functions on my website.
Preparation
To test this hypothesis, we began with a list of all the functionalities on the website of our client and those of two competitors. We didn’t think it would matter if there were functions on our list that were not on the website of our client. If they were on the website of competitors, we still wanted to know if users could find them and why.
Next, we chose two competitors that were most relevant to our client. There were actually a lot more competitors, but we couldn’t make users look at all of them, that would simply have taken too long.
We also set up screen-recording and eye-tracking.
Testing
We invited five users for our tests. We explained to them that they were going to get a list with functions that they needed to find on three different websites.
Because we didn’t care about how long it took them to find a function, they did not have to follow any particular order. They could just browse the website like they would normally do, and tick off any functionality on the list they came across. While they were browsing, we asked if they could tell us what they were doing and why.
After looking at each website, we made sure our users checked the list one more time to see if they didn’t forget any functions. We recorded their answers digitally, so we could easily collect the results in a spreadsheet and calculate the scores.
Analysis
We looked back at our eye-tracking and recording data to see what users did. Often, users told us that they found a certain functionality while they found something else. Or they told us that they did not see a function, but their eyes actually went over it. In that case it probably just didn’t seem interesting to them at the time.
Because we had recordings we could rewatch the tests later to see if users checked off any functionalities incorrectly. Knowing about this was vital information because it told us something about why they could not find certain functions.
Results
In the end, we had three scores for every website that we wanted to compare. We knew which website performed best, which functions were easy to find, and which were not. But most importantly, we knew why functions were hard to find because we asked the right questions and analyzed our recordings.
We were able to confirm that despite our clients website having more functionality, the functionality on the websites of competitors was easier to find. But more importantly, we were able to tell why those functions were hard to find.
We discovered that functionality was scattered across the website of our client, while competitors offered their funcionality in organised flows, resembling the way people find inspiration for their new kitchen in real life.
We even discovered that men and women prefer different flows when it comes to buying a new kitchen. While men want to start planning and visualising, women are more likely to look for inspiration and respond to personal style.
The second test is for when your competitors are offering a functionality that your website does not have, and you want to know if your users could benefit from this functionality on your website. In my example, where we wanted to check if product reviews would be relevant to our users, I had the following hypothesis.
My pet food webshop does not enable users to leave reviews and read reviews of other people, but other pet food webshops do offer this functionality to their users. I think my users will also be interested in this functionality.
Preparation
Normally when you would like to test a new function, you would need to build a design or a prototype that users can see and use before they can tell you anything insightful. But when your competitors already have similar functionality that you can access, you can just use that to show to your users in a usability test.
In my case our client gave us a list with competitors that offered similar webshops. We chose two competitors that had the product reviews we wanted to test, and where those product reviews worked well in our experience.
I also set up screen recording and eye-tracking.
Testing
I asked five users to take a look at product pages on each website, and decide whether they would be interested in buying the products offered on those pages. I did not explicitly tell them to look at the reviews, because I wanted to know if they would focus on the reviews by themselves.
For comparison, I also added a product page at the webshop of our client for users to look at, but without the product reviews. That way I could see how users would react if there were no product reviews, in contrast to the product pages with reviews.
During each test, I would ask users what they were looking at, why they were looking at it, and how it influenced their choice in deciding to buy a product. After each test, I would only ask users to fill out one quesion: Would they be interested in buying the product, and why.
Analysis
Often, their choice didn’t stroke with what their behavior showed according to the scores, eye tracking and recordings. But when they were asked to choose, they opened up and started telling more because they thought they were given control of the research.
Tricking your users like this can bring you insights that you wouldn’t have been able to expect. This way you can find out what else you can look into for new research. Or you can come up with new unexpected concepts you can now also test.
When I was done testing, I could see on which pages my users most likely wanted to buy a product. By looking back at the questions I had asked from my notes, the sceen recordings and eye-tracking data, I could see that users almost always looked at the reviews if they were present.
Results
I found out that most of the time users didn’t base their choice on the product reviews primarily. But they still looked at the reviews for a number of users. They wanted to see whether their own choice would be confirmed by other users, or whether there would be positive reviews from users who had a pet animal similar to theirs.
But sometimes my users would tell me they didn’t think reviews were relevant to look at, because they had already been advised by their vetirinarian to buy a specific product for a disease of their pet. In the test however, those users still looked at the reviews because they found the reviews entertaining to read.
The users also told me what they liked about the presentation of the reviews on the product pages. Some aspects made things more clear to them, while others were disturbing to them.
So after the test, I could advice our client not only about the relevance of product reviews to them, but also about how to implement this feature. Even though I hadn’t designed or prototyped anything myself.
When I started my career in UX, I didn’t know about competitive usability tests, or even about usability testing in general. Because of that, I also didn’t want to feel bad because of looking at competive products and copying their functionality.
But now that I do know about these techniques, I know how to learn from competitive websites and I have found a way to tell clients exactly what they should and should not copy.
I hope that you, after reading this article, also know what to do with competitors and you can really start learning from them instead of just copying functionality that your clients have seen and like.
If you have any questions about these kind of usability tests, feel free to ask me and I’ll try to answer as best as I can. If you still think functionality of competitors should never be copied and have good reasons for this, I’m also welcoming you to let me know by commenting.