Customer experience: how the most popular metrics can lead you astray

In a previous column, we explored Google's customer experience score as one of the best ways to measure the quality of your website. Today, I want to talk about the other side of the equation: the metrics to avoid or treat with caution.

When it comes to measuring customer experience and happiness, businesses have traditionally relied on methods like self-reports and surveys, which can lead to bias and misinterpretation. As larger sections of the customer journey move into digital spaces, new tools have sprung up to capture hard data about real actions and interactions, quantifying how fast, successful, and enjoyable these experiences really are. By combining this quantitative data with traditional data about the customer's thoughts and feelings, we start to see the full picture for the first time. 

If you're not sure how to achieve this level of insight, you're not alone: only 38% of executive leaders say their organisation does well at measuring customer experience, and 27% say they struggle (with the rest falling somewhere in the middle). Measuring the wrong indicators or measuring too many variables can easily get misleading and overwhelming, and even contradictory. Sure, data is powerful, but only if it's relevant, accurate, and actionable. 

Now, let's preface this by saying that if you are using either of the metrics we talk about today, don't panic — they're not innately bad or harmful and are definitely still helpful at the right times. It's just that these metrics, while popular, are often overused or misapplied. Let's see why. 

Averages

An average is the best way to understand where the majority stands, right? Well, sometimes. Averaging works well when your data set is clustered within a relatively small range. But when it comes to measuring digital experience, an average can actually work to conceal the reality of large portions of your customer base. Because experience signals like page load time tend to fall across a broad range, it's easy for a few outliers to skew your average and lead you astray. 

It's also easy for the outer values (which need the most attention) to be hidden. At Raygun, we use the median as a better indication of where the 'typical' customer actually sits because your median, by definition, will show where most of your customers are. 

For load time, we also consider the P99 or the quality of experience for the 99th percentile of users. (The 100th percentile is better left out of the equation, as this is where total timeouts and bots tend to throw the numbers out wildly). Once your team has agreed on a minimum standard for acceptable customer experience, your P99 shouldn't exceed this. Compared to your ideal time, this is going to be slow, but you still want to keep this closer to five seconds than 25.

Tracking your P99 ensures that even the laggiest, buggiest, most frustrating experiences you deliver are still within reasonable parameters. Keeping your P99 in range acknowledges that we can't guarantee perfection every time, but we can still keep ourselves honest.

Net Promoter Score

NPS remains deeply ingrained in many businesses, processes, and industries. It's the most popular customer experience metric, and it's still valuable. But NPS has become increasingly controversial, and Gartner has predicted that 75% of organisations will no longer use it to measure customer service and support by 2025. 

I can't say whether that will come true — there are good reasons why NPS became an industry standard, and a strong focus on customer feedback has raised standards and reminded businesses why they exist (hint: it's the customer). However, many of us over-rely on the famous score and, in doing so, overlook the risks and misrepresentations NPS can introduce. NPS is flawed in a number of ways; it amplifies the most extreme user experiences, so it isn't representative of the majority. It can be challenging to accurately attribute or improve NPS scores because better scores could be a matter of working on your product, service, price, or even just changing the market you're selling to. 

You could deliver a slow and buggy website experience followed by an excellent support interaction and still score a nine based on the last contact the customer had. Our own team at Raygun learned the hard way how timing can drastically impact NPS scores. When we started using NPS, we put the survey inside our app without considering that users often come to Raygun when something is wrong and sometimes even in a crisis. Turns out, we might as well be trying to survey folks on their way into A&E; nobody was keen to stop and provide feedback in the midst of solving an urgent problem.

NPS also gathers qualitative responses, which can be one of its most productive uses, whether it's relaying service feedback and giving upsell opportunities to your customer-facing teams, sharing feature requests with your product team, or passing promoters to your marketing team as potential testimonials. In short, NPS remains relevant as a large-scale, subjective indication of how your customers are feeling, giving them the chance to tell you what's on their minds. But it doesn't stand alone as an objective measure of experience, particularly for your technical teams. 

So, what else should I use?

Nobody likes to hear problems without any suggested solutions. Raygun customers often ask us about useful and clear ways of measuring experience, and it's a question we've given a lot of time and attention to. To help businesses find answers, we've put together a guide to digital experience monitoring, sharing ten metrics that can help you see through your customer's eyes. If you're interested, grab a free copy here.