How to evaluate chatbot performance

How can chatbots be used? 

With more than two billion people regularly buying items online across the world, giving customers access to the service and answers they need via your website or social media is key to giving them a great shopping experience. 

Powered by artificial intelligence (AI), chatbots are software tools that can be used to answer customers’ basic questions or gather more information from them about their queries. Able to connect to businesses’ customer databases and CRM (customer relationship management) systems, there are two main types of chatbots available: live chat and messaging chatbots. 

Live chat chatbots are present on websites and mobile apps, usually through a pop-up or unique tool that customers can access easily. In contrast, messaging chatbots are available across any channel, including social media platforms like Whatsapp and Facebook. Depending on the requirements of a business, the chatbot can offer different levels of customer support.

For example, chatbots can be used to carry out basic workflows such as gathering customer information or answering frequently asked questions. Alternatively, they can help customers to continue a conversation with your business, regardless of the channel they’re on, deliver self-service information or even trigger automated workflows or actions from your service team. 

In short, if your business has a set of repeatable or straightforward service tasks that could be automated, chatbots can help you to deliver them, freeing up your customer team to deal with more complex queries or processes. 

What chatbot metrics should be measured?

As we’ve established, chatbots have a huge variety of uses depending on the needs of businesses and customer service teams. So when a business is looking to measure its chatbot’s performance, it’s important to consider how the tool is being used and what problem it was brought in to solve. 

For example, if you’ve brought in a chatbot to help you deal with basic customer queries quickly, then you’re going to measure its performance differently to one that is designed to help customers find the right products. 

So, to start evaluating your chatbot’s performance, you first need to decide on the metrics you want to measure. Some of the most common KPIs (key performance indicators) you can track include: 

• Activity volume: simply, how often is your chatbot being used by customers? Is it being used often or more frequently? 

• Bounce rate: how often is a customer starting a chatbot interacting before immediately closing the chat or moving to a new page? A high bounce rate indicates that your customers might not be finding the content in the chatbot useful or helpful. 

• Use rate by open sessions: chats might be opened fairly regularly, but how often are they actually used by customers? 

• Target audience session volume: how often are the customers you’re trying to target using your chatbot? 

• Response volume: how many questions has your chatbot answered? 

• Conversation length: how long are the conversations your chatbot is having before the query is resolved or the customer leaves the chat? Long conversation lengths may indicate that further optimisation may be required. 

• Usage distribution: is your chatbot most commonly used in the morning, evening or at the weekends? 

• Questions per conversation: similar to conversation length, this metric examines how many questions your chatbot needs to ask before it resolves a query or the customer leaves. If this rate is high, it may indicate that your chatbot needs optimising. 

• Interaction rate: this rate will show you the number of messages that are exchanged per conversation. This is a good measure of user engagement. 

• Goal completion rate: this measures how often a user completes a defined action within the chat, such as clicking a particular CTA or filling out a form. 

• Non-response rate: how often does your chatbot fail to respond to your customer’s query?

• Recurring FAQs: you might have set up your chatbot to answer customers’ most common queries. This metric will help you to identify which ones the chatbot answers most. Using this information you can create more self-service content and even create specialist chatbots who can deal with these types of queries more effectively. 

However, alongside this data, it’s important to gather qualitative measures from chatbots too, as they’re designed to be human-like tools and therefore the quality, as well as the efficiency, of their performance should be considered. Surveying customers after they’ve used a chatbot to find out if they’re satisfied with the service, if the chatbot responded to them promptly and appropriately and if their query was resolved will help you to uncover any areas of improvement.

How to track chatbot performance

Once you’ve considered and chosen the most relevant metrics, you can start to track your chatbot’s performance. 

If you have multiple chatbots, you might want to track different metrics to more closely reflect and mark their purpose and uses. To track your chatbot’s performance accurately using these metrics, it’s key that you: 

• Establish a benchmark: once you’ve set up your chatbot and it’s been running for a short amount of time, take a note of the metrics you’re looking to track. This will act as the starting point for measuring its performance. 

• Take measurements at regular intervals: whether it's every month, quarter or even every week, ensure you’re checking your analytics and noting down how these metrics regularly. This will help you to identify and solve any problems quickly as well as track your progress and performance over time. 

• Interpret and action the analytics: taking note of metrics is great, but that’s no use to your customers if you don’t take action on them. Make sure you understand the details as well as the bigger picture, point out any areas for improvement and set actions to resolve these problems. 

By following this process for each of your chatbots, you can keep a close eye on their performance and ensure they’re optimised to give your customers the best possible online service experience. 

Why measure chatbot performance?

Chatbots can bring plenty of advantages to a businesses’ customer service. However, to make sure the tool is working at its best for the benefit of your service teams and customers, it’s key that you address any problems quickly and effectively. 

By measuring chatbot performance regularly, you can identify and fix areas for improvement more easily. For example, if your pop-up chat has a high bounce rate, you can try putting your chat behind an easily-accessible button. This will make it less disruptive for customers, but it means that they can still access it when they need to. Alternatively, if your chatbot has a high non-response rate, you may need to investigate the reasons why the technology isn’t working as it should. 

Plus, tracking chatbot performance can help to continuously improve its outcomes. Qualitative feedback is particularly useful for this, as it will help you to get direct and detailed information from customers about what is working and what isn’t within your chatbots. By actioning this feedback, you can make sure you take full advantage of this tool and see great results. This isn’t just great for your customers and teams but will help you to make the case for further chatbot usage within your business.