Measuring Chatbot Effectiveness: 16 KPIs to Track

What do users think about your chatbot? Are they satisfied with the responses it provides? Are they taking full advantage of this tool? Does the conversational agent have a positive impact on recurring contacts? If you don’t know the answers to these questions, it’s probably because you don’t have the necessary key performance indicators in place. These measurements are indispensable for tracking the results of your chatbot, identifying any stumbling blocks and continuously improving its performance. But which metrics should you choose? Here are 16 KPIs to track and analyze in order to determine the effectiveness of your chatbot!

Quantitative KPIs: Is your chatbot being used sufficiently and does it respond to users’ needs?

Quantitative key performance indicators allow you to evaluate the effectiveness of your chatbot and the way it’s used by its target audience.

1. Chatbot Activity Volume

Measuring a chatbot’s Activity Volume means evaluating the number of interactions, from the time a user asks a simple question until a constructive dialogue takes place. This indicator helps answer two key questions.

  • Is your chatbot being used frequently?
  • Is the number of users increasing?
Voluntary or prompted use?

Usage can be broken down into two types:

  • Voluntary use, when users initiate the interaction with your chatbot of their own accord
  • Prompted use, when users initiate the interaction with your chatbot after receiving a notification

The Voluntary Usage Rate is an excellent indicator of your conversational agent’s popularity. It also allows you to verify that your chatbot is well-positioned in the course of the customer experience.

2. Bounce Rate

The Bounce Rate corresponds to the volume of user sessions that fail to result in the intended “specialized” use of your chatbot. An elevated rate indicates that your bot isn’t being consulted on subjects that are relevant to its area of competence. This should prompt you to update its content, rethink its placement in the customer experience, or both. It’s an indicator that should be observed closely.

3. Retention Rate

The Retention Rate refers to the proportion of users who have consulted your chatbot on repeated occasions over a given period. This indicator can be compared with the typical frequency of client contacts in your particular line of business. It will provide a good indication of your chatbot’s relevance and its level of acceptance among your clients.

4. Use Rate by Open Sessions

This is the number of sessions that are simultaneously active with your chatbot. To get a meaningful measurement, this rate must be weighted with the average number of open sessions during a given period.

5. Target Audience Session Volume

This indicator is essential for verifying that you are achieving your goals. If you are targeting a specific population, you can measure the penetration rate for this audience in order to verify that the intended people are making sufficient use of your chatbot. Otherwise, it’s imperative to rethink your change management or customer experience strategies in order to get your users on board!

6. Chatbot Response Volume

This is a concrete indicator that will tell you the number of questions your chatbot has answered.

7. Chatbot Conversation Length

This metric allows you to evaluate the average length of the interactions between your chatbot and its users. The figure will vary significantly from case to case: a chatbot that resolves computer issues or that provides online estimates will require a much longer dialogue than a chatbot that gives the current time in all the cities of the world! If your goal is increased efficiency, this KPI will help you quantify the amount of time saved by your clients, as well as your Help Desk.

8. Usage Distribution by Hour

At what times of day do users most frequently consult your chatbot? This indicator is particularly helpful, as it often serves to demonstrate how this new 24/7 channel enables you to cover 20, 30 or even 50 percent of the hours during which your user support services were previously unavailable.

9. Questions per Conversation

The more questions users have to ask, the more time it will take for them to obtain adequate responses. This indicator will help you determine how many questions your chatbot needs to be asked before it can provide the necessary information to its users. Please note that the interpretation of this metric depends heavily on your specific objectives.

10. Interaction Rate

If you want to measure user engagement during conversations with your chatbot, you’ll definitely want to observe this indicator. It will allow you to measure the average number of messages exchanged per conversation.

11. Goal Completion Rate

This metric enables you to measure the success rate of a given action performed through your chatbot, for example, clicking on a CTA button or link, filling out a form, proceeding to make a purchase, etc. However, it can only be applied to clearly identified actions for which customized indicators have been created.

12. Non-Response Rate

This metric measures the number of times your chatbot fails to respond to a question. Such failure may be the result of a lack of content or of your bot’s difficulty in comprehending user inquiries.

13. Most Frequently Asked Questions

What inquiries are most often addressed to your chatbot? Thanks to this statistic, you can adapt your chatbot to specialize in the subjects that come up most commonly and thereby improve its performance. Analyzing recurring questions will help guide your corrective work, allowing you to focus on the topics that are of greatest interest to your users and the mechanisms that will enable you to improve the quality of your bot’s responses, as well as its overall comprehension levels.
To make sense of these quantitative KPIs, you must compare them with other data, particularly the number of calls and the results produced by other channels (e.g. chatbot conversation volume vs. telephone call volume, relative satisfaction rates, etc.). This data will make it possible for you to evaluate the positioning of your chatbot and determine if it’s in the right place with the right knowledge.

Qualitative KPIs: Are your chatbot users satisfied?

Besides quantity, there’s also the matter of quality. The KPIs below will help you measure your chatbot’s “human performance,” including its levels of comprehension, the help it provides to its users and its user satisfaction rates.

14. Comprehension Level

Your chatbot will indicate its overall comprehension of user inquiries. This level is constantly evolving, as it depends on:

  • The chatbot’s comprehension of the questions it’s asked
  • The chatbot’s knowledge base

If your chatbot doesn’t understand an inquiry, it’s either because it has been asked a question that has no meaning for it or because it doesn’t have knowledge in the related field.
For example, a chatbot specializing in computer support won’t understand a legal question!

15. Self-Service Rate

This rate corresponds to the number of users who were able to obtain the help they needed through the responses given by your chatbot, without subsequently having to call Customer Service. It is calculated based on the percentage of sessions that were successfully completed through an interaction with your bot without being redirected to a live operator. In the process, it enables you to evaluate the level of client satisfaction. This is the equivalent of a call center’s First Call Resolution (FCR) Rate, the percentage of problems that are resolved through a single phone call.
This indicator is very important for analyzing the ROI of your chatbot project.

16. User Feedback

Finally, it’s indispensable to know what users think about your chatbot. Did it provide sufficient help? Are its users satisfied? There are two different ways you can find out:

  • You can simply ask users to reply “yes” or “no” to the question “Were you satisfied?”
  • You can offer users the opportunity to fill out a more in-depth questionnaire in order to obtain specific information (e.g. “Were the responses clear?” “Did you understand everything?” or “Do you have any suggestions for improving our chatbot?”).

This feedback will allow you to calculate two indicators:

  • The Satisfaction Rate (the average score received by your chatbot in evaluations by its users)
  • The Evaluation Rate (the percentage of sessions in which the user evaluated your bot’s responses at least once)

These qualitative KPIs should be measured regularly and analyzed over the long term. This will allow you to see how satisfaction rates evolve, whether the recurring questions are always the same, and more.


You might also be interested in our checklist for a successful chatbot project!


Knowing these 16 key indicators is essential for evaluating your chatbot’s performance. However, the best indicators to track aren’t always the same for every company or for every chatbot: it’s up to you to choose the most relevant ones based on your line of business, your goals and the needs of your users. Would you like some guidance to help you figure out which KPIs will be most helpful in measuring the effectiveness of your chatbot?

 

Credit photo: Pexels / Digital Buggu