.png)
Today in our ongoing Learning Series, Senior Associate Matt Little dives into the technology and strategy behind the use of chatbots driven by artificial intelligence. Matt is a member of our skilled Data & AI and Digital Solutions Team teams, who has previously shared his thoughts on RedCloud and the industry in our 10 Minutes With profiles.
AI Chatbots have quickly become an integral part of our digital lives, and more and more businesses are adopting them as an enhancement to their user’s experience every day. We already see Chatbots serving many purposes. They can be a virtual assistant/co-pilot, sales/lead generator, customer support agent, self-service portal for IT troubleshooting or HR questions, and much more. However, not all Chatbots are created equal. As a new adopter of Chatbots, you may be asking yourself “How do I ensure that my Chatbot is becoming more efficient and effective over time?”.
In this blog post, we will explore important concepts you need to know to analyze the performance of a Microsoft Power Virtual Agents Chatbot, so that you can feel confident you’re getting a great return on your investment in this new and exciting space.
Why Microsoft Power Virtual Agents (PVA)?
We chose to focus specifically on PVA for this post because it’s a quick, easy, and cost-effective application through which AI Chatbots can be built. The pace and sophistication of these Chatbots are improving all the time, as Microsoft continues to integrate functionality from Chat GPT’s large language model, many of which are already available in the preview version of PVA today. If your organization already uses Microsoft 365, you can sign up right now for a Free 30 Day Trial of Power Virtual Agents and get started on your Chatbot journey.
Intro to Chatbot Terms & Metrics
We’ll begin by defining the most critical words you’ll see when evaluating your Chatbot’s performance:
Session - when a user interacts with your Bot or the Bot sends a proactive message to the user.
Topic – a designation that determines how a Bot will respond to a user’s question, assigned based on the Bot’s understanding of the user’s intent or their use of specific trigger phrases.
Engaged Sessions – a Session where an in-scope Topic for the Bot has been triggered (i.e. the Topic is something the Bot can answer), or an escalation to a live agent has occurred. Once begun, this can result in 1 of 3 potential outcomes:
Resolved – User confirms the interaction was a success and/or that the conversation can end.
Escalated – The conversation is transferred to a live agent or support ticket.
Abandoned – The conversation times out after 30 minutes of inactivity, without being Resolved or Escalated.
Total Sessions – a count of the number of Sessions within a period of time.
Engagement Rate – Engaged Sessions divided by Total Sessions, expressed a percentage.
Resolution Rate – Resolved Sessions divided by Engaged Sessions, expressed a percentage.
Escalation Rate - Escalated Sessions divided by Engaged Sessions, expressed a percentage.
Abandoned Rate – Abandoned Sessions divided by Engaged Sessions, expressed a percentage.
Customer Satisfaction (CSAT) – Score(s) provided by the user via a survey at the conclusion of a Session.
Impact Score – how much a particular Topic contributes to a Rate, based on volume.
Optimizing Bot Performance Over Time
Now that we’ve got a baseline understanding of how a Chatbot’s performance is evaluated, let’s move on to how you’ll recognize trends in these metrics and start to drive them in the right direction.
Accuracy of the Bot’s Responses
Indicated by increasing Resolution Rate and/or decreasing Escalation Rate, while controlling for Abandonment Rate over time. Taking a more granular look at the Topics with the highest Impact Scores for these measures will give you a starting point for where the biggest improvements can be found.
Accuracy will be improve over time if you are:
regularly training your Bot on new/updated information that’s relevant to its purpose iterating on the paths & nodes the bot will follow once a Topic has been identified
optimizing the trigger phrases used by the Bot to determine the Topic of the conversation
use Microsoft 365’s Copilot to allow the AI to write new Topics, nodes, and responses for itself based on your inputs of what is missing
Speed of the Bot’s Understanding and Response
User expectations for Speed will be most clearly indicated in CSAT responses, and can be compared against the number of prompts needed to understand the user’s question, and the time spent by the Bot once it does generate a response. In cases where the Bot has do things like make calls out to trigger Power Automate Flows, you can also compare the duration of these Flow Runs against User expectations for Speed.
Speed will improve over time if you are:
optimizing the trigger phrases used by the Bot to determine the Topic of the conversation
simplify the Conversation Path and/or the variables & calculations the Bot needs to perform to generate its response
pre-filtering/loading less data in Power Automate flows, returning Flow Outputs to the Bot in earlier steps, or running Flow actions in parallel branches rather than sequentially
User Satisfaction & User Engagement
CSAT responses typically serve as a proxy for overall User Satisfaction. This data can be gathered in the form of an end of conversation survey, a prompt for the user to leave a rating, and/or by performing a sentiment analysis of the user’s tone and word choice while conversing with the Bot.
User Engagement can be evaluated by sampling the behaviors of a representative group of users of the Bot. How are their Session counts changing over time? How much time are they spending with the Bot per Session before Escalation or Resolution? Are they returning to use the Bot as it gets trained on new information? Are the questions they ask evolving to take better advantage of the Bot’s full range of capabilities?
Improving User Satisfaction and User Engagement over time will be highly contextual for each Bot. Most importantly though, the data gathered for these measures should be paired up with anecdotal evidence in order to gain a deeper understanding of specifically how the Bot can offer a better user experience in the future.
Regularly monitoring and analyzing these metrics will give you a solid understanding of your Chatbot’s current level of performance, how to identify areas for optimization, and how to continuously refine the conversational abilities of your Bot. Now, go enjoy unlocking the potential that AI Chatbot’s have to offer!