Rocketbots Academy Part 2
Welcome to Part 2 of the Rocketbots Academy, where you’ll begin training your chatbot.
There’s a lot to get done, so let’s get started.
Today you’ll learn:
- About Chatbot Training
- An overview of the training module in DialogFlow
- The basics on confidence levels
- Your second task: To train a confident bot
Time for round 2!
1. An introduction to Chatbot Training
During Task 1, you may have noticed that by adding additional Sample Data to the user says section, your chatbot was able to increase the number of inputs classified successfully.
This is because an Agent learns both from the Sample Data you provide in the user says section and Machine Learning language models developed by Dialogflow.
Based on this data, it builds a mathematical model (algorithm) for making decisions on which intent should be triggered by a user input. This algorithm is unique to your agent.
Improving this algorithm is one of the key elements of a successful chatbot. In DialogFlow, there are several ways to improve this classification accuracy:
- Adjusting the Confidence Threshold
- Switching to ML only mode
In this section, we will explore the use of these tools and best practices for their use.
The Importance of Training
Training is one of the basic building blocks of your chatbot. Training should be done with due care and diligence. The result of poorly thought out training is many long days and nights of debugging your chatbot.
2. An overview of the training module in DialogFlow
The training module is a purpose-built set of functions used for chatbot training.
To enter the training module Click the Training Button on the left panel of DialogFlow.
Training Module Functions
The training module is as it appears above, this module holds all the historical user inputs from both the Console & the connected messaging platforms there are several functions.
Training & History
The two tabs at the top will show different versions of the training module. Here we will discuss the Training tab, the History tab is only useful in some specific situations.
Located to the right of the tabs below the upload button. Use the refresh button to see the most recent logs from the connected messenger or from the Console.
The list of items which begin with quotation marks are called sessions. Each unique user that speaks with your bot will begin a new session, once this session has reached 50 items or the user is inactive for an extended period, a new session will be created for their next input. Inputs from the Console are also included as a separate session.
- Dialog Column: The dialog column displays the first input of each session.
- Requests: The requests column shows the number of individual inputs in each session.
- Unmatched: The unmatched columns show the number of intents that could not be classified in each session.
- Time: The time column shows how long ago the sessions occurred.
Now Click on One of the Sessions and let’s take a look inside.
Training Sessions Screen
In Task 1, you trained your agent by opening the intent and adding Sample Data directly into the users says field. With small amounts of sample data this is possible, but when there is a larger amount, especially from actual users, the training module saves time.
The training session screen has several functions.
This area shows the input from the user, inputs are shown in the same order as they are entered by the user.
This line shows the intent which was triggered or will show click to assign if an intent was not triggered and the agent instead delivered the Fallback.
This button is found to the right of the User Says input. When clicked it will add the exact input to the Sample Data of the intent indicated by Intent.
Once clicked the changes made in this sessions will be executed. In other words, the sample data will be added to each respective intent.
This information above may be difficult to understand. Please proceed to see how the functions above work in practice.
Training Your Chatbot
In the sessions screen you may encounter one of the following conditions:
- An input is classified correctly.
- The intent desired by the designer is listed under the input.
- An input was classified incorrectly.
- The intent listed under the input will show an intent which is not desired by the designer
- An input has not been classified.
- You will know this because the intent will not be listed, the log will instead show, “Click to assign”
When the input is classified incorrectly or unclassified
- Click on the blue text next to intent.
- This will display a list of intents in your chatbot.
- Find the intent you would like to be triggered when this phrase is used and click it.
- You may use the filter at the top of the intent list to help you find the intent.
- Once this is complete, you will see that the arrow next to the inputs where you selected the intent have become green. This indicates that once approved, all the assignments made in this session will be executed. The Sample Data will be added to the respective intent.
- Now click approve.
Once you click approve, this session will close and you will see the gear spinning next to the DialogFlow logo. The spinning indicates that DialogFlow is updating the algorithm to include the new sample data.
3. The basics on confidence levels
What is an ML Confidence Threshold
When an Input is sent by a user and passes through DialogFlow, it will be assigned an intent classification along with a confidence score. This score is between 0 and 1. In practice, the score will have to decimal places, such as .31, .32, .33, etc. In DialogFlow we can set a score a threshold for delivering the response. When an input is sent, if the score for the matching intent does not pass the confidence interval, the agent will deliver the fallback.
To find the confidence Threshold setting in DialogFlow click the gear.
Then, click the tab ML Settings.
Setting the Confidence Threshold
Before setting the confidence threshold, Set the Match Mode to ML only. Remember you will need at least 30 expressions per intent of Sample Data to ensure the Machine Learning is working effectively.
Once you are in ML only mode, you can go on to raising the confidence threshold. The purpose of doing so is to reduce false positives:
- In the case where you have similar intents, the match will have to have a higher score to perform well.
- In the case where you would prefer the chatbot to go to fallback, rather than giving the best matching answer.
By default, DialogFlow will set your confidence interval at 0.30. When raising the confidence threshold it is best practice to do so in .01 – .02 increments as it will affect the performance of the entire chatbot.
You’ve learned what you need to know about training, now let’s move on to the next task.
4. Your second Task: To train a confident bot
Training & Confidence
Up until now, your chatbot has had a very simple set of disparate intents. But have you noticed that whether you use the phrases:
- Are you smart?
- You are smart.
- Can you get smarter?
- You have gotten smarter!
Your chatbot just uses the response from Intent: Are you Smart. Although these phrases are similar, they do not have the same Intention. This is not exactly a fantastic user experience.
During this task you will:
- Gain experience in using the training module
- Learn the importance of having high-quality Sample Data
- Learn how raising the confidence interval affects the performance of your chatbot
√ Add intents for the following intentions to your chatbot:
- Who are you?
- Who gave you your name?
- How old is your creator?
- Good morning.
- You are smart.
- You have gotten smarter.
- Are you a chatbot?
√ Slowly raise your confidence interval so that the following intentions go to the default fallback intent:
- Who are you working for?
- How are you going to help me?
- At what age did you become a chatbot?
- Can you get smarter?
√ Then ensure all your previously existing intents are working correctly.
Best Practice Notes
When creating new intents:
- Add 20 to 30 lines of high-quality sample data to the user says section of each intent.
- Test intents with similar intentions using the console, 10 – 20 test messages per intent.
- Use the training module to assign inputs to the correct intents, when they have been classified incorrectly.
- If you see too many intents classified incorrectly, raise the confidence interval and see what happens.
You just trained your chatbot.
Halfway there, we’ve got even more to cover in Part 3.
-The Rocketbots Academy Team