How to build a Rule-Based Chatbot ?

Tutorial

Rule-Based Chatbot from scratch : Communication using a python API and regex.

Dates
  • Creation: 04/12/2020
  • Update: 04/13/2020
axel_thevenotapi-pythonlearn-regular-expression

Build a conversationnal chatbot with a rule-based approach


All companies try to create an authentic customer experience. That's why we see chatbots on many websites and applications with whom we can converse. Usually, they are used to provide quick answers to users by responding to their questions written in a "human" language (a language that a machine does not understand).


There are several types of conversational chatbots but they can be classified into two categories corresponding to the approach used to handle a conversation.

  • Rule-Based Approach: The chatbot responds to rules that are clearly defined, programmed.
  • Self-learning approach: Literally, an approach that is based on self-learning techniques. These chatbots respond to rules based on Machine Learning or even Deep Learning, which makes enabling more complex questions to be answered than with a Rule-Based approach.



In this tutorial we will create a chatbot with a Rule-Based approach. We will do it from scratch since it is the logical continuation of two other tutorials. We will use our chatbot using an API. I invite you to have a look at the tutorial to design an API on python with Flask. And we will build our rules from regular expressions. If you are not familiar with this I also strongly encourage you to follow the tutorial on how to use regex.


So for the purpose of our tutorial we will deal with a simple and visual case. Our chatbot, with which we will be able to interact from a web interface, will allow us to change the size, color, speed of a ball when we request it. Before being able to communicate with our chatbot we must create a web page that will be the interface between the user and the chatbot, which will be accessible through an API.


Building a web page - User Interface


What we expect from our web page is to have a chat window to communicate with the chatbot. We also need a drawing zone (canvas) to have a display of the ball that we will try to modify. Finally we also add a table to display in real time the state of the ball in the canvas.


File index.html
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Chatbot</title>
  <link rel="stylesheet" href="style.css">
  <script src="script.js"></script>
</head>
<body onLoad="init();">
  <h1>Custom Chatbot</h1>
  <canvas id="ballCanvas", width="1000px" height="500px"></canvas>
  <table id="stateOfBall">
    <tr>
      <th>Position (px)</th>
      <th>Speed (px/s)</th>
      <th>Radius (px)</th>
      <th>color (hexa)</th>
    </tr>
    <tr>
      <td>(, )</td><td>(, )</td><td></td><td>#</td>
    </tr>
  </table>
  <div id="chatContainer">
    <h2 onClick="display_chat();"n>Chat Window</h2>
    <div id="messageContainer"></div>
    <div id="inputContainer">
      <input id="inputMessage" type="text" placeholder="Write here...">
      <button id="sendButton" onclick="sendMessage()" width="50%">Send</button>
    </div>
  </div>
</body>
</html>


To see the web page written in HTML in the index.html file, just open it with your browser.


So we have all the components in place. But when we were looking about user experience, it was obvious that design and ergonomics were involved. So we're also going to change the appearance of this page to make it more pleasant to view/use.



I let you copy and paste the style.css file which gives a cleaner rendering to our web page. It also allows us to have the chat window at the bottom right and the other part centered on the page.


File style.css
body {
 background-color: #fff;
}

canvas {
  border: 10px solid #2f9599;
  background-color: #999;
  border-radius: 50px;
  height: 80%;
  width: 80%;
  margin:auto;
  display: block;
}

h1 {
  text-align: center;
}

h2 {
  margin: 0px;
  padding: 5px;
  font-size: 20px;
  color: #fff;
  background-color: #2f9599;
}

table {
  height: 80%;
  width: 80%;
  margin:auto;
  text-align: center;
}

.chatbotMessage{
  background-color: #2f9599;
  color: #fff;
  margin: 1% 1% 1% 5%;
  padding: 0px 5px 0px 5px;
  max-width: 80%;
  border-radius: 10px;
  border: 1px solid;
  word-wrap: break-word;
}

.userMessage{
  background-color: #fff;
  margin: 1% 1% 1% 15%;
  padding: 0px 5px 0px 5px;
  max-width: 80%;
  border-radius: 10px;
  border: 1px solid #999;
  text-align: right;
  word-wrap: break-word;
}

#chatContainer {
  position: absolute;
  bottom: 2%;
  right: 2%;
  height: 400px;
  width: 400px;
  border: 1px solid #999999;
  background-color: #ffffff;
  overflow: hidden;
}

#messageContainer {
  height: 315px;
  overflow-y: scroll;
}

#sendButton{
  width: 15%;
}

#inputMessage{
  width: 75%;
  margin-left: 2%;
}

#inputContainer {
  margin-top: 10px;
  height: 30px;
}




Now that our web page is nice and clean, we need to make it dynamic. The script.js file has the following roles:

  • Dynamically display the chat
  • Send messages to the chatbot via the API we'll write right afterwards
  • Compute and display the position of the ball
  • Change the properties of the ball when requested by the chatbot
  • Open/close the chat by clicking on the top of the chat window


I invite you to copy and paste the code below into the script.js file. What you need to understand in this code for interactions with the chatbot is that it interacts via an API and the response is returned to it as JSON object. If in this response it is asked to change properties of the ball, then those properties are changed. In all other cases they remain unchanged.


File script.js
var chatContainer, messageContainer, inputMessage, stateOfBall; // dynamic divs
var chatIsOpen;
var x, y; // Current position of the ball
var dx, dy, r, color// Changeable variables
var Http, url; // To send requests with the chatbot

function init()
{
  // used to request the chatbot host on 127.0.0.1:4000
  Http = new XMLHttpRequest();
  url  = 'http://127.0.0.1:4000/chatbot-request/';
  Http.onreadystatechange = function() {
    if (Http.readyState == 4 && Http.status == 200) {
      // If return from the chatbot : extract the json response
      var jsonResponse = JSON.parse(Http.responseText)
      getMessage(jsonResponse);
    } else if (Http.readyState == 4) {
      var message = "I am not connected to the chatbot API."
      message += "\n Please check the API is running on http://127.0.0.1:4000/"
      stackChatMessage(message, false);
    }
  };
  // Get the dynamic elements of the HTML
  context = ballCanvas.getContext('2d');
  chatContainer = document.getElementById("chatContainer");
  messageContainer = document.getElementById("messageContainer");
  inputMessage = document.getElementById("inputMessage");
  stateOfBall = document.getElementById("stateOfBall");
  // Execute a function when the user releases a key on the keyboard
  inputMessage.addEventListener("keyup", function(event) {
    // Number 13 is the "Enter" key on the keyboard
    if (event.keyCode === 13) {
      // Cancel the default action, if needed
      event.preventDefault();
      // Trigger the button element with a click
      document.getElementById("sendButton").click();
    }
  });

  // Set default start values
  chatIsOpen = true;
  display_chat(); // close the chat first
  r = 20;
  x = r;
  y = r;
  dx = 150 / 100;
  dy = 100 / 100;
  color = "#8800ff";
  // run the draw 100 times by second
  setInterval(draw, 1000 / 100);
}

function display_chat(){
  // Open or close the chat window
  if (chatIsOpen){
    chatContainer.style.height = "30px";
    chatIsOpen = false;
  } else {
    chatContainer.style.height = "400px";
    chatIsOpen = true;
  }
}

function stackChatMessage(message, fromUser=true){
  // stack a new message the in chat window
  var sender = fromUser ? 'userMessage' : 'chatbotMessage';
  message = message.replace(/\</g, '\u2039');
  message = message.replace(/\>/g, '\u203A');
  message = message.replace(/\n/g, '<br/>');
  newMessage = `<div class=\"${sender}\"><p>${message}</p></div>`;
  messageContainer.innerHTML = messageContainer.innerHTML + newMessage;
  messageContainer.scrollTop = messageContainer.scrollHeight;
  inputMessage.value = "";
}

function sendMessage() {
  // send message to the chatbot
  if (inputMessage.value){
    Http.open("GET", url + encodeURIComponent(inputMessage.value));
    Http.send();
    stackChatMessage(inputMessage.value);
  }
}

function getMessage(jsonResponse){
  console.log(jsonResponse.answer_to_intent);  // catched intent
  console.log(jsonResponse.t0); // Timestamp of resquest receipt
  console.log(jsonResponse.t1); // Timestamp of resquest return
  response = jsonResponse.response; // Get response

  // Display the message
  stackChatMessage(response.message, false);

  // Make the changes if asked
  dx = ("dx" in response) ? response.dx / 100 : dx;
  dy = ("dy" in response) ? response.dy / 100 : dy;
  r = ("r" in response) ? response.r : r;
  color = ("color" in response) ? response.color : color;
}


function draw()
{
  // Clear the draw
  context.clearRect(0, 0, context.canvas.width, context.canvas.height);
  context.beginPath();
  context.fillStyle = color;                   // set the color
  context.arc(x, y, r, 0, Math.PI * 2, true);  // draw the circle
  context.closePath();
  context.fill();
  // Boundary Logic
  if (x < 0 + r / 2 || x > context.canvas.width - r / 2) dx =- dx;
  if (y < 0 + r / 2 || y > context.canvas.height - r / 2) dy =- dy;
  x += dx;  // update x position according to the dx speed
  y += dy;  // update y position according to the dy speed

  // actualize cells of state
  cells = stateOfBall.getElementsByTagName('td');
  cells[0].innerHTML = `(${parseInt(x)}, ${parseInt(y)})`// position
  cells[1].innerHTML = `(${dx * 100}, ${dy * 100})`;  // speed
  cells[2].innerHTML = r;  // radius
  cells[3].innerHTML = color;  // color
}



Interface - Chatbot communication using an API




The code below is the result of the turorial to design an API with python using Flask. To make it very simple, it's the server. The user will send requests as a message to the API from the web interface. This API will request the chatbot with the same message and it will be returned a reply in the format of a python dictionary. This dictionary will be converted to JSON format and returned as a response to the web interface.

I invite you to copy and paste the code below into the main.py file.


WARNING: We don't have a chatbot.py file yet. So we will have to wait for the next step for this script to work. Also keep in mind that the only entry point of the chatbot from the API is the Chatbot.query() function.


File main.py
import time
import json
import threading
from datetime import datetime

from flask import Flask, jsonify
from flask_cors import CORS

from chatbot import Chatbot

# create the Flask application
app = Flask(__name__)
# make cross-origin AJAX possible
CORS(app)

@app.route("/chatbot-request/<string:request>")
def send_request_result(request):
    """
    Respond a request made from a client
    :param request: chat request to respond as string
    :return: response and its metadata
    """
    data = {'t0': time.time()}  # get the timestamp at beginning
    # set the response of th chatbot in the dictionnary
    data['response'] = app.chatbot.query(request)
    data['t1'] = time.time()  # get the timestamp at beginning
    # convert into JSON format
    return jsonify(data)

def start_api(host, port):
    """
    Start the API on a thread
    :param host: host of the API
    :param port: port of the API
    """
    # get the host and the port as keywords attributes for app.run()
    app_kwargs = {'host':host, 'port':port}
    # run the app on a thread
    threading.Thread(target=app.run, kwargs=app_kwargs).start()

if __name__ == '__main__':
    app.chatbot = Chatbot()
    # set the host and the port (here in localhost)
    HOST, PORT = '127.0.0.1', 4000
    # start the API on the thread
    start_api(host=HOST, port=PORT)


Rule-Based Chatbot : Basics


Now we have everything in place for user-chatbot communication. It will be necessary to understand the sentence that the user will have written in a "human" way. In other words, understand what the user's intent is. Then we will have to extract the key data. To put it simply, when a user says hello, we must understand that the user's intent is a greeting. Once we know that the intent is a greeting then we can extract the data which is, for example here, greeting that matches the greeting used.


To make our chatbot as flexible as possible, we will configure it from a rules.json file in JSON format. You can see some of the content below. To each rule corresponds an intent name, a regex and we can put a list of sentences to test our regex.


Obviously you won't be able to understand how it works if you don't know the regular expressions. (cf. regex tutorial)


Let's start with the basics. We want our chatbot to be able to understand and respond to greetings, goodbyes, thanks and requests for help. So we have the rules.json file below.


File rules.json
[
    {
        "intent": "greetings",
        "regex": "(?P<greeting>hi|hello|hey|hola)",
        "positive_tests": [
            "Hi there",
            "hello my dear !",
            "Hey !",
            "Hola",
            "I say you hello !"
        ]
    },
    {
        "intent": "goodbye",
        "regex": "(bye|see\\syou|goodbye)",
        "positive_tests": [
            "Bye",
            "see you !",
            "googbye !",
            "Goodbye",
            "See you later :) "
        ]
    },
    {
        "intent": "thanks",
        "regex": "(thanks?\\s?[.!]?|(help|use)ful).*[^\\?]$",
        "positive_tests": [
            "thanks",
            "Thank you !",
            "Awesome, thanks",
            "You are very useful !",
            "I want to thank you for your help"
        ]
    },
    {
        "intent": "help",
        "regex": "(^help$|(work|provide|What.*do|useful).*\\?)",
        "positive_tests": [
            "help",
            "How do you work ?",
            "What options you provide ?",
            "What can you do ?",
            "How can you be useful ?"
        ]
    }
]


Now that some rules are explicitly written down, we can finally get on with the building of our chatbot. In the chatbot.py file, you can copy and paste the code below. It allows you to initialize a chatbot from rules written in the rules.json file. It tests all the rules with the test sentences one by one and then intersects the different regex with the different test sentences to create and print a confusion matrix. This allows us to check that our regex are only matching with our test sentences that we want to be positive and are not matching with the others. Finally there is the query() function. As we saw in the previous API schema, this is the function called by the API with the user's query (the text message) as input from the interface.


This query is tested with all regex. If it doesn't match with any regex then the chatbot notifies that it didn't understand the user's intent. If it matches a regex then it calls the function exactly named by the intent defined in the rules.json file. If this function is not implemented or doesn't work, the chatbot will send the information back to the user (which allows to debug when building the chatbot).


File chatbot.py
import re
import json
import random

class Chatbot:
    def __init__(self, rules_path='rules.json'):
        self.response = {}
        # open the rules and read the rules, which define the chatbot
        with open(rules_path, 'r') as json_rules:
            self.rules = json.load(json_rules)
        self.test_rules()  # test the rules at the beginning

    def test_rules(self):
        # test each rules with each of its positive sentences given
        for rule in self.rules:
            regex = re.compile(rule['regex'], re.I)
            positive_tests = rule['positive_tests']
            tests = []
            for sentence in positive_tests:
                tests.append(len(regex.findall(sentence)) > 0)

            accuracy = sum(tests) / len(tests) * 100
            print(f'Test rule of intent {rule["intent"]} : {accuracy:.0f}%')

        # compute the confusion matrix of regex with positive sentences
        range_N = range(len(self.rules))
        confusion = [[0 for i in range_N] for j in range_N]
        print('\nConfusion Matrix :')
        for i in range_N:  # for each regex
            regex = re.compile(self.rules[i]['regex'], re.I)
            for j in range_N:  # for each batch of positive test
                positive_tests = self.rules[j]['positive_tests']
                res = [len(regex.findall(test)) > 0 for test in positive_tests]
                confusion[i][j] = sum(res)
        # replace zeros by dots to easily visualize
        confusion = [[str(y) if y else '.' for y in x] for x in confusion]
        confusion = '\n'.join([' '.join(x) for x in confusion])
        print(confusion)

    def query(self, request):
        # here is the entry point of the chatbot
        self.response = {}

        for rule in self.rules:  # test the request with each rule
            m = re.compile(rule['regex'], re.I).search(request)
            # if match, catch the associated method
            if m:
                intent = rule['intent']
                # check if the method is implemented and functionnal
                try:
                    eval(f'self.{intent}')(request, m)
                except Exception as e:
                    response = f'The "{intent}" method'
                    response += ' is not implemented or is not functionnal'

                    self.response['message'] = response
                    self.response['error'] = str(e)
                    print(e)
                finally:
                    self.response['answer_to_intent'] = intent
                    break
        # case of no match between the request and the rules
        if self.response == {}:
            response = 'Sorry I did not undestand your intent...'
            self.response['message'] = response
        return self.response


We can then try from the interface to communicate with the chatbot.


WARNING: Don't forget to run the main.py script in parallel.





We can see by trying that our regex matches well with the user's text requests. On the other hand, the chatbot functions to respond to the intents are not yet implemented.


So let's go.


We add to our chatbot the functions corresponding to the intent defined in the rules.json file. First the greetings() function which responds by greeting with the same politeness formula as the user. Then the goodbye() function which randomly chooses an answer from an explicitly given answer set. The same is true for the thanks() and help() functions.


File chatbot.py continuation
    def greetings(self, request, m):
        greeting = m['greeting'# extracted greeting from regex
        # repeat the greeting in a sentence
        # be sure the first character is in uppercase
        greeting = greeting[0].upper() + greeting[1:]
        self.response['message'] = greeting + ' to you too !'



    def goodbye(self, request, m):
        # set of response
        possible_response = [
            'See you !',
            'Goodbye !',
            'Have a nice day !',
            'Bye ! Come back again soon.',
        ]
        # select a response randomly
        self.response['message'] = random.choice(possible_response)

    def thanks(self, request, m):
        # set of response
        possible_response = [
            'Happy to help !',
            'Any time !',
            'My pleasure !',
            'You\'re welcome !',
        ]
        # select a response randomly
        self.response['message'] = random.choice(possible_response)

    def help(self, request, m):
        # enumerate the options
        options = '\n'.join([
            'I can :',
            '  - Answer to greetings and goodbye',
            '  - Answer to thanks',
            '  - Change the speed of the ball on an axis',
            '  - Change the size of the ball',
            '  - Change the color of the ball'
        ])
        self.response['message'] = options




Rule-Based Chatbot : Change the ball


Now that everything is in place it is very simple to add new intentions, which the chatbot will understand and be able to respond to.


We start with the color of the ball. With a regex you try to catch the words calling for a change and at the same time you try to catch the color in hexadecimal. (So it won't work if the user asks for "red", he will have to ask for "#f00", "#ff0000", "#F00" or "#FF0000". But you can easily add the colors of your choice in the regex)


File rules.json continuation
    {
        "intent": "change_color",
        "regex": "(change|set|update|see).*(?P<color>#(([A-Fa-f0-9]{3}){1,2}))",
        "positive_tests": [
            "can you change the color to #ffffff ?",
            "Set the color of the circle to #2f9599",
            "I want to see a color of #8800ff",
            "Change to #999 the color",
            "update the color to #abc"
        ]
    }


The rule is defined in the rules.json file so we just have to define the method of the same name in the chatbot. I added the random_confirmation_sentence() function to make the chatbot's response more realistic.


File chatbot.py continuation
    def random_confirmation_sentence(self):
        # set of response
        possible_confirmation = ['Fine ', 'Ok ', 'Don\'t worry ', 'Great']
        possible_ponctuation = ['.', '!', '...', ':']
        possible_new_start = [' ', '\n', '\n\n']
        response = random.choice(possible_confirmation)
        response += random.choice(possible_ponctuation)
        response += random.choice(possible_new_start)
        return response

    def change_color(self, request, m):
        response = self.random_confirmation_sentence()
        response += f'I have set the color to {m["color"]}'
        self.response['message'] = response
        self.response['color'] = m['color']




Then you can also enjoy setting up a new rule to change the size of the ball. A little more tricky because the user can talk about size, radius or diameter. We will consider that the size corresponds to the default radius.


File rules.json continuation
    {
        "intent": "change_radius",
        "regex": "(change|set|update|see).*(?P<change>radius|diameter|size)",
        "positive_tests": [
            "can you change the radius to 20px ?",
            "Set the size of the circle to 30px",
            "I want to see a diameter of 75",
            "Change to 10 the radius",
            "update the diameter to 50px"
        ]
    }


You may have noticed that with this regex, we did not extract the radius value requested by the user as we did with the color. This is simply to avoid having an extended regex because the user can put this value in too many different places in his sentence. That's why we will extract this value directly into the change_radius() function with another regex.



File chatbot.py continuation
    def change_radius(self, request, m):
        # extract the radius from the request (can be float or int)
        regex_radius = '\s(?P<radius>[+-]?([0-9]*[.])?[0-9]+)(\s|\-|\.|px|$)'
        radius = re.compile(regex_radius, re.I).search(request)['radius']
        radius = float(radius)
        # if the user talk about diameter of course it two time the radius
        if m['change'] == 'diameter':
            radius /= 2

        response = self.random_confirmation_sentence()
        response += f'I have set the radius of the ball to {radius}px'
        self.response['message'] = response
        self.response['r'] = radius





Finally we can also create a regex to change the speed of the ball. Looking at the positive sentences and the regex, we can see that we will have to extract the value requested by the user in the same way as in the function to change the radius. In addition to this, we will allow to the user to also specify an axis on which to change the speed. By default we will speak of normal speed but if we succeed in extracting an axis in the sentence then we will only change the value on that axis.



File rules.json continuation
    {
        "intent": "change_speed",
        "regex": "(change|set|update|see).*(speed|velocity|rapid|celerity)",
        "positive_tests": [
            "can you change the speed on x axis at 500px ?",
            "Set the y-axis velocity to 180px",
            "I want to see a rapid on x of 350",
            "Change to 800 the y rapidness",
            "update the celerity for x to 750px"
        ]
    }



File chatbot.py continuation
    def change_speed(self, request, m):
        # extract the speed from the request (can be float or int)
        regex_speed = '\s(?P<speed>[+-]?([0-9]*[.])?[0-9]+)(\s|\-|\.|px|$)'
        m_speed = re.compile(regex_speed, re.I).search(request)['speed']
        speed = float(m_speed)

        # check if the user asked for a specific axis or speed
        regex_axis = '\s(?P<axis>[xy])(\s|\-|_)'
        m_axis = re.compile(regex_axis, re.I).search(request)
        if m_axis is None:
            response = self.random_confirmation_sentence()
            response += f'I have set the speed to {speed}px/s'
            self.response['message'] = response
            speed = speed / (2 ** 0.5)
            self.response['dx'] = speed
            self.response['dy'] = speed
        else:
            axis = m_axis['axis'].lower()
            response = self.random_confirmation_sentence()
            response += f'I have set the {axis}-axis speed to {speed}px/s'
            self.response['message'] = response
            self.response[f'd{axis}'] = speed




If you have understood all this mechanism you will now be able to create a rule to add as many balls as the user wants for example.If you want to try this you will need to modify the script.js file.