When we initially developed Cloudbot, it could only understand very specific commands based on a word-matching algorithm [1]. One of our next goals was to add the capability to understand commands in natural language. This meant that our users are no longer required to remember complex commands and flows. Instead, they can chat with our bot as they would with a human.

The key features of our cognitive Cloudbot are to:

  • Understand natural language
  • Extract entities from natural language

  • Continuously learn from interactions with users
Share learned data with developers and other deployed bots

In this blog post, we describe how we’re implementing each of these features. Our bot is extensible, and the code is open source, so you can use it as a starting point to build your own cognitive bot.

Understanding natural language

If you have interacted with bots in the past, you may have noticed that a common problem is that most of these bots have trouble understanding the meaning of your requests. That’s because they try to match specific words or phrases, but the bot doesn’t have the intelligence to understand the meaning of those words it’s trying to match.

We use Watson’s Natural Language Classifier service to match the statements from the user to the commands our bots know. This allows the user to speak naturally to the bots. Before this can happen, the service is trained with the intents (commands) that the bot supports and a few sample statements that match each intent. The service requires as little as two training statements per intent to start generating good results when presented with new statements. The accuracy can be improved by adding more training statements, in the “Continuous Learning” section I explain the tools we have created to help you improve your training data by learning from your users.

Extracting entities from natural language

Most Cloudbot commands have variable entity values associated with them. For instance, when starting an application, the name of the application is required. Once a class is associated with a statement, the next challenge is to extract these variable entity values from the statement.

To make this possible, each command defines the set of entities to expect. This definition enables the entity processing to make a best guess at finding the value in the statement. In some cases, the entity value can not be found in the statement. This could be because it was not actually entered or there were multiple possibilities. If the entity value is required, then a bot conversation is started with the user to ask them to specify the entity value.

Various facilities are used to extract entity values from a statement. The Watson Alchemy API is used to find city names for our weather commands. A parts-of-speech parser is used to find nouns and numbers within a statement in order to extract entities, keywords, or numbers. Regular expressions are used to find some entity values such as those related to repositories.

Continuous learning from interactions with users

Our goal is to continuously train Cloudbot from the user’s actions to improve Cloudbot’s ability to match the user’s statements and ultimately understand and match every statement correctly.

When a statement is classified, we sort it into one of three categories: high confidence, medium confidence, and low confidence. The bot responds to each confidence level differently and each bot has the option to record the user statements along with confidence level in a database to allow for future analysis and learning.

High confidence statements are processed immediately by the bot because the high confidence indicates that the bot understands the natural language used by the user. The classified statement and the selected class are recorded in the database for analytics and performance calculations.

A classification of medium confidence indicates that the bot may have some understanding of the statement, but it is not confident enough to perform the action. Therefore, we learn the medium confidence statements by giving the user a list of the top choices and asking the user for feedback. If the user chooses a class from the list, the statement and the learned classification is saved in the database.

Low confidence statements indicate minimal understanding. In this case, the bot tells the user it does not know how to process the statement and asks for the user to rephrase the statement. These low confidence statements are also stored in the database, along with the conversation logs. The bot is also capable of identifying statements that indicate a negative reaction or frustration from the user. Statements from the user such as, “That’s not what I meant,” are recorded with the context of the conversation for further analysis.

We created a web application to help a bot admin quickly review the medium and low confidence statements. The web app displays the statement and top classes along with additional information collected from the user either regarding the chosen classification for the medium confidence statements or the conversation logs for the low confidence statements. From this web application the bot admin can choose to add any of these statements to the training data for Watson’s Natural Language Classifier. This web app can be used for any bot developed with our NLC integration and is open source, so you can use it to help train your bot.

Cloudbot user interface

Data sharing with synchronization

To enable actionable insights, the developer has the option to synchronize to an external database the interactions with the bot that are useful for natural language training and post analyses. It’s important to capture the chat from users with the bot in a way that does not impact the responsiveness of the bot. Writing chat metadata locally and synchronizing that data from all users of the bot at periodic intervals maintains the usability and responsiveness of the bot.

However, synchronization is a difficult problem to solve. We want the data flow to be two ways—both writing from the bot and then having this data reviewed and synchronized back to the bot when it has been marked as approved. We have made design choices to reduce the number of potential data conflicts by using an eventually consistent, multi-master document database. For Cloudbot, we developed in NodeJS and are using PouchDB with a Cloudant-managed database service for synchronization.

Chat metadata is written locally within the container to a PouchDB database which periodically synchronizes this data over HTTPS to the Cloudant Database as a Service (DBaaS). The local PouchDB database also receives changes from the Cloudant database to trigger that the newly learned data has been approved and that the bot should retrain.

The synchronization protocol is an open protocol. Read the documentation.

To avoid having to resolve document revision conflicts, the bot data model adds a new document for every interaction that requires review or is used in post-analysis. The data flow is as follows:

  1. An administrator for the bot marks the data as approved within Cloudant.
  2. This data is then synchronized back to the bot.
  3. The bot initiates the retrieval of the the approved data from the remote server.

With this data flow we don’t have the issues of having to have all the bots online when data is approved and pushed out.

We hope this was a useful introduction to the natural language processing capabilities of Cloudbot. We hope that you try out Cloudbot and help us continue to evolve its cognitive capabilities!

Reference notes

[1] Each Cloudbot command would have a regular expression used to match the user input to. The regular expression could contain optional words but it could not understand natural language.


Authors of this blog post are Jorge Padilla and Megan Becvarik.

The post Develop a cognitive Cloudbot with IBM Watson appeared first on developerWorks Open.

via developerWorks Open http://ibm.co/2cyX8VV

September 27, 2016 at 09:03AM