Let’s assume that you have a very simple business, and you want to deploy a chatbot for customer support. Let’s assume 100 questions and answers (Q/A)s would cover all your issues. It looks very simple, and you may be tempted to deploy one of the deep learning methods to build your chatbot. Here are the problems you are going to face:

COMBINATORY EXPLOSION IN NATURAL LANGUAGES

Unless you are a trained linguist, you might easily undermine how flexible natural language can be, and how explosive the combinations will emerge out of a single question. Let’s say your first Q/A starts with a basic complaint the users will have something like “I have a problem with my cable.” This simple statement can be expressed in more than a dozen ways as shown below, and the combinations do not end there!

If we take one of the possible expressions above, there can be another dozen combinations only by morphological and synonymous variations:

As you can see, this is only the first Q/A from your set of 100. Just imagine if some of the (Q/A)s you have are more complicated than this simple starting expression.

Your set of 100 (Q/A)s can easily mount to 10,000 different equivalent expressions the users may type which must be detected and understood by your chatbot software.

SO, WHAT IS THE PROBLEM?

The problem is not the deep learning method itself, but what it needs to function properly. You need to have a data set of 10,000 questions, if not more, that are linguistically equivalent expressions as shown above. Also, these 10,000 questions should map to 100 answers in this hypothetical case. Unless someone sits down and types them one by one, such a data set will be a nightmare to acquire.

If you already have a customer support system and collected, let’s say, 1 million (Q/A)s, there is still no guarantee that this 1 million (Q/A)s will cover the 10,000 linguistic variations to detect the 100 main (Q/A)s. Considering the Gaussian distribution of a typical user response analytics, 1 million (Q/A)s would cover less than 30% of your required data set. Your chatbot solution will remain vulnerable to undetected responses after all that trouble.

Consequently, someone who is deploying a deep learning method will find himself/herself in a data crises situation quickly. No matter what type of deep learning method you deploy, the data requirement described here holds. Neural networks cannot discover themselves equivalent variations of natural language without being provided ample examples. And I want to underline the word “ample” here.

OTHER TYPES OF DATA CRISES WITH DEEP LEARNING

Going back to the hypothetical case where you have a service operation and you can pull 1 million (Q/A)s. To make sure this data set will not cause any harm, someone must manually go through the set to clean it up. You cannot just dump data to a deep learning system without verifying it. Remember the Microsoft case, where Tay, the chatbot developed using twitter feeds, started to produce racial statements.

Learn-as-you-go approach also poses problems. Deep learning methods require a training process and convergence before deployment. This can be a long process. Once trained, the system cannot simply absorb new data in an addition mode. The entire data set must be trained again. As a result, if you plan to add new data to your chatbot every week, you need a team of AI specialists training the system every week and re-deploy. As one can imagine, this does not seem like a scalable business solution.

WILL USING BUTTONS SOLVE THE PROBLEM?

Facebook, when they launched the chatbot platform, assumed that buttonizing conversations could solve part of the combinatory explosion problem described here. First of all, let’s make one thing clear:

If the user is not allowed to enter free expressions any time during conversation, it is no longer a chatbot, or conversational AI. It is a toy.

Most Facebook chatbot developers jumped on the idea of buttonizing entire conversations, thus yielding nothing more than a toy. Most of the 30,000 plus chatbots developed in this fashion flopped big time, only few succeeded as reported in several recent articles prior to Facebook’s recent summit meeting. Entirely buttonized conversations can rarely provide successful solutions for very particular business types. If buttons are used alongside free expressions successfully detected, then this combination can be powerful.

WHERE IS THE SOLUTION?

I intend to write more about the solutions later. However, in a nut shell, solutions to the chatbot problem require independent NLP solutions before a deep learning methods can be used. One thing is for sure, deep learning alone is not a good fit, and has no future with this “silver bullet” engineering mentality.

——- FOLLOW US ———-

Join our CHATBOTS linkedin group

Follow exClone in Linkedin or on Facebook

For exClone’s Platform, click here for free trial.

There is a very easy distinction between a chatbot and a search engine which explains almost everything: SHORT-TERM MEMORY.

A search engine, like Google, has no short-term memory. Google will take your query, and bring results. The job is done. The next query you have is completely new to it. It is a new session with no ties to the previous query.

A chatbot, on the other hand, can remember 2, 3, 4, or N steps back, which gives it a huge advantage in responding better, more focused, and with higher accuracy. Especially, applications like “advisor” chatbots can take advantage of this fact. However, remembering N steps back poses a challenging technical problem that can grow in a combinatory fashion. Without getting into such technical details, let’s see an example.

Multiple Questioning Before Presenting Answers
A showcase example is the chatbot Davis Hunter which is designed to find you new travel destinations based on your choices. The multiple questioning operation uses short-term memory which is shown below.

At the end of the questioning steps, the chatbot presents travel destinations with precision. It has used its short-term memory to remember all your inputs before making a decision on its list of destinations. Once the user selects from the options, then Davis will start to present more information about the destination using the free content from Wikivoyage.

The operation shown above is a blue print of any kind of advisory chatbot in any subject.

If you type the same query to Google: “island in spain that has festivals and good seafood restaurants” you will end up with poor results as shown below. Simply because your query is too long and falls into long-tail, a region where search engines cannot handle queries.

Search will Shift to Specialized Chatbots in the Near Future
It is fair to assume that conventional search will die out as “Google generation” is steadily replaced by “Siri generation” who are more inclined to use messaging and chatting platforms. This transformation is already at works and is expected to accelerate as chatbots get better and spread in every vertical.

The expectation that a search engine user will sift through dozens of inaccurate results is increasingly becomming obsolete and intolerable for the new generation who grew up with persistent messaging habits highly suitable for chatbot interaction.

The key point in this transformation is the ability to create quality chatbots with an easy and familiar effort (like writing a blog entry) that would accelerate the proliferation of viable chatbots in every subject.

————–

Join our CHATBOTS linkedin group

Follow exClone in Linkedin or on Facebook

For exClone’s Platform, click here for free trial.