Not As Easy As It Looked
Eeny meeny miny moe, which test case do I want to show?
Test Case 0 Failed!!!
Input: Will this test pass?
Output: I’m sorry, could you try rewording that?
Expected: I predict that this test will pass
This doesn’t look so bad. We already wrote a rule for “Is X Y?” so writing a rule for “Will X Y?” should be as easy as copy pasting and switching a few words. Behold!
push(@chatPatterns,
[qr/\AWill ([a-zA-Z]+) (.+)\?\z/,
"I predict that UIF0 will UIF1"]);
I’ll just drop that into the rules list right after the “Is X Y?” rule and we should be good to go.
Test Case 0 Failed!!!
Input: Will this test pass?
Output: I predict that this will test pass
Expected: I predict that this test will pass
Uh oh. That didn’t quite work. DELPHI did manage to figure out that test 0 was a “Will X Y?” style question but when generating the answer it put the “will” in the wrong place. Can you figure out why?
[Please use this break in the blog’s flow to consider why this happened.]
The problem here has to do with how we defined the rule. We’ve been calling it “Will X Y?” but the rule is actually more like “Will Noun Verb?” or “Will Noun-Phrase Verb-Phrase?”.
Our current dumb rule assumes that the noun will always be the first word after the word “Will” and that everything else will be part of the verb phrase. This works out great for sentences like “Will Batman catch the villain?” but completely falls apart when you start adding adjectives to the noun and get things like “Will the police catch the villain?”
So what we really need is a “Will” rule that is smart enough to group common adjectives with the noun and treat them all like one big super-noun. Here is a quick first pass (WARNING: WEIRD REGULAR EXPRESSION AHEAD):
/\AWill ((?:(?:this|the|that|a|an) )*[a-zA-Z]+) (.+)\?\z/
Don’t panic just yet, this rule is actually a lot simpler than it looks. But first you need to understand what all those “?:” symbols are doing. Hopefully you remember that parenthesis create capture groups that group patterns together and then store their matches for future use. But sometimes you want to group patterns together without storing them for later. You can accomplish this by starting your capture group with the special symbols “?:”, which then turns of the capturing and lets you use the parenthesis as a simple grouping tool.
This is important for our “Will” rule because we want to capture the entire noun-phrase and the entire verb-phrase but we don’t want to capture any of the individual parts of those phrases. For example, we have improved our noun-phrase by adding in two groups of nested parenthesis for handling common article adjectives. The inner parenthesis match common adjectives and the outer parenthesis make sure there is a space following each adjective. We mark both these rules as “?:” noncapturing because while we certainly do want to match nouns that start with a series of adjectives we only want to capture those adjectives as part of the noun and not on their own.
What would happen without those noncapturing symbols? Well, the first big parenthesis set would capture the entire noun-phrase and substitute it into the output just like we want. But the second capture group wouldn’t be the verb-phrase like we originally wanted. Instead the second capture group would be the inner parenthesis matching the articles leading to all sorts of problems. See for yourself:
Input: Will this test pass?
Output: I predict that this test will this
Expected: I predict that this test will pass
See what I mean? We successfully grabbed “this test” and put it into the answer as a noun-phrase but we then grabbed “this ” as our second capture group while the verb-phrase “pass” got pushed into a later capture group slot. Not what we wanted at all.
Instead we’ll just tell the inner parenthesis not to capture. Now the noun-phrase always goes in slot one and the verb-phrase always goes in slot two and everything works wonderfully.
Test Case 0 Passed
…
Passed 7 out of 11 tests
Wait A Minute, Isn’t This A Problem For “Is” Rules Too?
Clever readers might be asking themselves “If adjectives broke our simple “Will X Y?” rule, then won’t they break our old “Is X Y?” rule too?” Well good job for noticing that clever readers, because that’s the exact problem we see in our next test case:
Test Case 1 Failed!!!
Input: Is the sky blue?
Output: Fate indicates that the is sky blue
Expected: Fate indicates that the sky is blue
Fortunately we can fix it the exact same way:
/\AIs ((?:(?:this|the|that|a|an) )*[a-zA-Z]+) (.+)\?\z/
Test Case 1 Passed
…
Passed 8 out of 11 tests
You Shouldn’t Copy Paste Code
There is one little problem with this approach to adjectives: I’m hard coding a big list of words and then copy pasting it into multiple functions. This will be a real pain if we ever have to update the list in the future. For instance, if we wanted to add possessive adjectives into the list (my, your, his, her, their) we would have to rewrite two different rules. And if we ever decide a third rule needs access to the list we’ll have to copy paste the whole thing.
Much better to turn that portion of the rules into a separate variable that can be included in multiple functions. Which in Perl you can do like this:
#put this before the code starts to build the pattern and response array my $commonAdjectives=qr/(?:this|the|that|a|an)/; my $noncaptureAdjectiveChain=qr/(?:$commonAdjectives )*/;
And now we can just update the rules to use these handy variables anywhere we want to match an arbitrarily long chain of adjectives with a single space after every word.
/\AIs ($noncaptureAdjectiveChain[a-zA-Z]+) (.+)\?\z/
/\AWill ($noncaptureAdjectiveChain[a-zA-Z]+) (.+)\?\z/
Those of you following along in a language other than Perl will have to figure out on your own how and if your language handles inserting variables into a regular expression. If all else fails you can always just go back to the copy pasting thing.
Let’s Test By Adding Some More Adjectives
Now that we can add new adjectives to two different rules by just updating a single variable we should write a few new tests and make sure it works. How about these?
$testCases[11][0] = "Will his code compile?" $testCases[11][1] = "I predict that his code will compile"; $testCases[12][0] = "Is this big blue box actually a time machine?"; $testCases[12][1] = "Fate indicates that this big blue box is actually a time machine";
The first test is a straightforward test to make sure we can add possessives to the adjective list. The second test is a little bit more complex, requiring us to not only add two new adjectives to our list (big and blue) but also testing to make sure the code can chain multiple adjectives together into a row.
Of course, right now they both fail. The first test doesn’t recognize “his” as an adjective so it assumes it is a noun and puts the “will” in the wrong place. The second test recognizes “this” as an adjective but not “big” and does the same thing.
Test Case 11 Failed!!!
Input: Will his code compile?
Output: I predict that his will code compile
Expected: I predict that his code will compile
Test Case 12 Failed!!!
Input: Is this big blue box actually a time machine?
Output: Fate indicates that this big is blue box actually a time machine
Expected: Fate indicates that this big blue box is actually a time machine
But after updating our list of adjectives:
my $commonAdjectives=qr/(?:this|the|that|a|an|his|her|my|your|their|big|blue)/;
Test Case 11 Passed
Test Case 12 Passed
——————–
Passed 10 out of 13 tests
Test Failure!!!
How Many Adjectives Do We Need?
DELPHI now knows how to handle 12 different adjectives. And while that is pretty nifty it’s worth pointing out that the English language has a lot more than just 12 adjectives. In fact, English is one of the world’s largest languages* and easily has several tens of thousands of adjectives. Even worse, English allows you to “adjectivify”** other words to creating new adjectives on the spot, like so:
“These new computery phones have a real future-licious feel to them but with the default battery they’re actually kind of brickish.”
My spell checker is convinced that sample sentence shouldn’t exist but even so you probably understood what I meant. Which just goes to show the huge gap in how good humans are at flexible language processing and how bad computers still are.
But what does this mean for DELPHI? Do we need to generate a giant adjective list? Do we need to teach it how to handle nouns and verbs that have been modified to act like adjectives? Do we need to spend twelve years earning multiple PhDs in computer science and linguistics in order to build a more flexible generateResponse function?
Well… no. Remember, our goal isn’t to create a program that can fully understand the human language. We just want a bot that can answer simple questions in an amusing way like some sort of high-tech magic eight ball. As long as DELPHI can handle simple input and gracefully reject complex input it should feel plenty intelligent to the casual user.
Furthermore, we can actually depend on users to play nice with DELPHI. Most people, after being scolded by DELPHI once or twice for trying to be clever will start to automatically pick up on what sorts of inputs do and don’t work. The fact that DELPHI can’t handle obscure adjectives will eventually teach users to stick to straightforward questions.
All things considered we can probably “solve” the adjective problem by teaching DELPHI the hundred most common adjectives in the English language and then hoping that users never bother going beyond that. Later on we can have some test users talk to DELPHI and use their experiences to decide whether or not we need to add more adjectives or build a more complex system.
Conclusion
Today we caught a glimpse of how simple pattern matching chatbots can completely fall apart when confronted with real English. But we also saw a quick way to band-aid over the worst of these problems and we have hope that our bot can be written in such a way that users never noticing that DELPHI is too dumb to understand that “house” and “that big house over there” are actually the same thing.
Next time, more test cases and more examples of English language features that are annoying to program around.
* As the popular saying goes: English has pursued other languages down alleyways to beat them unconscious and rifle their pockets for new vocabulary.
** Look, I just verbed a noun!