Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't extract numbers from intents. #5

Open
mig82 opened this issue Jun 19, 2021 · 4 comments
Open

Can't extract numbers from intents. #5

mig82 opened this issue Jun 19, 2021 · 4 comments

Comments

@mig82
Copy link

mig82 commented Jun 19, 2021

I've tried everything I could think of and have not been able to extract a number from an intent sent by the JovoWebClient lib. Meaning that this.$inputs is always empty —i.e. {}.

My intent looks like this in models/en-US.json:

{
	"name": "LoremIpsumResolve",
	"phrases": [
		"{wordCount} words",
		"{wordCount} lorem words",
		"{wordCount} lorem ipsum words",
		"{wordCount} latin words",
		"say {wordCount} words",
		"say {wordCount} words in latin",
		"speak {wordCount} words in latin",
		"say {wordCount} latin words",
		"speak {wordCount} latin words",
		"speak {wordCount} words of lorem ipsum",
		"give me {wordCount} words in latin"
	],
	"inputs": [
		{
			"name": "wordCount",
			"type": {
				"alexa": "AMAZON.NUMBER",
				"googleAssistant": "actions.type.Number",
				"nlpjs": "number"
			}
		}
	]
}

It works perfectly fine with Alexa or Google Assistant, but won't work when the request comes from a client web application. I've also tried "nlpjs": "Number", "nlpjs": "integer" and "nlpjs": "Integer".

I noticed that $request coming from Alexa and Google assistant bears "type": "IntentRequest", but intents coming from the JovoWebClient lib bear "type": "TEXT", so then I tried sending the request as a RequestType.Intent from the web client:

client.createRequest({
	//type: RequestType.Text,
	type: RequestType.Intent,
	body: { text }
}).send()

This resulted in the $request bearing "type": "INTENT", but that didn't do the trick either.

I also noticed that this.$nlu carries NLP.js entities (https://github.com/axa-group/nlp.js/blob/master/docs/v3/builtin-entity-extraction.md#number-extraction) but there's no resolution.

@mig82
Copy link
Author

mig82 commented Jun 22, 2021

Just to clarify, the console output I'm getting while trying this out is below.

Notice how I start by saying say 5 words of lorem ipsum and it starts off by getting it wrong, sending me to the intent handler where I ask How many words do you need?, as if I had not specified the number.

That aside, I reply with 5 words, and then it gets it right and goes to the intent handler I meant for this, but printing out $inputs renders {}.

Now, also notice that on this last intent handler, I log this.$nlu...

console.log(JSON.stringify(this.$nlu, null, 4))

and there's an NlpjsNlu object attached to it, that includes a classifications array showing it was able to get the intent right, but also "entities": [], meaning it failed to extract the entity, in this case 5.

 >>>>> Request - 2021-06-22T14:24:33.114Z 
{
   "version": "3.4.0",
   "type": "jovo-platform-web",
   "request": {
      "id": "417e5bce-cde1-47b6-9bf6-41051c8f7cd0",
      "timestamp": "2021-06-22T14:24:25.032Z",
      "type": "TEXT",
      "body": {
         "text": "say 5 words of lorem ipsum"
      },
      "locale": "en",
      "data": {}
   },
   "context": {
      "device": {
         "type": "BROWSER",
         "capabilities": {
            "AUDIO": true,
            "HTML": true,
            "TEXT": true
         }
      },
      "session": {
         "id": "d9a409ec-2791-4bd2-82e3-4249d5d16010",
         "data": {}
         },
         "lastUpdatedAt": "2021-06-22T13:37:05.098Z"
      },
      "user": {
         "id": "bcb22484-1a34-4e24-94cf-d764b40f8b90",
         "data": {}
      }
   }
}

DEBUG: ****** extending jovo handler WebApp
LOG: ******************************** NEW_SESSION
DEBUG: User ID: 'bcb22484-1a34-4e24-94cf-d764b40f8b90'
LOG: ******************************** ON_REQUEST

DEBUG: 
ON_REQUEST WebApp access_token: undefined

LOG: ******************************** onResponse

 <<<<< Response - 2021-06-22T14:24:45.315Z 
{
   "version": "3.4.0",
   "actions": [
      {
         "plain": "Ok! Let's get some Lorem Ipsum. How many words do you need?",
         "ssml": "<speak>Ok! Let's get some Lorem Ipsum. How many words do you need?</speak>",
         "type": "SPEECH"
      }
   ],
   "reprompts": [
      {
         "plain": "Ok! Let's get some Lorem Ipsum. How many words do you need?",
         "ssml": "<speak>Ok! Let's get some Lorem Ipsum. How many words do you need?</speak>",
         "type": "SPEECH"
      }
   ],
   "user": {
      "data": {}
   },
   "session": {
      "data": {},
         "_JOVO_STATE_": "LoremIpsumSlots.AwaitSlots"
      },
      "end": false
   },
   "context": {
      "request": {
         "nlu": {
            "intent": {
               "name": "LoremIpsumStart"
            }
         }
      }
   }
}

 >>>>> Request - 2021-06-22T14:25:11.197Z 
{
   "version": "3.4.0",
   "type": "jovo-platform-web",
   "request": {
      "id": "85c9e579-fbe6-4dc3-9649-7d98364179d8",
      "timestamp": "2021-06-22T14:25:11.192Z",
      "type": "TEXT",
      "body": {
         "text": "5 words"
      },
      "locale": "en",
      "data": {}
   },
   "context": {
      "device": {
         "type": "BROWSER",
         "capabilities": {
            "AUDIO": true,
            "HTML": true,
            "TEXT": true
         }
      },
      "session": {
         "id": "d9a409ec-2791-4bd2-82e3-4249d5d16010",
         "data": {},
            "_JOVO_STATE_": "LoremIpsumSlots.AwaitSlots"
         },
         "lastUpdatedAt": "2021-06-22T13:37:05.098Z"
      },
      "user": {
         "id": "bcb22484-1a34-4e24-94cf-d764b40f8b90",
         "data": {}
      }
   }
}

DEBUG: ****** extending jovo handler WebApp
LOG: ******************************** NEW_SESSION
DEBUG: User ID: 'bcb22484-1a34-4e24-94cf-d764b40f8b90'
LOG: ******************************** ON_REQUEST

DEBUG: 
ON_REQUEST WebApp access_token: undefined

WARN: Inputs: {}
DEBUG: LoremIpsumResolve Inputs: {}
DEBUG: typeof this.$nlu object
DEBUG: this.$nlu.prototype undefined
DEBUG: this.$nlu.constructor function Object() { [native code] }
DEBUG: {
    "intent": {
        "name": "LoremIpsumResolve"
    },
    "NlpjsNlu": {
        "locale": "en",
        "utterance": "5 words",
        "languageGuessed": false,
        "localeIso2": "en",
        "language": "English",
        "nluAnswer": {
            "classifications": [
                {
                    "intent": "LoremIpsumResolve",
                    "score": 1
                },
                {
                    "intent": "YesIntent",
                    "score": 0
                },
                {
                    "intent": "GetForexRateStart",
                    "score": 0
                },
                {
                    "intent": "TransferStart",
                    "score": 0
                },
                {
                    "intent": "NoIntent",
                    "score": 0
                },
                {
                    "intent": "LogoutIntent",
                    "score": 0
                },
                {
                    "intent": "test",
                    "score": 0
                }
            ]
        },
        "classifications": [
            {
                "intent": "LoremIpsumResolve",
                "score": 1
            },
            {
                "intent": "YesIntent",
                "score": 0
            },
            {
                "intent": "GetForexRateStart",
                "score": 0
            },
            {
                "intent": "TransferStart",
                "score": 0
            },
            {
                "intent": "NoIntent",
                "score": 0
            },
            {
                "intent": "LogoutIntent",
                "score": 0
            },
            {
                "intent": "test",
                "score": 0
            }
        ],
        "intent": "LoremIpsumResolve",
        "score": 1,
        "domain": "default",
        "sourceEntities": [],
        "entities": [],
        "answers": [],
        "actions": [],
        "sentiment": {
            "score": 0,
            "numWords": 0,
            "numHits": 0,
            "average": 0,
            "locale": "en",
            "vote": "neutral"
        }
    }
}
LOG: ******************************** onResponse

 <<<<< Response - 2021-06-22T14:25:34.194Z 
{
   "version": "3.4.0",
   "actions": [
      {
         "plain": "How many words do you need?",
         "ssml": "<speak>How many words do you need?</speak>",
         "type": "SPEECH"
      }
   ],
   "reprompts": [
      {
         "plain": "How many words do you need?",
         "ssml": "<speak>How many words do you need?</speak>",
         "type": "SPEECH"
      }
   ],
   "user": {
      "data": {}
   },
   "session": {
      "data": {},
         "_JOVO_STATE_": "LoremIpsumSlots.AwaitSlots"
      },
      "end": false
   },
   "context": {
      "request": {
         "nlu": {
            "intent": {
               "name": "LoremIpsumResolve"
            }
         }
      }
   }
}

@m-ripper
Copy link
Contributor

m-ripper commented Jun 22, 2021

Hello there,

we just checked and it seems that the built-in entity extraction of nlp.js is not enabled by default.

You will need to do the following:

  1. npm i @nlpjs/builtin-default @nlpjs/lang-en - Installs the packages that are required for the entity extraction

  2. Make changes in the app.ts/js-file:

const { NlpjsNlu } = require('jovo-nlu-nlpjs');
const { BuiltinDefault } = require('@nlpjs/builtin-default');
const { LangEn } = require('@nlpjs/lang-en');

//
// Other content of app.ts/js
//

const nlpjsNlu = new NlpjsNlu({
  setupModelCallback: async (handleRequest, nlp) => {
    // force entity extraction
    nlp.forceNER = true;
    // register module for entity extraction
    nlp.container.use(BuiltinDefault, 'extract-builtin');
    // register module for English
    nlp.use(LangEn);
    // add corpus via path to the models directory
    // you might change it to pass an absolute path
    nlpjsNlu.addCorpus('./models');
  },
});

webApp.use(nlpjsNlu)

We will probably make this easier in the future, but for now, this workaround should work.

If you have any more questions, just let us know.

@mig82
Copy link
Author

mig82 commented Jul 1, 2021

Thank you @m-ripper. I've tried your suggestion but I'm getting this error:

src/app.ts:26:12 - error TS2341: Property 'addCorpus' is private and only accessible within class 'NlpjsNlu'.
26      nlpjsNlu.addCorpus('./models');

How did you get around this?

Also, how do you declare the type of NLP.js in the model? Is this correct?

"inputs": [{
	"name": "wordCount",
	"type": { "nlpjs": "number" }
}]

Thanks,

@mig82
Copy link
Author

mig82 commented Jul 2, 2021

@m-ripper, I got around the typescript compilation problem by switching to Javascript, but I'm still facing a few issues. Could you update one of the templates with a working example?

Here's what's happening:

  • All my requests are routed to the Unhandled handler. This only happens when the request comes from the web client. Requests from Alexa and Google Assistant work fine.
  • I tried creating an intent named SumsIntent that just does sums, and expects you to say something like "Add four plus five" or "Add 4 plus 5", and answers "The answer is 9" or "4 plus 5 is 9". What's happening is that only the last of the two numbers is found in the $inputs object.

image

Here's my adapted HelloWorld project to test this.

'use strict';

const { App } = require('jovo-framework');
const { Alexa } = require('jovo-platform-alexa');
const { GoogleAssistant } = require('jovo-platform-googleassistant');
const { JovoDebugger } = require('jovo-plugin-debugger');
const { FileDb } = require('jovo-db-filedb');
const { WebPlatform } = require ('jovo-platform-web');
const { NlpjsNlu } = require ('jovo-nlu-nlpjs');
const { BuiltinDefault } = require ('@nlpjs/builtin-default')
const { LangEn } = require ('@nlpjs/lang-en')
const app = new App()
const webPlatform = new WebPlatform();
const nlpjsNlu = new NlpjsNlu({
	setupModelCallback: async (handleRequest, nlp) => {
		// force entity extraction
		nlp.forceNER = true;
		// register module for entity extraction
		nlp.container.use(BuiltinDefault, 'extract-builtin');
		// register module for English
		nlp.use(LangEn);
		// add corpus via path to the models directory
		// you might change it to pass an absolute path
		nlpjsNlu.addCorpus('/Users/miguelangel/ws/node/my-embeddedchat/hello/models');
	},
})
webPlatform.use(nlpjsNlu);

app.use(
	new Alexa(),
	new GoogleAssistant(),
	webPlatform,
	new JovoDebugger(),
	new FileDb()
);

app.setHandler({
	LAUNCH() {
		return this.toIntent('HelloWorldIntent');
	},
	ON_REQUEST(){
		console.log(`**** ON_REQUEST Inputs: ${JSON.stringify(this.$inputs)}`) 
	},
	SumsIntent(){
		console.log(`**** SumsIntent with ${this.$inputs}`)
		let sum = parseFloat(this.$inputs.firstNumber.key) + parseFloat(this.$inputs.secondNumber.key)
		this.tell(`The answer is ${sum}`)
	},
	HelloWorldIntent() {
		this.ask("Hello World! What's your name?", 'Please tell me your name.');
	},
	MyNameIsIntent() {
		this.tell('Hey ' + this.$inputs.name.value + ', nice to meet you!');
	},
	Unhandled(){
		console.log(`**** Unhandled`)
		console.log(`this.getMappedIntentName: ${this.getMappedIntentName()}`)
		console.log(`this.getRoute: ${JSON.stringify(this.getRoute())}`)
		this.tell(`Darn! can't do that yet.`)
	}
});
module.exports = { app };

Here's my model for my SumsIntent:

{
	"name": "SumsIntent",
	"phrases": [
		"how much is ${firstNumber} plus {secondNumber}",
		"add ${firstNumber} plus {secondNumber}",
		"add ${firstNumber} and {secondNumber}",
		"${firstNumber} plus {secondNumber}"
	],
	"inputs": [
		{
			"name": "firstNumber",
			"type": {
				"alexa": "AMAZON.NUMBER",
				"googleAssistant": "actions.type.Number"
			}
		},
		{
			"name": "secondNumber",
			"type": {
				"alexa": "AMAZON.NUMBER",
				"googleAssistant": "actions.type.Number"
			}
		}
	]
}

Notice how I removed the "nlpjs": "number" property from the type property for both firstNumber and secondNumber. It appears to make no difference. Should I add it back? This is not documented anywhere that I could find.

Here are the request and response.

>>>>> Request - 2021-07-02T13:08:20.462Z 
{
   "version": "3.4.0",
   "type": "jovo-platform-web",
   "request": {
      "id": "c69ed9a2-8823-4376-940c-1da4efd9a858",
      "timestamp": "2021-07-02T13:08:20.385Z",
      "type": "TEXT",
      "body": {
         "text": "add 4 and 5"
      },
      "locale": "en",
      "data": {}
   },
   "context": {
      "device": {
         "type": "BROWSER",
         "capabilities": {
            "AUDIO": true,
            "HTML": true,
            "TEXT": true
         }
      },
      "session": {
         "id": "a26b810e-0b50-4528-8d8d-09a8207879da",
         "data": {},
         "new": true,
         "lastUpdatedAt": "2021-07-02T12:59:31.452Z"
      },
      "user": {
         "id": "0db85816-eab4-408d-a060-7c8098413b12",
         "data": {}
      }
   }
}

{
  start: 4,
  end: 4,
  len: 1,
  accuracy: 0.95,
  sourceText: '4',
  utteranceText: '4',
  entity: 'number',
  resolution: { value: 4, type: 'integer' }
}
{
  start: 10,
  end: 10,
  len: 1,
  accuracy: 0.95,
  sourceText: '5',
  utteranceText: '5',
  entity: 'number',
  resolution: { value: 5, type: 'integer' }
}
{
  start: 4,
  end: 4,
  len: 1,
  accuracy: 0.95,
  sourceText: '4',
  utteranceText: '4',
  entity: 'number',
  resolution: { value: 4, type: 'integer' }
}
{
  start: 10,
  end: 10,
  len: 1,
  accuracy: 0.95,
  sourceText: '5',
  utteranceText: '5',
  entity: 'number',
  resolution: { value: 5, type: 'integer' }
}
**** ON_REQUEST Inputs: {"number":{"value":"5"}}
**** Unhandled
this.getMappedIntentName: None
this.getRoute: {"intent":"None","path":"Unhandled","type":"INTENT"}

 <<<<< Response - 2021-07-02T13:08:20.517Z 
{
   "version": "3.4.0",
   "actions": [
      {
         "plain": "Darn! can't do that yet.",
         "ssml": "<speak>Darn! can't do that yet.</speak>",
         "type": "SPEECH"
      }
   ],
   "reprompts": [],
   "user": {
      "data": {}
   },

Notice how just before the **** ON_REQUEST line, Something in NLP.js or in Jovo's port for it proactively prints the entities extracted, twice. Which clearly shows that NLP.js does extract the inputs. However, on the **** ON_REQUEST you can see how only the last of the two inputs is included, and it does not bear the secondNumber key.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants