Adding NLP interfaces

[argenos]

• adding the required launch files in mas_domestic

[PatrickNa]

Does this issue include the integration of NLP into our speech recognition solution? If not then maybe we could open another issue for that?

[argenos]

Yeah, this issue is basically for that. On the todos for @ErickKramer there’s still: - making the appropriate changes in the nlu fork, specially limiting the dependencies installed on the robot and documenting the resources that need to be added and how to get them - subscribing to the right speech topic (we were using recognized_speech in the lab) - adding the required launch files in mas_domestic

[PatrickNa]

I think the topic will remain the same. In case you have not taken care of this already, we will need to check what else needs to be adjusted in order to get the NLP component to work correctly.

[argenos]

Is that the same for the questions and the commands?

[PatrickNa]

For questions it is taken care of... but when is a command a command in order to forward it to NLP? We have to think of how to categorize this correctly. Are we going to maintain all command possibilities in the ontology, e.g. "go to kitchen", "go to living room", "go to bathroom" etc. or do we just reduce the threshold for this category, so that the "go to" will already be enough to mark it as a command and instead of forwarding what we have in our ontology we forward the original content?!

[argenos]

I think we need to check for typical ways in which questions are asked, e.g. what, where, which, etc.
we can take a look at previous years for existing questions and come up with a strategy.
Then we could address this the other way around, anything that is not in the ontology and answerable as a question, we deal with as a command.

But I guess in the end what matters for now is not really how we separate the two, but what is the information flow? When it's detected as a command, is it published in recognized_speech and when it's detected as a question it's just dealt with internally? Or is it also published to some other topic?

[PatrickNa]

We made our thoughts about that... We are publishing everything that is recognized as speech to speech_recognizer (name might have been changed by now) and this is subscribed by the speech_matching node that analyzes it further and publishes its result in the following format ["category", "message", "distance"] to the speech_matcher topic.

[PatrickNa]

mdr_speech_recognition>mdr_speech_matching>[mdr_question_answering, NLP?]

[argenos]

I think that the processing should be done before reaching the answering or the nlp topic. Both of them only care about the string they have to process, not the category or the distance.

Architecturally, a component can make requests to the nlp module and the answer action server, and decide whether to send it to the right topic or make a request with the next best option. Which I think is part of the responsibility of the speech matcher, right?

[PatrickNa]

This was the original idea, but rather than making a publisher for each category we need only one. Otherwise we would have two +x (for every possible future category).

[ErickKramer]

Perhaps, we could try to catch whether the sentence interpreted by the speech recognition is a question or a command. This could be done by looking for the Wh words and/or the symbol "?". And depending on the result of this, the sentence could be either send to the NLU (mbot) or to the module getting the answer from the ontology.

If I am not mistaken, the GPSR command generator will always add the "?" at the end of a question.

PS: This is how socrob did a module to catch questions.

[PatrickNa]

We check whether it is a question or a command, but don't differentiate the receiver of this information. Rather the NLU has to check if "msg.type == command then do something otherwise ignore".

Regarding your thoughts about recognizing questions, keep in mind that there are questions without the W's and also you don't get punctuation from our speech recognition modules. However we have a solution for that. If you want I can show it to you on Friday.

[ErickKramer]

Yeah, I'd be glad to take a look into that. I also found the answer to your question about expanding the knowledge of the model. I'll show it to you on friday as well.

[argenos]

I disagree, NLU expects a command, the processing of whether is a command or not should be done beforehand, most likely in the speech matcher.

You are right about the questions, going over details during Friday’s meeting sounds good to me!

[argenos]

@ErickKramer I have updated the description of this issue, and opened b-it-bots/mbot_natural_language_processing#1 for you

[argenos]

Can this be closed?

[argenos]

Related PRs: #92 #85 #137

[PatrickNa]

We have RASA implemented, but the I'm not sure about @ErickKramer 's RnD work...

[ErickKramer]

Mbot is currently implemented on Lucy and the classifier for GPSR category 1 and 2 is there as well. The only thing missing is to put it on mas_domestic repository.

[argenos]

Isn't this what you added in #92? Or what is missing?

[ErickKramer]

No, that one is an interface that collects the output of mbot and execute an action (at the moment, just navigation).

It's missing everything that it's inside the Mbot folder (physically on Lucy) and to add my dataset to the dataset repository you mentioned last time.

[argenos]

Closed in favor of #155