explain morpheme detail

[mh-northlander]

fix #102.

return surface, dictionaryForm, normalizedForm, readingForm and partOfSpeech when explain: true.

• We may change key names (e.g. reading instead of readingForm, following analysis-kuromoji)
• We may drop morpheme grouping.
• We may drop instance information (from morphemeConsumerAttibute, see below example)
  • Let's keep it to see which tokenizer/filter writes the term attribute

implementation note

Key-value pairs set in reflectWith method of each *AttributeImpl classes are used to construct explain contents.
Those value should have primitive type/standard collection type or implement ToXContent interface.

Document for reflectWith says:

This method is for introspection of attributes, it should simply add the key/values this attribute holds to the given {@link AttributeReflector}

So here I choosed to set "morpheme": morpheme in reflectWith and implement a wrapper class for morpheme with ToXContent.

example

curl -X POST "localhost:9200/_analyze?pretty" \
 -H 'Content-Type: application/json' \
 -d @- <<EOF
{
    "text":"すだち",
    "tokenizer": "sudachi_tokenizer",
    "explain":true
}
EOF

{
  "detail" : {
    "custom_analyzer" : true,
    "charfilters" : [ ],
    "tokenizer" : {
      "name" : "sudachi_tokenizer",
      "tokens" : [
        {
          "token" : "すだち",
          "start_offset" : 0,
          "end_offset" : 3,
          "type" : "word",
          "position" : 0,
          "bytes" : "[e3 81 99 e3 81 a0 e3 81 a1]",
          "instance" : "com.worksap.nlp.lucene.sudachi.ja.SudachiTokenizer",
          "morpheme" : {
            "surface" : "すだち",
            "dictionaryForm" : "すだち",
            "normalizedForm" : "酢橘",
            "readingForm" : "スダチ",
            "partOfSpeech" : [
              "名詞",
              "普通名詞",
              "一般",
              "*",
              "*",
              "*"
            ]
          },
          "positionLength" : 1
        }
      ]
    },
    "tokenfilters" : [ ]
  }
}

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud