Some questions of LM score

[zh794390558]

https://github.com/coqui-ai/STT/blob/main/native_client/ctcdecode/ctc_beam_search_decoder.cpp#L181

bool bos = ngram.size() < ext_scorer_->get_max_order();
score = ( ext_scorer_->get_log_cond_prob(ngram, bos) + hot_boost ) * ext_scorer_->alpha;

1. ext_scorer_->get_log_cond_prob(ngram, bos) , eos is not set and also has not default value.
2. lm score only the ngram word, what's the meaning when not score the whole prefix?
3. why when ngram.size() < ngram order, we need the bos.

float hot_boost = 0.0;
              if (!hot_words_.empty()) {
                std::unordered_map<std::string, float>::iterator iter;
                // increase prob of prefix for every word
                // that matches a word in the hot-words list
                for (std::string word : ngram) {
                  iter = hot_words_.find(word);
                  if ( iter != hot_words_.end() ) {
                    // increase the log_cond_prob(prefix|LM)
                    hot_boost += iter->second;
                  }
                }
              }

1. hot words only work for English, but chinese like 中国人 20.0， who does this work?
2. what the performance improve when using byte is all you need in chinese.

lm rescore in decode not correct:
#1810

should be below, and no hot word boost support.

  // score the last word of each prefix that doesn't end with space
  if (ext_scorer_) {
    for (size_t i = 0; i < beam_size_ && i < prefixes_copy.size(); ++i) {
      PathTrie* prefix = prefixes_copy[i];
      PathTrie* prefix_boundary = ext_scorer_->is_utf8_mode() ? prefix : prefix->parent;
      if (prefix_boundary && ext_scorer_->is_scoring_boundary(prefix_boundary, prefix->character)) {
        float score = 0.0;
        std::vector<std::string> ngram = ext_scorer_->make_ngram(prefix_boundary);
        bool bos = ngram.size() < ext_scorer_->get_max_order();
        score = ext_scorer_->get_log_cond_prob(ngram, bos) * ext_scorer_->alpha;
        score += ext_scorer_->beta;
        scores[prefix] += score;
      }
    }
  }