Simplify class `TokenCollector` to avoid two versions of maximal token logic#500

Merged

sungshik merged 4 commits intosemantic-tokenizer-fall2024from semantic-tokenizer-fall2024-refactor-collector

Nov 4, 2024

+48-83

Commits on Nov 1, 2024

Refactor class TokenCollector of the semantic tokenizer
sungshik
committed

Commits on Nov 4, 2024