explore different segmentation methods

[thatbudakguy]

it's an open question whether this is worthwhile or possible on old chinese, but there are certainly existing options outside of pure single-character segmentation. spaCy offers both jieba and pkuseg for its chinese model.

we could explore retraining some general-purpose models, or work on assembling a gold corpus of segmented (and tagged?) text - again, if such a thing is possible. pkuseg's training API looks simple to use, at least.

[thatbudakguy]

two much more attractive options for OC are UD-Kanbun for CC, which might be able to be minimally retrained for OC, and the dictionary method mentioned in this article (which also serves as a nice summary of segmentation methods).

sentence segmentation is I think even trickier, but worthwhile pursuing in any case.