New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Document segmenter options #183

Open

anne17 opened this issue Dec 13, 2023 · 1 comment

Assignees

Labels

documentation fixed-unreleased

Member

anne17 commented Dec 13, 2023

Right now the different options for the available segmenters can only be found in the code:

whitespace=nltk.WhitespaceTokenizer
linebreaks=LinebreakTokenizer
blanklines=nltk.BlanklineTokenizer
punkt_sentence=nltk.PunktSentenceTokenizer
punctuation=PunctuationTokenizer
better_word=BetterWordTokenizer
crf_tokenizer=CRFTokenizer
simple_word_punkt=nltk.WordPunctTokenizer
fsv_paragraph=FSVParagraphSplitter

They should be visible to the user somehow!

The text was updated successfully, but these errors were encountered:

anne17 added the documentation label

MartinHammarstedt self-assigned this

Member

MartinHammarstedt commented Feb 7, 2024

They are now listed when running sparv modules segment (and included in the schema).

MartinHammarstedt added the fixed-unreleased label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment