ChatGPT for punctuation

[dgoryeo]

Would it be an idea to use ChatGPT to perform a punctuation on results before regrouping by punctuation? Specially for languages that punctuation can be quite tricky Chinese, Japanese, Korean, Arabic?

[jianfch]

Can you elaborate what you mean by "punctuation on results"?

[dgoryeo]

Here is an example -- I hope this makes sense:

https://user-images.githubusercontent.com/36184621/275307911-13919eb1-44d7-450e-9637-2995d8786ac3.png

[dgoryeo]

By results I meant the text content of the jason file.

[jianfch]

For this example above, something like this will work:

def update_punctuations(result, new_text: str, verbose=False):
    all_words = result.all_words()
    for i, word in enumerate(all_words):
        space, text = (' ', word.word[1:]) if word.word.startswith(' ') else ('', word.word)
        try:
            i1 = new_text.index(text)
        except ValueError:
            raise ValueError(f'"{text}" not found in "{new_text[:len(text)*3]}"')
        j1 = i1 + len(text)
        new_word = space + new_text[i1:j1]
        new_text = new_text[j1:]
        j1 = new_text.index(all_words[i + 1].word[0]) if i + 1 != len(all_words) else len(new_text)
        new_word += new_text[:j1]
        new_text = new_text[j1:]
        if word.word != new_word:
            if verbose:
                print(f'{word.word} -> {new_word}')
            word.word = new_word

This will only append new punctuations. Keep mind that results might be inconsistent or even fail because GPT hallucinates which might remove words, add new words, or return something unexpected (e.g. a word in a different unicode representation).

[dgoryeo]

Thanks @jianfch! Meanwhile here is another helpful response I received for specific Japanese language punctuation, it looks very promising: punctuation_tool.