[Test] add test to verify that BufferedTokenizerExt doesn't apply enc…

[andsel]

…oding twice

Release notes

What does this PR do?

Why is it important/What is the impact to the user?

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files (and/or docker env variables)
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: f718ef4

Failed CI Steps

• 🛂 License check
• 🛂 License check
• 🛂 License check
• 🛂 License check
• :java: Java unit tests
• :java: Java unit tests
• :java: Java unit tests
• :java: Java unit tests
• 🥼 Integration Tests / part 1

History

• 💔 Build #1864 failed c9581ce

cc @andsel

[mergify]

This pull request does not have a backport label. Could you fix it @andsel? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• backport-8.x is the label to automatically backport to the 8.x branch.

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label.

[mergify]

This pull request is now in conflicts. Could you fix it @andsel? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b andsel/test_buftok_respect_charset_encoding upstream/andsel/test_buftok_respect_charset_encoding
git merge upstream/main
git push upstream andsel/test_buftok_respect_charset_encoding

[andsel]

Code added in #16968