v0.6.0

Minor version release signifies a breaking change in 0.x

What's Changed

• Add support for chatgpt-4o-latest by @Congyuwang in #85
• Refactor internals to make future updates to tiktoken easier to merge
• Do not expose tiktoken internal modules and functions
• Update dependencies

New Contributors

• @Congyuwang made their first contribution in #85

Full Changelog: v0.5.9...v0.6.0

v0.5.9

What's Changed

• Add gpt-4-0125 and gpt-3.5-turbo-1106 context sizes by @mikaelsouza in #59
• Add gpt-3.5-turbo-0125 by @mikaelsouza in #60
• Update README.md by @imalsogreg in #61
• Add support: o200k_base tokenizer. by @CaffeeLake in #63

New Contributors

• @mikaelsouza made their first contribution in #59
• @imalsogreg made their first contribution in #61
• @CaffeeLake made their first contribution in #63

Full Changelog: v0.5.8...v0.5.9

v0.5.8

What's Changed

• Add tokenizer prefixes for fine-tuned models by @jbgriesner in #53

New Contributors

• @jbgriesner made their first contribution in #53

Full Changelog: v0.5.7...v0.5.8

v0.5.7

What's Changed

• Add gpt-4-1106-preview model by @JosephTLyons in #49

New Contributors

• @JosephTLyons made their first contribution in #49

Full Changelog: v0.5.6...v0.5.7

v0.5.6

What's Changed

• Expose _decode_native methods by @zurawiki in #43

Full Changelog: v0.5.5...v0.5.6

v0.5.5

What's Changed

• return correct context length for text-embedding-ada-002 by @ursachec in #41

New Contributors

• @ursachec made their first contribution in #41

Full Changelog: v0.5.4...v0.5.5

v0.5.4

Restore minimum supported Rust version (MSRV) to 1.61.0.

• No other functional changes

v0.5.3

Update dependencies

Full Changelog: v0.5.1...v0.5.3

v0.5.1

What's Changed

• Update tests by @kojix2 in #28
• Derive Debug for CoreBPE struct by @tristan-jl in #32
• Update to async-openai 0.13.0 by @Dreaming-Codes in #31

New Contributors

• @kojix2 made their first contribution in #28
• @tristan-jl made their first contribution in #32
• @Dreaming-Codes made their first contribution in #31

Full Changelog: v0.5.0...v0.5.1

v0.5.0

What's New

• Added support for OpenAI function calling. See https://platform.openai.com/docs/guides/gpt/function-calling for more details.

What's Changed

• Update async-openai to 0.12.0 by @DaviRain-Su in #26

New Contributors

• @DaviRain-Su made their first contribution in #26

Full Changelog: v0.4.5...v0.5.0