Skip to content

Commit

Permalink
Update README.md with install from source
Browse files Browse the repository at this point in the history
  • Loading branch information
ArthurZucker authored Nov 15, 2024
1 parent cc5fb01 commit f0c48bd
Showing 1 changed file with 13 additions and 0 deletions.
13 changes: 13 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,19 @@ We provide bindings to the following languages (more to come!):
- [Python](https://github.com/huggingface/tokenizers/tree/main/bindings/python)
- [Node.js](https://github.com/huggingface/tokenizers/tree/main/bindings/node)
- [Ruby](https://github.com/ankane/tokenizers-ruby) (Contributed by @ankane, external repo)

## Installation

You can install from source using:
```bash
pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python
```

our install the released versions with

```bash
pip install tokenizers
```

## Quick example using Python:

Expand Down

0 comments on commit f0c48bd

Please sign in to comment.