Skip to content

Commit

Permalink
Revise and split requirements files (#340)
Browse files Browse the repository at this point in the history
Moves dependencies defined in the root `requirements.txt` to
`sharktank/` and splits out test only deps to `requirements-tests.txt`
file. Dependencies only used for development / in the CI are moved to
`requirements-dev.txt` and the root requirements file is now used to
pull in all the deps. Furthermore, some no longer used dependencies are
removed.
  • Loading branch information
marbre authored Oct 28, 2024
1 parent 98392d0 commit f925a5b
Show file tree
Hide file tree
Showing 7 changed files with 36 additions and 36 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/ci-sharktank.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,14 @@ on:
paths:
- '.github/workflows/ci-sharktank.yml'
- 'sharktank/**'
- '*requirements.txt'
- '*requirements*.txt'
push:
branches:
- main
paths:
- '.github/workflows/ci-sharktank.yml'
- 'sharktank/**'
- '*requirements.txt'
- '*requirements*.txt'

concurrency:
# A PR number if a pull request and otherwise the commit hash. This cancels
Expand Down Expand Up @@ -52,7 +52,7 @@ jobs:
id: cache-pip
with:
path: ${{ env.PIP_CACHE_DIR }}
key: pip-${{ steps.setup_python.outputs.python-version }}-${{ hashFiles('*requirements.txt') }}
key: pip-${{ steps.setup_python.outputs.python-version }}-${{ hashFiles('*requirements*.txt','sharktank/requirements*.txt') }}

- name: Install pip deps
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci_eval.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ jobs:
id: cache-pip
with:
path: ${{ env.PIP_CACHE_DIR }}
key: pip-${{ steps.setup_python.outputs.python-version }}-${{ hashFiles('*requirements.txt') }}
key: pip-${{ steps.setup_python.outputs.python-version }}-${{ hashFiles('*requirements*.txt','sharktank/requirements*.txt') }}

- name: Install sharktank deps
run: |
Expand Down
10 changes: 10 additions & 0 deletions requirements-dev.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Used for managing pre-commit flows.
pre-commit

# Type checking
mypy==1.8.0
types-requests==2.31.0.20240125

# Testing
pytest==8.0.0
pytest-xdist==3.5.0
34 changes: 4 additions & 30 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,30 +1,4 @@
# Runtime deps.
gguf==0.6.0
numpy==1.26.3
onnx==1.15.0

# Model deps.
huggingface-hub==0.22.2
transformers==4.40.0
sentencepiece==0.2.0

# It is expected that you have installed a PyTorch version/variant specific
# to your needs, so we only include a minimum version spec.
# TODO: Use a versioned release once 2.3.0 drops.
torch>=2.3.0.dev1

# Used for managing pre-commit flows.
pre-commit

# Type checking
mypy==1.8.0
types-requests==2.31.0.20240125

# Testing
parameterized
pytest==8.0.0
pytest-xdist==3.5.0

# Serving deps.
fastapi==0.112.2
uvicorn==0.30.6
-r sharktank/requirements.txt
-r sharktank/requirements-tests.txt
-r shortfin/requirements-tests.txt
-r requirements-dev.txt
2 changes: 2 additions & 0 deletions sharktank/requirements-tests.txt
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
datasets==3.0.0
parameterized
pytest==8.0.0
17 changes: 16 additions & 1 deletion sharktank/requirements.txt
Original file line number Diff line number Diff line change
@@ -1 +1,16 @@
gguf
# Runtime deps.
gguf==0.6.0
numpy==1.26.3

# Model deps.
huggingface-hub==0.22.2
transformers==4.40.0
datasets

# It is expected that you have installed a PyTorch version/variant specific
# to your needs, so we only include a minimum version spec.
torch>=2.3.0

# Serving deps.
fastapi==0.112.2
uvicorn==0.30.6
1 change: 0 additions & 1 deletion sharktank/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,6 @@ def initialize_options(self):
extras_require={
"testing": [
f"pytest{get_version_spec('pytest')}",
f"pytest-xdist{get_version_spec('pytest-xdist')}",
],
},
cmdclass={"build": BuildCommand},
Expand Down

0 comments on commit f925a5b

Please sign in to comment.