Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Autoshard] Auto-parallelism solver #96

Open
wants to merge 36 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
f449537
Add solver
chhzh123 May 19, 2023
340121c
Add test_autoshard
chhzh123 May 21, 2023
2dff6e2
Add dump_node
chhzh123 May 21, 2023
54884ac
Add fx_op_map & param_dump
chhzh123 May 21, 2023
d5cf94f
Add dump_z3_graph
chhzh123 May 21, 2023
9d1c176
Refactor & print
chhzh123 May 22, 2023
fcf1a2e
Add reshard schedule
chhzh123 May 22, 2023
bf542cf
Fix generate_output
chhzh123 May 22, 2023
35f1505
Remove useless functions
chhzh123 May 22, 2023
15b3358
Add test
chhzh123 May 22, 2023
5f8f2d2
Fix pylint
chhzh123 May 22, 2023
4df1956
Fix z3 graph
chhzh123 May 22, 2023
f0e5f90
Update ln and softmax rules
chhzh123 May 23, 2023
efc0a44
Add penalty for splitting
chhzh123 May 23, 2023
472a27c
Use total shape size to calculate communication cost
chhzh123 May 23, 2023
27067ab
Dump z3 graph with specs
chhzh123 May 23, 2023
9a326ef
Add attn test
chhzh123 May 23, 2023
0b1de58
Fix ViewOp
chhzh123 May 23, 2023
ef2c565
Fix test and output
chhzh123 May 23, 2023
8759140
Fix pylint
chhzh123 May 23, 2023
285e4a7
Update dependency
chhzh123 May 26, 2023
49945c6
Add test_bert
chhzh123 May 27, 2023
59e0b9b
Fix viewop
chhzh123 May 28, 2023
e8d126a
Add seq_par
chhzh123 May 28, 2023
6324524
Efficient seq_par
chhzh123 May 28, 2023
14483c1
Add comments
chhzh123 May 28, 2023
b61f536
Move file
chhzh123 May 29, 2023
ff024a1
Fix device
chhzh123 May 29, 2023
9a55da0
Fix TensorMetadata
chhzh123 May 29, 2023
83c6053
Support more ops
chhzh123 May 29, 2023
4bce113
Add gpt support
chhzh123 May 29, 2023
f933098
Support seq par for GPT
chhzh123 May 30, 2023
5a66148
Add TODO
chhzh123 May 30, 2023
5ccac8b
Add gpt_attn unit test
chhzh123 May 30, 2023
8d92d72
Rename
chhzh123 May 30, 2023
1f165a2
Update bs
chhzh123 May 30, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ci/install_test_pkgs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@
python3 -m pip install black==22.10.0
python3 -m pip install transformers==4.25.1 --no-deps
python3 -m pip install pylint==2.14.0 astroid==2.11.6 mock==4.0.3
python3 -m pip install z3-solver tabulate
2 changes: 2 additions & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,8 @@ def setup():
long_description_content_type="text/markdown",
setup_requires=[],
install_requires=[
"z3-solver",
"tabulate",
"packaging",
"psutil",
],
Expand Down
1 change: 1 addition & 0 deletions slapo/sharding/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@
scatter_forward_output,
reduce_forward_output,
)
from .solver import Solver
Loading