Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Major data model refactor #177

Merged
merged 471 commits into from
Nov 12, 2024
Merged

Major data model refactor #177

merged 471 commits into from
Nov 12, 2024

Conversation

pavlovicmilena
Copy link
Member

Major updates:

  • Data model refactored and simplified to use bionumpy and be internally AIRR-compatible as well
  • Improved usability: reports and docs
  • Two new workflows [to be developed further]: clustering and generative models

pavlovicmilena and others added 30 commits October 29, 2023 20:54
@pavlovicmilena
Merge remote-tracking branch 'origin/master' into nar
c05f22a 
# Conflicts:
#	requirements.txt
@pavlovicmilena
add device to vae
feed634
@pavlovicmilena
add vae summary report
0afcf7a
@pavlovicmilena
add pypi publish dev setup
0591198
@pavlovicmilena
update dev version
3e49981
@pavlovicmilena
remove condition from pypi setup
1aafbf7
@pavlovicmilena
update readme to reflect that this is dev version
2bfb2db
@pavlovicmilena
add conda package workflow
064b851
@pavlovicmilena
update PyPI action
99f58bd
@pavlovicmilena
revert to trusted publisher for PyPI
f604f25
@pavlovicmilena
add install conda-build
56300bf
@pavlovicmilena
fix github action
a261485
@pavlovicmilena
add publish conda as separate action
06eea30
@pavlovicmilena
fix paths in github action
3badbe2
@pavlovicmilena
update conda in github action
679b483
@pavlovicmilena
update conda in github action
d1e87c1
@pavlovicmilena
tmp remove unused action
080850f
@pavlovicmilena
remove uses key in github action
8b7b8fe
@pavlovicmilena
update publish-dev-to-conda.yml
7205d47
@pavlovicmilena
update publish-dev-to-conda.yml
a90fb96
@pavlovicmilena
update publish-dev-to-conda.yml
08548a0
@pavlovicmilena
update publish-dev-to-conda.yml
64ad364
@pavlovicmilena
update publish-dev-to-conda.yml
58fb89e
@pavlovicmilena
update publish-dev-to-conda.yml
2be7254
@pavlovicmilena
update publish-dev-to-conda.yml
28d5d7f
@LonnekeScheffer
Updated AminoAcidFrequencyDistribution report: plot in addition 'freq…
3e2f234 
…uency change'
@pavlovicmilena
keep only pypi-publish action
6610263
@pavlovicmilena
integrate ligo with immuneML
ebd3ee3
@pavlovicmilena
update README.md
0a24ad9
@pavlovicmilena
add IMGT position fix
a8694d7
pavlovicmilena and others added 29 commits November 4, 2024 11:31
@pavlovicmilena
minor galaxy train gen model updates
e5b5e7a
@pavlovicmilena
limit scipy version because it doesn't work with gensim
ffe9619
@pavlovicmilena
fix css for index.html
35c3afa
@LonnekeScheffer
add to airr import test: test for label import sequence/receptor dataset
c672794
@LonnekeScheffer
add to airr import test: test for label import sequence/receptor dataset
74e2c9a
@LonnekeScheffer
remove metadata_column_mapping from DatasetImportParams.py
36bafe2
@LonnekeScheffer
galaxy test print statement
8646733
@LonnekeScheffer
do not add dataset_ prefix to .yaml filename (messes up galaxy)
1b04714
@LonnekeScheffer
fixes for galaxy datasets:
7e422cc 
- removed obsolete build_Dataset_yaml to avoid confusion
- datasets are always called datasetname.yaml (avoid 'dataset_dataset.yaml' since it's confusing)
- in build_dataset_overview_yaml: remove option to have datasets with multiple names, it is always called 'dataset'
- central dataset discovery in Galaxy Util
- renamed ambiguous name build_yaml_from_arguments to build_train_ml_model_yaml.py
- training split size for clustering instruction
@LonnekeScheffer
fix for dataset simulation sequences: dataset name, added test
bcfb482
@LonnekeScheffer
- Added postprocessing for galaxy datasets, removing file paths
14f0c0c 
- Made some tests faster by using smaller dummy datasets
@LonnekeScheffer
- standardized writing of dataset.yaml file
68300ae 
- discover dataset type from dataset.yaml file if provided
@LonnekeScheffer
version bump for galaxy testing
30a4e5b
@pavlovicmilena
fix dataset io for galaxy
d082c5c
@pavlovicmilena
update version
e95f63d
@LonnekeScheffer
label update:
b96fbe1 
- add explicit parameter 'label_columns' to import params -> imports only those labels if specified (avoids clutter on galaxy)
- update importer docs
@LonnekeScheffer
Merge remote-tracking branch 'origin/nar' into nar
c1dc4ec
@LonnekeScheffer
attempt start fixing data import for generative models
432aa06
@LonnekeScheffer
minor bugfix SequenceCountDistribution.py
8124868
@LonnekeScheffer
minor fix: improved error message for failed reports
c57a03c
@LonnekeScheffer
minor: add random dup. counts for random generated data
0e0abc6
@LonnekeScheffer
minor: improved display SequenceCountDistribution (x axis sorted)
8bf165b
@LonnekeScheffer
improved filter_illegal_receptors, instead of removing any without co…
297480b 
…unt = 2, does the following:

- if count >2, the top chain pair is selected based on duplicate_count. if duplicate_count is not available, they are selected randomly
- if count <2, receptor is removed. this removal now also includes receptors with 2 identical chains (e.g.,TRB+TRB)
@LonnekeScheffer
improved receptor filter for receptors with >2 chains (first check ba…
7a7a6ab 
…sed on umi count, then duplicate count)
@LonnekeScheffer
added missing default param region_type for filters
84765a4
@LonnekeScheffer
pass number_of_processes to preprocessing in MLProcess & Expl analysis
dbc2dfa
@pavlovicmilena
update error message when file not found on import
42aea52
@pavlovicmilena
Merge remote-tracking branch 'origin/nar' into nar
5782589
@pavlovicmilena
update docs; fix vdjdb test
600da81
@pavlovicmilena pavlovicmilena merged commit 7bbf443 into master Nov 12, 2024
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@pavlovicmilena @LonnekeScheffer @ivargr @mmamica @knutdrand