Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New 661k HQ indexes, COBS installation, make test and CI #40

Merged
merged 44 commits into from
Jul 17, 2022
Merged

New 661k HQ indexes, COBS installation, make test and CI #40

merged 44 commits into from
Jul 17, 2022

Conversation

leoisl
Copy link
Collaborator

@leoisl leoisl commented Jul 17, 2022

Sorry again for a PR that does many things.

Main changes:

  1. New 661k HQ indexes. Closes [Important] COBS binary format changed; the 661k HQ indexes must be recomputed #32
  2. New rule to install COBS automatically. We install COBS through this conda env. On linux, all dependencies to build COBS are there, so it does not matter the specific configs of the user. On mac, we just don't specify gcc, but I've added a check on the pipeline that if you are on Mac and don't have gcc-11 installed, you have to brew install it: I don't think this is the ideal solution, but should work for all linux, and the big majority of Mac. Closes Looking for alternate ways to install COBS on Mac #35 but opens Improve COBS delivery with a conda or homebrew recipe #39
  3. Added make test command that mof-search the query against 3 batches only to quickly check if the full pipeline works;
  4. Added a CI with GH actions, and it shows it does work on both linux and mac.

Minor changes:

  1. Removed decomp_thr and download_thr from config.yaml, and added cobs_thr. All downloads are made with a fixed 1 thread, xz decompression and cobs are run with cobs_thr threads;
  2. Due to 1, yq is only used to get a single parameter from config.yaml. Replaced yq with a grep and awk to reduce dependencies;
  3. Fixed fix_query rule not working on Mac (sed on mac is different from sed on linux);
  4. Fixed small issue with the minimap2 conda env;
  5. Updated README.md: added a warning for Mac users that they need to brew install gcc-11, updated dependencies, added a walkthrough

@leoisl leoisl requested a review from karel-brinda July 17, 2022 10:07
.github/workflows/build.yaml Outdated Show resolved Hide resolved
.github/workflows/build.yaml Outdated Show resolved Hide resolved
.github/workflows/build.yaml Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
Snakefile Show resolved Hide resolved
Snakefile Show resolved Hide resolved
Snakefile Show resolved Hide resolved
Snakefile Show resolved Hide resolved
readme.md Show resolved Hide resolved
readme.md Show resolved Hide resolved
@leoisl
Copy link
Collaborator Author

leoisl commented Jul 17, 2022

@karel-brinda I did not understand the meaning of things like:

    resources:
        download_thr=1,

Now I understand it can be used for load balancing! Never saw this is snakemake and misunderstood the usage of it, I am rolling back some of the changes

@karel-brinda karel-brinda merged commit 1c2e651 into karel-brinda:master Jul 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants