From a538d30a59a8319bafbaeb450205f34008e89717 Mon Sep 17 00:00:00 2001 From: deliaBlue <103108590+deliaBlue@users.noreply.github.com> Date: Tue, 19 Sep 2023 12:34:44 +0200 Subject: [PATCH 01/23] build: set correct channel in `environment.root.yml` The installation fails when the channel is `Bioconda` and the user has the `priority_channel` set to `strict`. --- environment.root.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/environment.root.yml b/environment.root.yml index e5c33b3..b2e3ea3 100644 --- a/environment.root.yml +++ b/environment.root.yml @@ -1,5 +1,5 @@ name: mirflowz channels: - - bioconda + - conda-forge dependencies: - singularity>=3.5.2 From e1906c8adf1571679f59af8ea867028962c7a170 Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Fri, 22 Sep 2023 14:18:20 +0200 Subject: [PATCH 02/23] docs: update installation steps --- README.md | 72 +++++++++++++++++++++++++------------------------------ 1 file changed, 32 insertions(+), 40 deletions(-) diff --git a/README.md b/README.md index b0282e8..69b115c 100644 --- a/README.md +++ b/README.md @@ -5,8 +5,8 @@ _MIRFLOWZ_ is a [Snakemake][snakemake] workflow for mapping miRNAs and isomiRs. ## Table of Contents 1. [Installation](#installation) + - [Requirements](#requirements) - [Cloning the repository](#cloning-the-repository) - - [Dependencies](#dependencies) - [Setting up the virtual environment](#setting-up-the-virtual-environment) - [Testing your installation](#testing-your-installation) 2. [Usage](#usage) @@ -23,17 +23,7 @@ _MIRFLOWZ_ is a [Snakemake][snakemake] workflow for mapping miRNAs and isomiRs. The workflow lives inside this repository and will be available for you to run after following the installation instructions layed out in this section. -### Cloning the repository - -Traverse to the desired path on your file system, then clone the repository and -change into it with: - -```bash -git clone https://github.com/zavolanlab/mirflowz.git -cd mirflowz -``` - -### Dependencies +### Requirements For improved reproducibility and reusability of the workflow, as well as an easy means to run it on a high performance computing (HPC) cluster managed, @@ -43,48 +33,50 @@ environments). As a consequence, running this workflow has only a few individual dependencies. These are managed by the package manager Conda, which needs to be installed on your system before proceeding. -If you do not already have Conda installed globally on your system, -we recommend that you install [Miniconda][miniconda-installation]. For faster -creation of the environment (and Conda environments in general), you can also -install [Mamba][mamba] on top of Conda. In that case, replace `conda` with -`mamba` in the commands below (particularly in `conda env create`). +The installation requires the following: -### Setting up the virtual environment +- Linux (tested with Ubuntu `20.04` and `22.04`; macOS has not been tested yet) +- [Conda][conda] (tested with `conda 23.1.0`) +- [Mamba][mamba] (tested with `mamba 1.4.1`) -Create and activate the environment with necessary dependencies with Conda: +> Other versions, especially older ones, are not guaranteed to work. -```bash -conda env create -f environment.yml -conda activate mirflowz -``` +### Cloning the repository -If you plan to run _MIRFLOWZ_ via Conda, we recommend to use the following -command for a faster environment creation specially if it you will run it on a -HPC cluster. +Traverse to the desired path on your file system, then clone the repository and +change into it with: ```bash -conda config --set channel_priority strict +git clone https://github.com/zavolanlab/mirflowz.git +# or git clone git@github.com:zavolanlab/mirflowz +cd mirflowz ``` -If you plan to run _MIRFLOWZ_ via Singularity and do not already -have it installed globally on your system, you must further update the Conda -environment using the `environment.root.yml` with the command below. -Mind that you must have the environment activated to update it. + +### Setting up the virtual environment + +You now need to create and activate the environment with necessary +dependencies. For that purpose, there exist four different environment files. +Use the decision matrix to pick the most suitable one for you: + +| I have root privileges on the machine/ I want to run _MIRFLOWZ_ via Singularity | I want to run pre-packaged tests | Environment file to use              | +|:---:|:---:| --- | +| | | `install/environment.yml` | +| :check_mark: | | `install/environment.root.yml` | +| | :check_mark: | `install/environment.dev.yml` | +| :check_mark: | :check_mark: | `install/environment.dev.root.yml` | + +To set up the environment, execute the call below, but do not forget to replace +the placeholder `ENVIRONMENT` with the appropiate file from the table above: ```bash -conda env update -f environment.root.yml +mamba env create -f ENVIRONMENT ``` -> Note that you will need to have root permissions on your system to be able -> to install Singularity. If you want to run _MIRFLOWZ_ on an HPC cluster -> (recommended in almost all cases), ask your systems administrator about -> Singularity. - -If you would like to contribute to _MIRFLOWZ_ development, you may find it -useful to further update your environment with the development dependencies: +Finally, activate the Conda environment with: ```bash -conda env update -f environment.dev.yml +conda activate mirflowz ``` ### Testing your installation From c3e867e643faae9359725b99eddda2fa38eb0c38 Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Fri, 22 Sep 2023 14:18:43 +0200 Subject: [PATCH 03/23] build: create dev environmwnt with singularity --- install/environment.dev.root.yml | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 install/environment.dev.root.yml diff --git a/install/environment.dev.root.yml b/install/environment.dev.root.yml new file mode 100644 index 0000000..cd9564e --- /dev/null +++ b/install/environment.dev.root.yml @@ -0,0 +1,19 @@ +name: mirflowz +channels: + - conda-forge + - bioconda +dependencies: + - coverage >=7.2 + - flake8 >=3.8.4 + - flake8-docstrings >=1.6.0 + - gffutils >=0.11.1 + - graphviz=7.1.0 + - mypy >=1.5 + - pylint >=2.7.1 + - pysam >=0.21.0 + - pytest >=7.1.2 + - pytest-cov + - python=3.9.16 + - singularity>=3.5.2 + - snakefmt + - snakemake=7.24.0 From d4ae82fb00b5650b2bf134f9d3048c7a99815733 Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Fri, 22 Sep 2023 15:25:01 +0200 Subject: [PATCH 04/23] ci: update testest --- .github/workflows/CI.yml | 46 +++++++++++++++++++++++++++------------- 1 file changed, 31 insertions(+), 15 deletions(-) diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml index e099624..ee14632 100644 --- a/.github/workflows/CI.yml +++ b/.github/workflows/CI.yml @@ -10,7 +10,7 @@ on: jobs: - snakemake-test: + snakemake-test-via-singularity: runs-on: ubuntu-latest defaults: run: @@ -26,14 +26,8 @@ jobs: with: mamba-version: "*" activate-environment: mirflowz - environment-file: environment.yml + environment-file: install/environment.root.yml auto-activate-base: false - - - name: Update mirflowz env with root packages - run: mamba env update -n mirflowz -f environment.root.yml - - - name: Update mirflowz env with dev packages - run: mamba env update -n mirflowz -f environment.dev.yml - name: Run test for rule graph run: bash test/test_rule_graph.sh @@ -45,9 +39,37 @@ jobs: - name: Run local test with Singularity run: bash test/test_workflow_local_with_singularity.sh + + snakemake-test-via-conda: + runs-on: ubuntu-latest + defaults: + run: + shell: bash -l {0} + + steps: + + - name: Checkout Repository + uses: actions/checkout@v3 + + - name: Setup environment + uses: conda-incubator/setup-miniconda@v2 + with: + mamba-version: "*" + activate-environment: mirflowz + environment-file: install/environment.yml + auto-activate-base: false + + - name: Run test for rule graph + run: bash test/test_rule_graph.sh + + - name: Check workflow descriptor files for lints + working-directory: ./test + run: snakemake --snakefile="../workflow/Snakefile" --configfile="config.yaml" --lint + - name: Run local test with Conda run: bash test/test_workflow_local_with_conda.sh + unit-testing: runs-on: ubuntu-latest defaults: @@ -64,14 +86,8 @@ jobs: with: mamba-version: "*" activate-environment: mirflowz - environment-file: environment.yml + environment-file: install/environment.dev.yml auto-activate-base: false - - - name: Update mirflowz env with root packages - run: mamba env update -n mirflowz -f environment.root.yml - - - name: Update mirflowz env with dev packages - run: mamba env update -n mirflowz -f environment.dev.yml - name: run unit tests working-directory: ./scripts/tests From 3bd5f33247b045baf1ffc8c243bac149290c44fb Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Fri, 22 Sep 2023 15:26:26 +0200 Subject: [PATCH 05/23] docs: move environemtn files --- install/environment.dev.yml | 18 ++++++++++++++++++ install/environment.root.yml | 9 +++++++++ install/environment.yml | 8 ++++++++ 3 files changed, 35 insertions(+) create mode 100644 install/environment.dev.yml create mode 100644 install/environment.root.yml create mode 100644 install/environment.yml diff --git a/install/environment.dev.yml b/install/environment.dev.yml new file mode 100644 index 0000000..c498294 --- /dev/null +++ b/install/environment.dev.yml @@ -0,0 +1,18 @@ +name: mirflowz +channels: + - conda-forge + - bioconda +dependencies: + - coverage >=7.2 + - flake8 >=3.8.4 + - flake8-docstrings >=1.6.0 + - gffutils >=0.11.1 + - graphviz=7.1.0 + - mypy >=1.5 + - pylint >=2.7.1 + - pysam >=0.21.0 + - pytest >=7.1.2 + - pytest-cov + - python=3.9.16 + - snakefmt + - snakemake=7.24.0 diff --git a/install/environment.root.yml b/install/environment.root.yml new file mode 100644 index 0000000..a1ab01e --- /dev/null +++ b/install/environment.root.yml @@ -0,0 +1,9 @@ +name: mirflowz +channels: + - conda-forge + - bioconda +dependencies: + - graphviz=7.1.0 + - python=3.9.16 + - snakemake=7.24.0 + - singularity>=3.5.2 diff --git a/install/environment.yml b/install/environment.yml new file mode 100644 index 0000000..e833f1a --- /dev/null +++ b/install/environment.yml @@ -0,0 +1,8 @@ +name: mirflowz +channels: + - conda-forge + - bioconda +dependencies: + - graphviz=7.1.0 + - python=3.9.16 + - snakemake=7.24.0 From 2bca79910c08e53d2df7d79517750aa51527485b Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Fri, 22 Sep 2023 16:43:32 +0200 Subject: [PATCH 06/23] ci: add environment display --- .github/workflows/CI.yml | 17 ++++++++++++++++- environment.dev.yml | 11 ----------- environment.root.yml | 5 ----- environment.yml | 8 -------- 4 files changed, 16 insertions(+), 25 deletions(-) delete mode 100644 environment.dev.yml delete mode 100644 environment.root.yml delete mode 100644 environment.yml diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml index ee14632..1ec4920 100644 --- a/.github/workflows/CI.yml +++ b/.github/workflows/CI.yml @@ -29,6 +29,11 @@ jobs: environment-file: install/environment.root.yml auto-activate-base: false + - name: Display environment info + run: | + conda info -a + conda list + - name: Run test for rule graph run: bash test/test_rule_graph.sh @@ -58,7 +63,12 @@ jobs: activate-environment: mirflowz environment-file: install/environment.yml auto-activate-base: false - + + - name: Display environment info + run: | + conda info -a + conda list + - name: Run test for rule graph run: bash test/test_rule_graph.sh @@ -89,6 +99,11 @@ jobs: environment-file: install/environment.dev.yml auto-activate-base: false + - name: Display environment info + run: | + conda info -a + conda list + - name: run unit tests working-directory: ./scripts/tests run: pytest --cov=scripts --cov-branch --cov-report=term-missing diff --git a/environment.dev.yml b/environment.dev.yml deleted file mode 100644 index 99b4c0d..0000000 --- a/environment.dev.yml +++ /dev/null @@ -1,11 +0,0 @@ -name: mirflowz -channels: - - conda-forge - - bioconda -dependencies: - - coverage>=7.2 - - gffutils>=0.11.1 - - pysam>=0.21.0 - - pytest>=7.1.2 - - pytest-cov - - snakefmt diff --git a/environment.root.yml b/environment.root.yml deleted file mode 100644 index b2e3ea3..0000000 --- a/environment.root.yml +++ /dev/null @@ -1,5 +0,0 @@ -name: mirflowz -channels: - - conda-forge -dependencies: - - singularity>=3.5.2 diff --git a/environment.yml b/environment.yml deleted file mode 100644 index e833f1a..0000000 --- a/environment.yml +++ /dev/null @@ -1,8 +0,0 @@ -name: mirflowz -channels: - - conda-forge - - bioconda -dependencies: - - graphviz=7.1.0 - - python=3.9.16 - - snakemake=7.24.0 From 70f1f0a94fe7e71ad3c90f8ee71216b588806e23 Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Fri, 22 Sep 2023 22:24:41 +0200 Subject: [PATCH 07/23] build: swap channels --- workflow/envs/samtools.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/workflow/envs/samtools.yaml b/workflow/envs/samtools.yaml index 2ed2b06..49742dd 100644 --- a/workflow/envs/samtools.yaml +++ b/workflow/envs/samtools.yaml @@ -1,7 +1,7 @@ --- channels: - - conda-forge - bioconda + - conda-forge dependencies: - samtools=1.16.1 -... \ No newline at end of file +... From 9aca07665cd659a82c2002269d7fef49b6507c46 Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Fri, 22 Sep 2023 22:28:22 +0200 Subject: [PATCH 08/23] build: swap channels --- workflow/envs/samtools.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/workflow/envs/samtools.yaml b/workflow/envs/samtools.yaml index 49742dd..153fe83 100644 --- a/workflow/envs/samtools.yaml +++ b/workflow/envs/samtools.yaml @@ -1,7 +1,7 @@ --- channels: - - bioconda - conda-forge + - bioconda dependencies: - samtools=1.16.1 ... From a66e46c1a446bb733335602c6ce8a4ba0b044c7b Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Wed, 27 Sep 2023 14:11:31 +0200 Subject: [PATCH 09/23] build: pin versions --- install/environment.dev.root.yml | 6 +++--- install/environment.dev.yml | 6 +++--- install/environment.root.yml | 6 +++--- install/environment.yml | 6 +++--- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/install/environment.dev.root.yml b/install/environment.dev.root.yml index cd9564e..4716086 100644 --- a/install/environment.dev.root.yml +++ b/install/environment.dev.root.yml @@ -7,13 +7,13 @@ dependencies: - flake8 >=3.8.4 - flake8-docstrings >=1.6.0 - gffutils >=0.11.1 - - graphviz=7.1.0 + - graphviz==7.1.0 - mypy >=1.5 - pylint >=2.7.1 - pysam >=0.21.0 - pytest >=7.1.2 - pytest-cov - - python=3.9.16 + - python==3.9.16 - singularity>=3.5.2 - snakefmt - - snakemake=7.24.0 + - snakemake==7.24.0 diff --git a/install/environment.dev.yml b/install/environment.dev.yml index c498294..75ebb0d 100644 --- a/install/environment.dev.yml +++ b/install/environment.dev.yml @@ -7,12 +7,12 @@ dependencies: - flake8 >=3.8.4 - flake8-docstrings >=1.6.0 - gffutils >=0.11.1 - - graphviz=7.1.0 + - graphviz==7.1.0 - mypy >=1.5 - pylint >=2.7.1 - pysam >=0.21.0 - pytest >=7.1.2 - pytest-cov - - python=3.9.16 + - python==3.9.16 - snakefmt - - snakemake=7.24.0 + - snakemake==7.24.0 diff --git a/install/environment.root.yml b/install/environment.root.yml index a1ab01e..de77527 100644 --- a/install/environment.root.yml +++ b/install/environment.root.yml @@ -3,7 +3,7 @@ channels: - conda-forge - bioconda dependencies: - - graphviz=7.1.0 - - python=3.9.16 - - snakemake=7.24.0 + - graphviz==7.1.0 + - python==3.9.16 + - snakemake==7.24.0 - singularity>=3.5.2 diff --git a/install/environment.yml b/install/environment.yml index e833f1a..9dcf985 100644 --- a/install/environment.yml +++ b/install/environment.yml @@ -3,6 +3,6 @@ channels: - conda-forge - bioconda dependencies: - - graphviz=7.1.0 - - python=3.9.16 - - snakemake=7.24.0 + - graphviz==7.1.0 + - python==3.9.16 + - snakemake==7.24.0 From 5df800547653752051b4111fc4d039af38d657cc Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Tue, 3 Oct 2023 15:37:24 +0200 Subject: [PATCH 10/23] ci: split conda and singularity snakemake tests --- .github/workflows/tests.yml | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 55dcbdb..8aefabd 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -55,7 +55,38 @@ jobs: snakemake --snakefile="../workflow/rules/quantify.smk" --configfile="config.yaml" --lint - snakemake-test: + snakemake-conda-test: + runs-on: ubuntu-latest + defaults: + run: + shell: bash -l {0} + + steps: + + - name: check out repository + uses: actions/checkout@v4 + + - name: setup Conda/Mamba + uses: conda-incubator/setup-miniconda@v2 + with: + mamba-version: "*" + activate-environment: mirflowz + environment-file: install/environment.yml + auto-activate-base: false + + - name: display environment info + run: | + conda info -a + conda list + + - name: run test for rule graph + run: bash test/test_rule_graph.sh + + - name: run local test with Conda + run: bash test/test_workflow_local_with_conda.sh + + + snakemake-singularity-test: runs-on: ubuntu-latest defaults: run: @@ -85,9 +116,6 @@ jobs: - name: run local test with Singularity run: bash test/test_workflow_local_with_singularity.sh - - name: run local test with Conda - run: bash test/test_workflow_local_with_conda.sh - unit-testing: runs-on: ubuntu-latest From 0b257d37f134e3ac07002f5615eac608cdf01abb Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Tue, 3 Oct 2023 15:45:56 +0200 Subject: [PATCH 11/23] build: update gffutils version --- workflow/envs/gffutils.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/workflow/envs/gffutils.yaml b/workflow/envs/gffutils.yaml index 2454b97..b375f37 100644 --- a/workflow/envs/gffutils.yaml +++ b/workflow/envs/gffutils.yaml @@ -3,5 +3,5 @@ channels: - conda-forge - bioconda dependencies: - - gffutils=0.11.1 -... \ No newline at end of file + - gffutils>=0.11.1 +... From 64e91434233b61a4c031c255feca673d4411489d Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Tue, 3 Oct 2023 16:01:24 +0200 Subject: [PATCH 12/23] build: disable channel --- install/environment.root.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/install/environment.root.yml b/install/environment.root.yml index de77527..e1c2f87 100644 --- a/install/environment.root.yml +++ b/install/environment.root.yml @@ -2,6 +2,7 @@ name: mirflowz channels: - conda-forge - bioconda + - nodefaults dependencies: - graphviz==7.1.0 - python==3.9.16 From f64361d00181676c437c4fa24ecdf68ec19b73f7 Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Tue, 3 Oct 2023 16:13:08 +0200 Subject: [PATCH 13/23] build: disable 'defaults' channel --- install/environment.dev.root.yml | 1 + install/environment.dev.yml | 1 + install/environment.yml | 1 + 3 files changed, 3 insertions(+) diff --git a/install/environment.dev.root.yml b/install/environment.dev.root.yml index 4716086..c3612b3 100644 --- a/install/environment.dev.root.yml +++ b/install/environment.dev.root.yml @@ -2,6 +2,7 @@ name: mirflowz channels: - conda-forge - bioconda + - nodefaults dependencies: - coverage >=7.2 - flake8 >=3.8.4 diff --git a/install/environment.dev.yml b/install/environment.dev.yml index 75ebb0d..c00fcbc 100644 --- a/install/environment.dev.yml +++ b/install/environment.dev.yml @@ -2,6 +2,7 @@ name: mirflowz channels: - conda-forge - bioconda + - nodefaults dependencies: - coverage >=7.2 - flake8 >=3.8.4 diff --git a/install/environment.yml b/install/environment.yml index 9dcf985..feb61be 100644 --- a/install/environment.yml +++ b/install/environment.yml @@ -2,6 +2,7 @@ name: mirflowz channels: - conda-forge - bioconda + - nodefaults dependencies: - graphviz==7.1.0 - python==3.9.16 From fb5576ffd665e03f1aa446674c12d1790664ec6c Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Wed, 18 Oct 2023 16:56:14 +0200 Subject: [PATCH 14/23] ci: update tests with mamba options --- .github/workflows/tests.yml | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 8aefabd..f68a084 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -23,7 +23,9 @@ jobs: - name: Setup Conda/Mamba uses: conda-incubator/setup-miniconda@v2 with: + use-mamba: true mamba-version: "*" + channel-priority: true activate-environment: mirflowz environment-file: install/environment.dev.yml auto-activate-base: false @@ -69,7 +71,9 @@ jobs: - name: setup Conda/Mamba uses: conda-incubator/setup-miniconda@v2 with: + use-mamba: true mamba-version: "*" + channel-priority: true activate-environment: mirflowz environment-file: install/environment.yml auto-activate-base: false @@ -100,7 +104,9 @@ jobs: - name: setup Conda/Mamba uses: conda-incubator/setup-miniconda@v2 with: + use-mamba: true mamba-version: "*" + channel-priority: true activate-environment: mirflowz environment-file: install/environment.root.yml auto-activate-base: false @@ -131,7 +137,9 @@ jobs: - name: setup Conda/Mamba uses: conda-incubator/setup-miniconda@v2 with: + use-mamba: true mamba-version: "*" + channel-priority: true activate-environment: mirflowz environment-file: install/environment.dev.yml auto-activate-base: false From 9791ed48d1d383ca9256fda99a6d99d359e24bac Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Wed, 18 Oct 2023 17:05:42 +0200 Subject: [PATCH 15/23] build remove nondefualts channel --- install/environment.dev.root.yml | 1 - install/environment.dev.yml | 1 - install/environment.root.yml | 1 - install/environment.yml | 1 - 4 files changed, 4 deletions(-) diff --git a/install/environment.dev.root.yml b/install/environment.dev.root.yml index c3612b3..4716086 100644 --- a/install/environment.dev.root.yml +++ b/install/environment.dev.root.yml @@ -2,7 +2,6 @@ name: mirflowz channels: - conda-forge - bioconda - - nodefaults dependencies: - coverage >=7.2 - flake8 >=3.8.4 diff --git a/install/environment.dev.yml b/install/environment.dev.yml index c00fcbc..75ebb0d 100644 --- a/install/environment.dev.yml +++ b/install/environment.dev.yml @@ -2,7 +2,6 @@ name: mirflowz channels: - conda-forge - bioconda - - nodefaults dependencies: - coverage >=7.2 - flake8 >=3.8.4 diff --git a/install/environment.root.yml b/install/environment.root.yml index e1c2f87..de77527 100644 --- a/install/environment.root.yml +++ b/install/environment.root.yml @@ -2,7 +2,6 @@ name: mirflowz channels: - conda-forge - bioconda - - nodefaults dependencies: - graphviz==7.1.0 - python==3.9.16 diff --git a/install/environment.yml b/install/environment.yml index feb61be..9dcf985 100644 --- a/install/environment.yml +++ b/install/environment.yml @@ -2,7 +2,6 @@ name: mirflowz channels: - conda-forge - bioconda - - nodefaults dependencies: - graphviz==7.1.0 - python==3.9.16 From 66b9448e79127899e95fe616d2b589481c5cdfd4 Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Wed, 18 Oct 2023 17:30:35 +0200 Subject: [PATCH 16/23] ci: specify installation channels --- .github/workflows/tests.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index f68a084..8ea5ba8 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -23,8 +23,8 @@ jobs: - name: Setup Conda/Mamba uses: conda-incubator/setup-miniconda@v2 with: - use-mamba: true mamba-version: "*" + channels: conda-forge,bioconda channel-priority: true activate-environment: mirflowz environment-file: install/environment.dev.yml @@ -71,8 +71,8 @@ jobs: - name: setup Conda/Mamba uses: conda-incubator/setup-miniconda@v2 with: - use-mamba: true mamba-version: "*" + channels: conda-forge,bioconda channel-priority: true activate-environment: mirflowz environment-file: install/environment.yml @@ -104,8 +104,8 @@ jobs: - name: setup Conda/Mamba uses: conda-incubator/setup-miniconda@v2 with: - use-mamba: true mamba-version: "*" + channels: conda-forge,bioconda channel-priority: true activate-environment: mirflowz environment-file: install/environment.root.yml @@ -137,8 +137,8 @@ jobs: - name: setup Conda/Mamba uses: conda-incubator/setup-miniconda@v2 with: - use-mamba: true mamba-version: "*" + channels: conda-forge,bioconda channel-priority: true activate-environment: mirflowz environment-file: install/environment.dev.yml From dd26a02b79fee66d4079e9d47c0be516edac6633 Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Thu, 19 Oct 2023 17:49:19 +0200 Subject: [PATCH 17/23] ci: include Mambaforge as an installer --- .github/workflows/tests.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 8ea5ba8..a697721 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -24,6 +24,7 @@ jobs: uses: conda-incubator/setup-miniconda@v2 with: mamba-version: "*" + miniforge-variant: Mambaforge channels: conda-forge,bioconda channel-priority: true activate-environment: mirflowz @@ -72,6 +73,7 @@ jobs: uses: conda-incubator/setup-miniconda@v2 with: mamba-version: "*" + miniforge-variant: Mambaforge channels: conda-forge,bioconda channel-priority: true activate-environment: mirflowz @@ -105,6 +107,7 @@ jobs: uses: conda-incubator/setup-miniconda@v2 with: mamba-version: "*" + miniforge-variant: Mambaforge channels: conda-forge,bioconda channel-priority: true activate-environment: mirflowz @@ -138,6 +141,7 @@ jobs: uses: conda-incubator/setup-miniconda@v2 with: mamba-version: "*" + miniforge-variant: Mambaforge channels: conda-forge,bioconda channel-priority: true activate-environment: mirflowz From abef37c65437cb3fa124a7c7da16edad52a84432 Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Fri, 20 Oct 2023 13:34:13 +0200 Subject: [PATCH 18/23] ci: fix static code analysis --- scripts/mirna_quantification.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/scripts/mirna_quantification.py b/scripts/mirna_quantification.py index 15d72c4..e1d6ac4 100755 --- a/scripts/mirna_quantification.py +++ b/scripts/mirna_quantification.py @@ -410,7 +410,7 @@ def main(arguments) -> None: alignment = next(samfile) current_species = alignment.get_tag(arguments.tag) if current_species: - read_ID = [alignment.query_name] + read_ID = [str(alignment.query_name)] count = get_contribution(alignment) alns_count = 1 @@ -428,17 +428,17 @@ def main(arguments) -> None: current_species = alignment.get_tag(arguments.tag) count = get_contribution(alignment) alns_count = 1 - read_ID = [alignment.query_name] + read_ID = [str(alignment.query_name)] continue if current_species == alignment.get_tag(arguments.tag): count += get_contribution(alignment) alns_count += 1 - read_ID.append(alignment.query_name) + read_ID.append(str(alignment.query_name)) else: - name = get_name(current_species) + name = get_name(str(current_species)) species = [name[1], str(count)] if arguments.count: @@ -456,9 +456,9 @@ def main(arguments) -> None: current_species = alignment.get_tag(arguments.tag) count = get_contribution(alignment) alns_count = 1 - read_ID = [alignment.query_name] + read_ID = [str(alignment.query_name)] - name = get_name(current_species) + name = get_name(str(current_species)) species = [name[1], str(count)] if arguments.count: From 4b46293cbf3386820200ebe80724dd0ad82314e0 Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Fri, 20 Oct 2023 13:34:20 +0200 Subject: [PATCH 19/23] ci: fix static code analysis --- scripts/iso_name_tagging.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/iso_name_tagging.py b/scripts/iso_name_tagging.py index 5963a30..47c9c66 100755 --- a/scripts/iso_name_tagging.py +++ b/scripts/iso_name_tagging.py @@ -155,7 +155,7 @@ def attributes_dictionary(attr: str) -> Dict[str, str]: def parse_intersect_output( intersect_file: Path, ID: str = "name", - extension: int = 0) -> Optional[Dict[list, list]]: + extension: int = 0) -> Optional[Dict[Optional[str], list]]: """Parse intersect BED file. Given a BED file generated by intersecting a GFF file (-a) with a BAM file @@ -263,7 +263,7 @@ def main(arguments) -> None: for alignment in samfile: alignment_id = alignment.query_name - intersecting_miRNAs = intersect_data.get(alignment_id, []) + intersecting_miRNAs = intersect_data[alignment_id] tags = get_tags(intersecting_mirna=intersecting_miRNAs, alignment=alignment, From bdc8758a550d01a594bc6fc06019264bd4bd9cf6 Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Wed, 25 Oct 2023 20:20:08 +0200 Subject: [PATCH 20/23] refactor: fix variable types --- scripts/mirna_quantification.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/scripts/mirna_quantification.py b/scripts/mirna_quantification.py index 15d72c4..e1d6ac4 100755 --- a/scripts/mirna_quantification.py +++ b/scripts/mirna_quantification.py @@ -410,7 +410,7 @@ def main(arguments) -> None: alignment = next(samfile) current_species = alignment.get_tag(arguments.tag) if current_species: - read_ID = [alignment.query_name] + read_ID = [str(alignment.query_name)] count = get_contribution(alignment) alns_count = 1 @@ -428,17 +428,17 @@ def main(arguments) -> None: current_species = alignment.get_tag(arguments.tag) count = get_contribution(alignment) alns_count = 1 - read_ID = [alignment.query_name] + read_ID = [str(alignment.query_name)] continue if current_species == alignment.get_tag(arguments.tag): count += get_contribution(alignment) alns_count += 1 - read_ID.append(alignment.query_name) + read_ID.append(str(alignment.query_name)) else: - name = get_name(current_species) + name = get_name(str(current_species)) species = [name[1], str(count)] if arguments.count: @@ -456,9 +456,9 @@ def main(arguments) -> None: current_species = alignment.get_tag(arguments.tag) count = get_contribution(alignment) alns_count = 1 - read_ID = [alignment.query_name] + read_ID = [str(alignment.query_name)] - name = get_name(current_species) + name = get_name(str(current_species)) species = [name[1], str(count)] if arguments.count: From e82a7c0e4d599b341d54be65b40bf29ff70d75dd Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Wed, 25 Oct 2023 20:20:24 +0200 Subject: [PATCH 21/23] refactor: fix return type --- scripts/iso_name_tagging.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/iso_name_tagging.py b/scripts/iso_name_tagging.py index 5963a30..47c9c66 100755 --- a/scripts/iso_name_tagging.py +++ b/scripts/iso_name_tagging.py @@ -155,7 +155,7 @@ def attributes_dictionary(attr: str) -> Dict[str, str]: def parse_intersect_output( intersect_file: Path, ID: str = "name", - extension: int = 0) -> Optional[Dict[list, list]]: + extension: int = 0) -> Optional[Dict[Optional[str], list]]: """Parse intersect BED file. Given a BED file generated by intersecting a GFF file (-a) with a BAM file @@ -263,7 +263,7 @@ def main(arguments) -> None: for alignment in samfile: alignment_id = alignment.query_name - intersecting_miRNAs = intersect_data.get(alignment_id, []) + intersecting_miRNAs = intersect_data[alignment_id] tags = get_tags(intersecting_mirna=intersecting_miRNAs, alignment=alignment, From a5a4380b2b0104b1a8af6d535f793115299463ad Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Wed, 25 Oct 2023 20:20:59 +0200 Subject: [PATCH 22/23] build: upgrade and pin dependency versions --- environment.dev.yml | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/environment.dev.yml b/environment.dev.yml index ec77966..618847a 100644 --- a/environment.dev.yml +++ b/environment.dev.yml @@ -3,14 +3,14 @@ channels: - conda-forge - bioconda dependencies: - - coverage>=7.2 - - flake8 >=3.8.4 - - flake8-docstrings >=1.6.0 - - gffutils>=0.11.1 - - mypy >=1.5 - - pylint >=2.7.1 - - pysam>=0.21.0 - - pytest>=7.1.2 + - coverage==7.3 + - flake8==6.1 + - flake8-docstrings==1.7 + - gffutils==0.12 + - mypy==1.6 + - pylint==3.0 + - pysam==0.22 + - pytest==7.4 - pytest-cov - snakefmt - snakemake==7.24.0 From 2063c38233506a2a2c5cbbe8bee8c0c99281c832 Mon Sep 17 00:00:00 2001 From: deliaBlue Date: Wed, 25 Oct 2023 20:41:43 +0200 Subject: [PATCH 23/23] build: upgrade and pin dependecies versions --- install/environment.dev.root.yml | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/install/environment.dev.root.yml b/install/environment.dev.root.yml index 4716086..107b310 100644 --- a/install/environment.dev.root.yml +++ b/install/environment.dev.root.yml @@ -3,15 +3,15 @@ channels: - conda-forge - bioconda dependencies: - - coverage >=7.2 - - flake8 >=3.8.4 - - flake8-docstrings >=1.6.0 - - gffutils >=0.11.1 + - coverage==7.3 + - flake8==6.1 + - flake8-docstrings==1.7 + - gffutils==0.12 - graphviz==7.1.0 - - mypy >=1.5 - - pylint >=2.7.1 - - pysam >=0.21.0 - - pytest >=7.1.2 + - mypy==1.6 + - pylint==3.0 + - pysam==0.22 + - pytest==7.4 - pytest-cov - python==3.9.16 - singularity>=3.5.2