Skip to content

Commit

Permalink
add gdb, add build/push container action, misc improvements (#21)
Browse files Browse the repository at this point in the history
* Include GNU debugger `gdb` to `mixer.sif`.
Fixes #20

* rebuilt mixer.sif container

* regressed build for westmere architecture

* rebuilt container for westmere arch

* update installer scripts, add build action

* path

* tar version

* bump Python=3.10

* update plink binaries

* updating docs

* Add opencontainers labels

* use mixer_sif namespace for oras push

* no quad

* remove -mfpmath=sse flag

* test -march=x86_64 -mfpmath=sse flags

* dash

* roll back edits

* documentation for Docker image

* documentation

* roll back mixer build

* updated changelog

* trigger on main branch and tags updates only

* edit path
  • Loading branch information
espenhgn authored May 21, 2024
1 parent 843f1aa commit 334f3f2
Show file tree
Hide file tree
Showing 18 changed files with 366 additions and 53 deletions.
87 changes: 87 additions & 0 deletions .github/workflows/container_build_push.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
name: container-build-push

on:
push:
branches:
- 'main'
tags:
- "v*.*.*"

jobs:
docker:
runs-on: ubuntu-latest
steps:
- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
# list of Docker images to use as base name for tags
images: |
ghcr.io/${{ github.repository_owner }}/mixer
# generate Docker tags based on the following events/attributes
tags: |
type=schedule
type=ref,event=branch
type=ref,event=pr
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}}
type=sha
type=raw,value=latest,enable={{is_default_branch}}
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to GitHub Container Registry
if: github.event_name != 'pull_request'
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push
uses: docker/build-push-action@v5
with:
context: "{{defaultContext}}:src"
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
file:
./dockerfiles/mixer/Dockerfile
platforms: linux/amd64

build-apptainer-container:
needs: docker
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
name: Build Apptainer Container
steps:
- name: Check out code for the container builds
uses: actions/checkout@v4
- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: |
ghcr.io/${{ github.repository_owner }}/mixer
tags: |
type=semver,pattern={{version}}
type=ref,event=branch
type=sha
type=raw,value=latest,enable={{is_default_branch}}
- name: Build and push Apptainer container
run: |
tags="${{ steps.meta.outputs.tags }}"
csv_tags=$(printf "%s\n" "$tags" | awk -F: 'NR==1{printf "%s,", $0; next} {printf "%s,", $NF}' | sed 's/,$//')
IFS= read -r first_tag <<EOF
$tags
EOF
push_tags="$(printf $csv_tags | sed -e "s/mixer/mixer_sif/g")"
echo ${{ secrets.GITHUB_TOKEN }} | oras login --username ${{ github.repository_owner }} --password-stdin ghcr.io
docker pull kaczmarj/apptainer:latest
docker run --rm --privileged -v $(pwd):/work kaczmarj/apptainer build mixer.sif docker://"$first_tag"
oras push "$push_tags" mixer.sif
rm mixer.sif
shell: sh
7 changes: 6 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,6 @@
.DS_Store
.DS_Store
.bash_history
usecases/partial.*
usecases/shared.*
usecases/unique.*
usecases/.ipython
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Updated

- Added installation information to the README.md file.
- Mambaforge for Python 3.10.14 environment (was Python 3.8)
- Updated PLINK1.9 (20231211) and PLINK2.0 (20231211) versions (and revised corresponding syntax in mixer_simu example)
- Updated apt package versions to current
- Revised installer scripts
- Updated READMEs and example files

### Added

- Building and pushing the MiXeR images in both Docker and Singularity format to the [ghcr.io](https://ghcr.io) registry is now automated using GitHub Actions.
- Added `gdb` to `apt_get_essentials.sh` for debugging.
- Version pinned for Ubuntu, conda and conda packages.
- Run `conda clean -a -y` in Dockerfile reducing image size.
- Added [`usecases/run_mixer.ipynb`](usecases/run_mixer.ipynb) Jupyter notebook for simplifying Slurm job submissions for lists of traits.
Expand All @@ -17,6 +28,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed

- Use Ubuntu 20.04 LTS as base image for Docker and Singularity containers.
- Rebuilt container w. md5 checksum b344119fd1f8f9c425f0edea78c08881 (commit: 3882739)

### Fixed

Expand Down
104 changes: 102 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# MiXeR

Singularity wrapper for <https://github.com/precimed/mixer>
This project provides [Singularity](https://sylabs.io/singularity/) wrapper for <https://github.com/precimed/mixer>,
as well as [Docker](https://www.docker.com/) images for the same software.

The original reference and example data are included in this repository.

Expand All @@ -14,4 +15,103 @@ To get started, see:

- examples of how to use mixer.sif container on [synthetic](usecases/mixer_simu.md) and [real](usecases/mixer_real.md) data.

- Jupyter notebook [usecases/run_mixer.ipynb] for submitting a bunch of MiXeR jobs at once, with lists of primary and secondary traits.
- Jupyter notebook [usecases/run_mixer.ipynb](usecases/run_mixer.ipynb) for submitting a bunch of MiXeR jobs at once, with lists of primary and secondary traits.

## Installation and set up

### Dependencies on host system

In order to set up these resource, some software may be required

- [Singularity/SingularityCE](https://sylabs.io/singularity/) or [Apptainer](https://apptainer.org)
- [Git](https://git-scm.com/)
- [Git LFS](https://git-lfs.com)
- [ORAS CLI](https://oras.land)

### Clone the repository

To download the last revision of this project, issue:

```bash
cd path/to/repositories
git clone --depth 1 https://github.com/comorment/mixer.git
cd mixer
git lfs pull # pull "large" files
```

### Update the `mixer.sif` container

To obtain updated versions of the Singularity Image Format (.sif) container file `, issue

```bash
cd path/to/repositories/mixer/singularity
mv mixer.sif mixer.sif.old # optional, just rename the old(er) file
apptainer pull docker://ghcr.io/comorment/mixer:<tag> # or
singularity pull docker://ghcr.io/comorment/mixer:<tag> # or
oras pull ghcr.io/comorment/mixer_sif:<tag>
```

where `<tag>` corresponds to a tag listed under [packages](https://github.com/comorment/mixer/pkgs/container/mixer),
such as `latest`, `main`, or `sha_<GIT SHA>`.
The `oras pull` statement pulls the `mixer.sif` file from [ghcr.io](https://github.com/comorment/mixer/pkgs/container/mixer_sif) using the [ORAS](https://oras.land) registry, without the need to build the container locally.

### Pulling and using Docker image

To pull the corresponding Docker image, issue:

```bash
docker pull ghcr.io/comorment/mixer:<tag>
```

If working on recent Macs, add the `--platform=linux/amd64` after `docker pull`.
This may allow replacing `singularity exec ...` or `apptainer exec ...` statements with appropriate `docker run ...` statements in the [usecases/mixer_simu](usecases/mixer_simu#docker-details) section,
on systems where Singularity or Apptainer is unavailable.
Functionally, the Docker image is equivalent to the Singularity container, but note that syntax for mounting volumes and invoking commands may differ.
Please refer to[docs.docker.com](https://docs.docker.com) for more information.

> [!NOTE] Note that the provided Docker image may not support all CPUs, and may not be able to run on all systems via CPU virtualization.
> An option may be to build the Docker image on the host machine (e.g., M1 Macs, older Intel CPUs), as:
>
>```bash
>docker build --platform=linux/amd64 -t ghcr.io/comorment/mixer -f dockerfiles/mixer/Dockerfile .
>```
An exampe of using the Docker image is provided in the [usecases/mixer_simu](usecases/mixer_simu#docker-details) section.
## Systems without internet access
Some secure platforms do not have direct internet access, hence we recommend cloning/pulling all required files on a machine with internet access as explained above, and archive the `mixer` directory with all files and moving it using whatever file uploader is available for the platform.
```bash
cd /path/to/mixer
SHA=$(git rev-parse --short HEAD)
cd ..
tar --exclude=".git/*" -cvf mixer_$SHA.tar mixer
```
## Citation info

If you use the software provided here, please cite our relevant preprint:

```
Akdeniz, B.C., Frei, O., Hagen, E., Filiz, T.T., Karthikeyan, S., Pasman, J.A., Jangmo, A., Bergsted, J., Shorter, J.R., Zetterberg, R., Meijsen, J.J., Sønderby, I.E., Buil, A., Tesli, M., Lu, Y., Sullivan, P., Andreassen, O.A., & Hovig, E. (2022). COGEDAP: A COmprehensive GEnomic Data Analysis Platform. arXiv:2212.14103 [q-bio.GN]. DOI: [10.48550/arXiv.2212.14103](https://doi.org/)
```

Bibtex format:
```
@misc{akdeniz2022cogedap,
title={COGEDAP: A COmprehensive GEnomic Data Analysis Platform},
author={Bayram Cevdet Akdeniz and Oleksandr Frei and Espen Hagen and Tahir Tekin Filiz and Sandeep Karthikeyan and Joelle Pasman and Andreas Jangmo and Jacob Bergsted and John R. Shorter and Richard Zetterberg and Joeri Meijsen and Ida Elken Sonderby and Alfonso Buil and Martin Tesli and Yi Lu and Patrick Sullivan and Ole Andreassen and Eivind Hovig},
year={2022},
eprint={2212.14103},
archivePrefix={arXiv},
primaryClass={q-bio.GN}
}
```

Note that this project will soon fall under the "[COSGAP](https://cosgap.readthedocs.io/en/latest/)" umbrella, and that the citation info will be updated accordingly.

For the [MiXeR software](https://github.com/precimed/mixer) itself, if you use MiXeR software for your research publication, please cite the following paper(s):

* for univariate analysis: D. Holland et al., Beyond SNP Heritability: Polygenicity and Discoverability Estimated for Multiple Phenotypes with a Univariate Gaussian Mixture Model, PLOS Genetics, 2020, https://doi.org/10.1371/journal.pgen.1008612
* for cross-trait analysis: O.Frei et al., Bivariate causal mixture model quantifies polygenic overlap between complex traits beyond genetic correlation, Nature Communications, 2019, https://www.nature.com/articles/s41467-019-10310-0
11 changes: 7 additions & 4 deletions src/Makefile
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
all: mixer.sif

%.sif: dockerfiles/%/Dockerfile
docker build -t $* -f dockerfiles/$*/Dockerfile . && scripts/convert_docker_image_to_singularity.sh $*
docker build -t $* -f dockerfiles/$*/Dockerfile . && \
scripts/convert_docker_image_to_singularity.sh $* && \
scripts/move_singularity_file.sh $*

# Example commands executed by this make file
#
# Run "make hello.sif" will trigger this:
# docker build -t hello -f dockerfiles/hello/Dockerfile . && scripts/convert_docker_image_to_singularity.sh hello # produces hello.sif
#
# Run "make mixer.sif" will trigger this:
# docker build -t mixer -f dockerfiles/mixer/Dockerfile . && \
# scripts/convert_docker_image_to_singularity.sh mixer && \ # produces mixer.sif
# scripts/move_singularity_file.sh mixer # moves mixer.sif to the <mixer>/singularity directory
25 changes: 15 additions & 10 deletions src/dockerfiles/mixer/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
FROM ubuntu:20.04

LABEL org.opencontainers.image.source=https://github.com/comorment/mixer
LABEL org.opencontainers.image.description="Container image for MiXeR (https://github.com/precimed/mixer)"
LABEL org.opencontainers.image.licenses=GPL-3.0

ENV TZ=Europe
ENV DEBIAN_FRONTEND noninteractive

Expand All @@ -9,32 +13,33 @@ RUN chmod +x apt_get_essential.sh && \
bash apt_get_essential.sh

WORKDIR /tmp
COPY /scripts/install_miniconda3.sh .
RUN chmod +x install_miniconda3.sh && \
bash install_miniconda3.sh

RUN conda install conda=23.9.0 && \
conda config --add channels conda-forge && \
conda config --set channel_priority strict && \
conda install \
COPY /scripts/install_mambaforge.sh .
RUN chmod +x install_mambaforge.sh && \
bash install_mambaforge.sh

RUN mamba install python=3.10.14 \
jupyterlab=4.0.7 \
matplotlib=3.7.3 \
matplotlib-venn=0.11.9 \
numpy=1.24.4 \
numdifftools=0.9.41 \
pandas=2.0.3 \
python=3.8 \
scipy=1.10.1 \
seaborn=0.13.0 \
statsmodels=0.14.0 \
--yes && \
conda clean -a -y
mamba clean -a -y

WORKDIR /tools/plink
COPY /scripts/install_plink.sh /tmp
RUN chmod +x /tmp/install_plink.sh
RUN bash /tmp/install_plink.sh

WORKDIR /tools
COPY /scripts/install_boost.sh /tmp
RUN chmod +x /tmp/install_boost.sh
RUN bash /tmp/install_boost.sh

WORKDIR /tools
COPY /scripts/install_mixer.sh /tmp
RUN chmod +x /tmp/install_mixer.sh
Expand Down
23 changes: 13 additions & 10 deletions src/scripts/apt_get_essential.sh
Original file line number Diff line number Diff line change
@@ -1,39 +1,42 @@
#!/bin/sh
set -eou pipefail

apt-get update && apt-get install -y \
ca-certificates=20230311ubuntu0.20.04.1 && \
update-ca-certificates

# (!) Keep the list below sorted (!)

apt-get install -y --no-install-recommends \
apt-utils=2.0.9 \
apt-utils=2.0.10 \
autoconf=2.69-11.1 \
build-essential=12.8ubuntu1.1 \
bzip2=1.0.8-2 \
cmake=3.16.3-1ubuntu1.20.04.1 \
curl=7.68.0-1ubuntu2.20 \
curl=7.68.0-1ubuntu2.22 \
dos2unix=7.4.0-2 \
gdb=9.2-0ubuntu1~20.04.1 \
gfortran=4:9.3.0-1ubuntu2 \
git=1:2.25.1-1ubuntu3.11 \
less=551-1ubuntu0.1 \
less=551-1ubuntu0.3 \
libatlas-base-dev=3.10.3-8ubuntu7 \
libcurl4=7.68.0-1ubuntu2.20 \
libcurl4-openssl-dev=7.68.0-1ubuntu2.20 \
libcurl4=7.68.0-1ubuntu2.22 \
libcurl4-openssl-dev=7.68.0-1ubuntu2.22 \
libgomp1=10.5.0-1ubuntu1~20.04 \
libgsl-dev=2.5+dfsg-6build1 \
libnss3=2:3.49.1-1ubuntu1.9 \
libgsl-dev=2.5+dfsg-6+deb10u1build0.20.04.1 \
libnss3=2:3.98-0ubuntu0.20.04.2 \
libpcre2-dev=10.34-7ubuntu0.1 \
libquadmath0=10.5.0-1ubuntu1~20.04 \
libxt-dev=1:1.1.5-1 \
make=4.2.1-1.2 \
pandoc=2.5-3build2 \
pandoc-citeproc=0.15.0.1-1build4 \
parallel=20161222-1.1 \
perl=5.30.0-9ubuntu0.4 \
tar=1.30+dfsg-7ubuntu0.20.04.3 \
perl=5.30.0-9ubuntu0.5 \
tar=1.30+dfsg-7ubuntu0.20.04.4 \
tofrodos=1.7.13+ds-4 \
unzip=6.0-25ubuntu1.1 \
vim=2:8.1.2269-1ubuntu5.18 \
vim=2:8.1.2269-1ubuntu5.22 \
wget=1.20.3-1ubuntu2 \
zlib1g-dev=1:1.2.11.dfsg-2ubuntu1.5

Expand Down
3 changes: 2 additions & 1 deletion src/scripts/convert_docker_image_to_singularity.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/usr/bin/env bash

set -e
set -eou pipefail

# This script documents how to build the singularity container
# from the Dockerfile

Expand Down
8 changes: 8 additions & 0 deletions src/scripts/install_boost.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#!/bin/sh
set -euo pipefail

cd /tools
wget https://boostorg.jfrog.io/artifactory/main/release/1.69.0/source/boost_1_69_0.tar.gz
tar -xzvf boost_1_69_0.tar.gz && cd boost_1_69_0
./bootstrap.sh --with-libraries=program_options,filesystem,system,date_time
./b2 --clean && ./b2 --j12 -a
9 changes: 9 additions & 0 deletions src/scripts/install_mambaforge.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#!/bin/sh
set -euo pipefail

curl -sSL https://github.com/conda-forge/miniforge/releases/download/24.3.0-0/Mambaforge-24.3.0-0-$(uname)-$(uname -m).sh -o /tmp/mambaforge.sh \
&& mkdir /root/.conda \
&& bash /tmp/mambaforge.sh -bfp /usr/local \
&& rm -rf /tmp/mambaforge.sh

export PATH=$PATH:/opt/conda/bin
Loading

0 comments on commit 334f3f2

Please sign in to comment.