Merge pull request #109 from fgcz/main

Release 1.13.9
fgcz · Dec 10, 2024 · 4ca5ba4 · 4ca5ba4
2 parents 8c74f22 + 8dde678
commit 4ca5ba4
Show file tree

Hide file tree

Showing 122 changed files with 2,800 additions and 1,834 deletions.
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
@@ -0,0 +1,18 @@
+version: 2
+updates:
+  - package-ecosystem: "pip"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    groups:
+      actions:
+        patterns:
+          - "*"
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    groups:
+      actions:
+        patterns:
+          - "*"
diff --git a/.github/workflows/build_app_runner.yml b/.github/workflows/build_app_runner.yml
@@ -0,0 +1,51 @@
+name: Build App Runner
+on:
+  workflow_dispatch:
+  release:
+    types: [published]
+    tags:
+      - "app-runner/*"
+jobs:
+  build_app_runner:
+    name: Build App Runner
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Build
+        run: |
+          bash app_runner/deploy/build.sh build-output app_runner
+          ls -l build-output
+      - name: Verify
+        run: |
+          build-output/app_runner/app_runner --help
+      - name: Upload
+        uses: actions/upload-artifact@v4
+        with:
+          name: app_runner_linux_x86_64
+          path: build-output
+  publish_release_artifact:
+    name: Publish Release Artifact
+    runs-on: ubuntu-latest
+    needs: build_app_runner
+    if: github.event_name == 'release'
+    steps:
+      - name: Download Artifact
+        uses: actions/download-artifact@v4
+        with:
+          name: app_runner_linux_x86_64
+          path: app_runner_linux_x86_64
+      - name: Zip for release
+        run: |
+          cd app_runner_linux_x86_64
+          zip -r app_runner_linux_x86_64.zip *
+          ls -l
+      - name: Print folder structure
+        run: |
+          tree
+      - name: Upload Release Assets
+        uses: softprops/action-gh-release@v2
+        with:
+          files: |
+            app_runner_linux_x86_64/app_runner_linux_x86_64.zip
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.github/workflows/run_unit_tests.yml b/.github/workflows/run_unit_tests.yml
@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - uses: actions/setup-python@v2
+      - uses: actions/setup-python@v5
         with:
           python-version: 3.9
       - name: Install nox

diff --git a/.gitignore b/.gitignore
@@ -1,7 +1,8 @@
 .idea/
 __pycache__
-bfabric.egg-info/
+*.egg-info/
 bfabric/scripts/query_result.txt
 build/
 dist/
 site/
+_build/
diff --git a/README.md b/README.md
@@ -1,271 +1,28 @@
-![unitTests](https://github.com/fgcz/bfabricPy/workflows/unit%20tests/badge.svg)
-[![EDBT'10](https://img.shields.io/badge/EDBT-10.1145%2F1739041.1739135-brightgreen)](https://doi.org/10.1145/1739041.1739135)
-[![JIB](https://img.shields.io/badge/JIB-10.1515%2Fjib.2022.0031-brightgreen)](https://doi.org/10.1515/jib-2022-0031)
-
 # bfabricPy
 
-This package connects the [bfabric](https://fgcz-bfabric.uzh.ch/bfabric/) system to the [python](https://www.python.org/) and [R](https://cran.r-project.org/) world while providing a JSON and REST interface using [Flask](https://www.fullstackpython.com).
-The [bfabricShiny](https://github.com/cpanse/bfabricShiny) R package is an extension and provides code snippets and sample implementation for a seamless R shiny bfabric integration.
-For more advanced users the *bfabricPy* package also provides a powerful query interface on the command-line though using the provided scripts.
-
-You can find the up-to-date documentation at [https://fgcz.github.io/bfabricPy](https://fgcz.github.io/bfabricPy).
-
-## CheatSheet
-
-### Read
-
-```{bash}
-bfabric_read.py storage
-bfabric_read.py application
-```
-
-Simple database query examples
-
-```{bash}
-bfabric_read.py user login cpanse
-bfabric_read.py project id 3000
-bfabric_read.py workunit id 199387
-bfabric_read.py sample name autoQC4L
-bfabric_read.py workunit status processing
-bfabric_read.py workunit status pending
-bfabric_read.py workunit status failed
-
-# list empty resources
-bfabric_read.py resource filechecksum d41d8cd98f00b204e9800998ecf8427e
-```
-
-Using the Python API:
-
-```{py}
-from bfabric import Bfabric
-
-client = Bfabric.from_config()
-
-user = client.read(endpoint = 'user', obj={'login': 'cpanse'})
-resource = client.read(endpoint = 'resource', obj={'id': 550327 })
-```
-
-### save
-
-```{bash}
-bfabric_save_workunit_attribute.py 199387 status available
-```
-
-```{python}
-import json
-rv = client.save('workunit', {'id': 254063, 'status': 'available'})
-print(json.dumps(rv.to_list_dict(), indent=2))
-```
-
-### Command line code snippet
-
-Find empty resource files in bfabric
-
-```{bash}
-bfabric_read.py resource filechecksum `md5sum < /dev/null | cut -c-32` \
-  | cat -n \
-  | tail
-```
-
-## Examples \[outdated\]
-
-### bash script generated by the yaml wrapper creator / submitter
-
-externaljobid-45939_executableid-15312.bash listing:
-
-```bash
-#!/bin/bash
-#
-# $HeadURL: http://fgcz-svn.uzh.ch/repos/scripts/trunk/linux/bfabric/apps/python/README.md $
-# $Id: README.md 2535 2016-10-24 08:49:17Z cpanse $
-# Christian Panse <[email protected]> 2007-2015
-
-# Grid Engine Parameters
-#$ -q PRX@fgcz-c-071
-#$ -e /home/bfabric/sgeworker/logs/workunitid-134923_resourceid-203236.err
-#$ -o /home/bfabric/sgeworker/logs/workunitid-134923_resourceid-203236.out
-
-
-set -e
-set -o pipefail
-
-export EXTERNALJOBID=45938
-export RESSOURCEID_OUTPUT=203238
-export RESSOURCEID_STDOUT_STDERR="203237 203238"
-export OUTPUT="[email protected]:/srv/www/htdocs//p1000/bfabric/Proteomics/gerneric_yaml/2015/2015-09/2015-09-02//workunit_134923//203236.zip"
-
-# job configuration set by B-Fabrics wrapper_creator executable
-_OUTPUT=`echo $OUTPUT | cut -d"," -f1`
-test $? -eq 0 && _OUTPUTHOST=`echo $_OUTPUT | cut -d":" -f1`
-test $? -eq 0 && _OUTPUTPATH=`echo $_OUTPUT | cut -d":" -f2`
-test $? -eq 0 && _OUTPUTPATH=`dirname $_OUTPUTPATH`
-test $? -eq 0 && ssh $_OUTPUTHOST "mkdir -p $_OUTPUTPATH"
-
-if [ $? -eq 1 ];
-then
-    echo "writting to output url failed!";
-    exit 1;
-fi
-
-cat > /tmp/yaml_config.$$ <<EOF
-application:
-  input:
-    mascot_dat:
-    - [email protected]//usr/local/mascot/:/data/20150807/F221967.dat
-    - [email protected]//usr/local/mascot/:/data/20150807/F221973.dat
-  output:
-  - [email protected]:/srv/www/htdocs//p1000/bfabric/Proteomics/gerneric_yaml/2015/2015-09/2015-09-02//workunit_134923//203236.zip
-  parameters:
-    gelcms: 'true'
-    mudpit: 'false'
-    qmodel: None
-    xtandem: 'false'
-  protocol: scp
-job_configuration:
-  executable: /usr/local/fgcz/proteomics/bin/fgcz_scaffold.bash
-  external_job_id: 45938
-  input:
-    mascot_dat:
-    - 201919
-    - 201918
-  output:
-    protocol: scp
-    resource_id: 203238
-    ssh_args: -o StrictHostKeyChecking=no -c arcfour -2 -l bfabric -x
-  stderr:
-    protocol: file
-    resource_id: 203237
-    url: /home/bfabric/sgeworker/logs/workunitid-134923_resourceid-203236.err
-  stdout:
-    protocol: file
-    resource_id: 203238
-    url: /home/bfabric/sgeworker/logs/workunitid-134923_resourceid-203236.out
-  workunit_id: 134923
-
-EOF
-
-# debug / host statistics
-hostname
-uptime
-echo $0
-pwd
-
-# run the application
-test -f /tmp/yaml_config.$$ && /usr/local/fgcz/proteomics/bin/fgcz_scaffold.bash /tmp/yaml_config.$$
-
-if [ $? -eq 0 ];
-then
-    /home/bfabric/.python/fgcz_bfabric_setResourceStatus_available.py $RESSOURCEID_OUTPUT
-    /home/bfabric/.python/fgcz_bfabric_setExternalJobStatus_done.py $EXTERNALJOBID
-else
-    echo "application failed"
-    /home/bfabric/.python/fgcz_bfabric_setResourceStatus_available.py $RESSOURCEID_STDOUT_STDERR $RESSOURCEID;
-    exit 1;
-fi
-
-
-# Should be available also as zero byte files
-
-/home/bfabric/.python/fgcz_bfabric_setResourceStatus_available.py $RESSOURCEID_STDOUT_STDERR
-
-exit 0
-```
-
-### curl example
-
-```{bash}
-#!/bin/bash
-
-query(){
-  url=$1 \
-  && curl \
-    ${url} \
-    -v \
-    --header "Content-Type: text/xml;charset=UTF-8" \
-    --header "SOAPAction: read" \
-    -d '
-  <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:end="http://endpoint.webservice.component.bfabric.org/">
-     <soapenv:Header/>
-     <soapenv:Body>
-        <end:read>
-           <parameters>
-              <login>XXX</login>
-              <password>XXX</password>
-              <query>
-                 <id>482</id>
-              </query>
-           </parameters>
-        </end:read>
-     </soapenv:Body>
-  </soapenv:Envelope>'
-}
-
-for url in https://fgcz-bfabric.uzh.ch/bfabric/user?wsdl https://fgcz-bfabric-test.uzh.ch/bfabric/user?wsdl;
-do
-  echo
-  echo "==== ${url} === "
-  query ${url}
-done
-
-echo $?
-```
-
-### Example usage
-
-remove accidentally inserted mgf files
-
-```
-bfabric_read.py importresource \
-  | grep mgf$ \
-  | awk '{print $1}' \
-  | tee /tmp/$$.log \
-  | while read i;
-  do
-    bfabric_delete.py importresource $i ;
-  done
-```
-
-## Send an E-mail \[outdated\]
-
-```
-# by CT,CP
-# not implemented yet 2022-10-19 ,
-rv = B.save_object(endpoint = 'mail',
-    obj={'subject': "TEST",
-       'recipientemail': '[email protected]',
-       'message': "TEST; ignore that email",
-       'parentId': 482,
-       'parentClassName': 'user'})
-# shown as mail for user id 482
-```
-
-## See also
-
-- [bfabric documentation](https://fgcz-bfabric.uzh.ch/wiki/HomePage)
-- [FAQ](faq.md)
-- [wsdl4BFabric](http://fgcz-intranet.uzh.ch/tiki-index.php?page=wsdl4BFabric) wiki page
-- WSDL Interface to B-Fabric [endpoints](http://fgcz-bfabric.uzh.ch/bfabric/workunit?wsdl)
-
-## FAQ
-
-### How to resolve `<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed`?
-
-on macOSX
-
-```
-cd /Applications/Python 3.12 && ./Install\ Certificates.command
-```
+[![PR Checks](https://github.com/fgcz/bfabricPy/actions/workflows/run_unit_tests.yml/badge.svg)](https://github.com/fgcz/bfabricPy/actions/workflows/run_unit_tests.yml)
+[![Nightly Integration Tests](https://github.com/fgcz/bfabricPy-tests/actions/workflows/nightly_tests.yml/badge.svg)](https://github.com/fgcz/bfabricPy-tests/actions/workflows/nightly_tests.yml)
+[![EDBT'10](https://img.shields.io/badge/EDBT-10.1145%2F1739041.1739135-brightgreen)](https://doi.org/10.1145/1739041.1739135)
+[![JIB](https://img.shields.io/badge/JIB-10.1515%2Fjib.2022.0031-brightgreen)](https://doi.org/10.1515/jib-2022-0031)
 
-### How is the version numbering working?
+## Documentation
 
-X.Y.Z
+| Package    | Link                                                                 |
+| ---------- | -------------------------------------------------------------------- |
+| bfabricPy  | [https://fgcz.github.io/bfabricPy](https://fgcz.github.io/bfabricPy) |
+| app-runner | https://fgcz.github.io/bfabricPy/app_runner/                         |
 
-X is used for major changes, that contain breaking changes
+## Introduction
 
-Y should be the current bfabric release
+This package implements a Python interface to the [B-Fabric](https://fgcz-bfabric.uzh.ch/bfabric/) system.
+Several pieces of functionality are available:
 
-Z is increased for feature releases, that should not break the API
+- Python API:
+    - General client for all B-Fabric web service operations (CRUD) and configuration management.
+    - A relational API for low-boilerplate read access to the B-Fabric system.
+- Scripts: Several scripts we use more or less frequently to interact with the system.
+- A REST API: A REST API to interact with the B-Fabric system. This allows us to interact with B-Fabric from R using [bfabricShiny](https://github.com/cpanse/bfabricShiny).
 
-### Howto cite?
+## Howto cite?
 
 Panse, Christian, Trachsel, Christian and Türker, Can. "Bridging data management platforms and visualization tools to enable ad-hoc and smart analytics in life sciences" Journal of Integrative Bioinformatics, 2022, pp. 20220031. [doi: 10.1515/jib-2022-0031](https://doi.org/10.1515/jib-2022-0031).
diff --git a/app_runner/.gitignore b/app_runner/.gitignore
@@ -0,0 +1,2 @@
+builds
+app_runner.spec
diff --git a/app_runner/README.md b/app_runner/README.md
@@ -0,0 +1,15 @@
+The app runner is an experimental tool that standardizes the integration of one-off apps into B-Fabric.
+
+**The API is subject to drastic changes in the next time.**
+
+The main idea is that an app provides a specification of the following steps:
+
+- dispatch -> create `inputs.yml` files and 1 `chunks.yml` file
+- process -> process a particular chunk (after inputs have been prepared)
+- collect -> collect the results of a chunk and create `outputs.yml` files
+
+The individual app can be in a container environment or a script running in the same environment as the app runner.
+
+To make this possible input and output staging is abstracted and communicated through `inputs.yml` and `outputs.yml`
+specification files.
+A command is available to stage the inputs or register the outputs respectively then.