Merge branch 'main' into add_optional_slow_tests

.github/workflows/deploy.yaml

@@ -9,6 +9,8 @@ jobs:
   build-n-publish:
     name: Build and publish Python 🐍 distributions 📦 to PyPI and TestPyPI
     runs-on: ubuntu-latest
+    permissions:
+      id-token: write  # IMPORTANT: this permission is mandatory for trusted publishing
     steps:
     - uses: actions/checkout@master
     - name: Set up Python 3.9
@@ -25,13 +27,11 @@ jobs:
         twine check dist/*
         ls -l dist
     - name: Publish distribution 📦 to Test PyPI
-      uses: pypa/gh-action-pypi-publish@master
+      uses: pypa/gh-action-pypi-publish@release/v1
       with:
         password: ${{ secrets.TEST_PYPI_API_TOKEN }}
         repository_url: https://test.pypi.org/legacy/
         skip_existing: true
     - name: Publish distribution 📦 to PyPI
       if: github.event_name == 'release'
-      uses: pypa/gh-action-pypi-publish@master
-      with:
-        password: ${{ secrets.PYPI_API_TOKEN }}
+      uses: pypa/gh-action-pypi-publish@release/v1

.readthedocs.yaml

@@ -0,0 +1,26 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the version of Python and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+  configuration: docs/conf.py
+
+formats:
+  - pdf
+
+# We recommend specifying your dependencies to enable reproducible builds:
+# https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
+python:
+  install:
+  - path: .
+  - requirements: docs/requirements.txt

CHANGELOG.rst

@@ -1,3 +1,11 @@
+2.3.3 - Feb 13, 2024. (sha 97405a5e8ab586b80037f036caa3cea0eabfeb10)
+--------------------------------------------------------------------
+* Fix get_dataset_size for datasets with null file_size column by @mvdbeek (PR #116)
+* Add contributing doc and update readme by @afgane (PR #122)
+* Add readthedocs configuration by @nuwang (PR #121)
+* Add to to_dict functions to Destination and TagSetManager classes by @pauldg (PR #119)
+* Add a helper to get the object store ids and the datasets size for every dataset in a job by @sanjaysrikakulam (PR #125)
+
 2.3.2 - Aug 25, 2023. (sha 26ddcfb024679a8bb2d698461e7fcbfa8453c45e)
 --------------------------------------------------------------------
 * Fix legacy version parsing by using Galaxy's version parsing if available by @bgruening (PR #112)

CONTRIBUTING.md

@@ -0,0 +1,8 @@
+# Contributing
+
+Contributions are welcome to the Total Perspective Vortex library. To help with
+code consistency across the Galaxy project ecosystem, TPV adheres to the same
+contribution guidelines as the Galaxy application.
+
+For details, please look at [Galaxy Contributing
+document](https://github.com/galaxyproject/galaxy/blob/dev/CONTRIBUTING.md).

README.md

@@ -4,20 +4,37 @@
 
 ## <img src="https://raw.githubusercontent.com/galaxyproject/total-perspective-vortex/main/docs/images/tpv-logo-wide.png" width="800" height="100">
 
-TotalPerspectiveVortex (TPV) provides an installable set of dynamic rules for the
-[Galaxy application](https://galaxyproject.org/) that can route entities (Tools, Users, Roles) to appropriate
-job destinations based on a configurable yaml file.
+Total Perspective Vortex (TPV) provides an installable set of dynamic rules for
+the [Galaxy application](https://galaxyproject.org/) that can route entities
+(Tools, Users, Roles) to appropriate job destinations based on a configurable
+YAML file.
 
-### Shared database
 
-A shared database of reusable rules are maintained in: https://github.com/galaxyproject/tpv-shared-database/
+### Documentation
+
+Documentation on how to enable TPV in your Galaxy instance and configure the
+relevant routing rules is available on [Read the
+Docs](http://total-perspective-vortex.readthedocs.org/).
+
+For a guided tutorial, also check out the [TPV tutorial on
+GTN](https://training.galaxyproject.org/training-material/topics/admin/tutorials/job-destinations/tutorial.html).
+
 
-### Installing latest development version
+### Installing latest released version
+
+Once configured in the Galaxy application, TPV will be automatically installed
+in the Galaxy environment. If you would like to manually install it instead, you
+can do so from [PyPI](https://pypi.org/project/total-perspective-vortex/) with
+the following command:
 
 ```python
 pip install total-perspective-vortex
 ```
 
-### Documentation
 
-[http://total-perspective-vortex.readthedocs.org](http://total-perspective-vortex.readthedocs.org/en/latest/?badge=latest)
+### Shared default resource database for tools
+
+A shared database of reusable default resource requirements and scheduling rules
+for TPV is maintained in: https://github.com/galaxyproject/tpv-shared-database/.
+(This is likely something you want to take a look at - it will save a lot of
+time.)

docs/index.rst

@@ -2,25 +2,23 @@
 
 .. centered:: Dynamic rules for routing Galaxy entities to destinations
 
-TotalPerspectiveVortex (TPV) provides an installable set of dynamic rules for the
-`Galaxy application`_ that can route entities (Tools, Users, Roles) to appropriate
-destinations based on a configurable yaml file. The aim of TPV is to build on and
-unify previous efforts, such as `Dynamic Tool Destinations`_, the `Job Router`_ and
-`Sorting Hat`_, into a configurable set of rules that that can be extended arbitrarily
-with custom Python logic.
+TotalPerspectiveVortex (TPV) is a plugin for the `Galaxy application`_ that can route
+entities (Tools, Users, Roles) to appropriate destinations with appropriate resource
+alloations (cores, gpus, memory), based on a configurable yaml file. For example, it could
+allocate 8 cores and 32GB of RAM to a bwa-mem job, and route it to a Slurm cluster, while
+allocating 2 cores and 4GB of RAM to an upload job, and route it to a local runner. These
+rules can also be shared community-wide, imported at runtime by any Galaxy deployment, and
+overridden locally when necessary.
 
 How it works
 ------------
-TPV provides a dynamic rule that can be plugged into Galaxy via ``job_conf.yml``.
-The dynamic rule will also have an associated configuration file, that maps entities
-(tools, users, roles) to specific destination through a flexible tagging system.
-Destinations can have arbitrary scheduling tags defined, and each entity can express a preference
-or aversion to specific scheduling tags. Based on this tagging, jobs are routed to the most appropriate
-destination. In addition, admins can also plugin arbitrary python based rules for making
-more complex decisions, as well as custom ranking functions for choosing between matching
-destinations. For example, a ranking function could query influx metrics to determine
-the least loaded destination, and route jobs there, providing a basic form of
-"metascheduling" functionality.
+TPV can be plugged into Galaxy via ``job_conf.yml``. TPVs configuration file specifies how entities
+(tools, users, roles) should be allocated resources (cores, gpus, memory) and in complex environments
+with multiple job destinations, where to map the resulting jobs to (through a flexible
+tagging system). Destinations can have arbitrary scheduling tags defined, and each entity can express a
+preference or aversion to specific scheduling tags. This tagging affects how jobs are routed to
+destinations. In addition, admins can also plugin arbitrary python based rules for making more complex
+decisions, as well as custom ranking functions for choosing between matching destinations.
 
 Shared database
 ---------------
@@ -29,34 +27,19 @@ A shared database of TPV rules are maintained in: https://github.com/galaxyproje
 These rules are based on typical settings used in the usegalaxy.* federation, which you can override
 based on local resource availability.
 
-Getting Started
----------------
-
-1. ``pip install total-perspective-vortex`` into Galaxy's python virtual environment
-2. Configure Galaxy to use TPV's dynamic destination rule
-3. Create the TPV job mapping yaml file, indicating job routing preferences
-4. Submit jobs as usual
-
-Standalone Installation
------------------------
-
-If you wish to install TPV outside of Galaxy's virtualenv (e.g. to use the ``tpv lint`` command locally or in a CI/CD
-pipeline), use the ``[cli]`` pip requirement specifier to make sure the necessary Galaxy dependency packages are also
-installed. **This should not be used in the Galaxy virtualenv**:
-
-.. code-block:: console
-
-   $ pip install 'total-perspective-vortex[cli]'
 
 .. toctree::
    :maxdepth: 2
    :caption: Contents:
 
+   topics/installation.rst
    topics/tpv_by_example.rst
+   topics/advanced_topics.rst
    topics/concepts.rst
-   topics/configure_galaxy.rst
+   topics/inner_workings.rst
    topics/shell_commands.rst
    topics/migration_guide.rst
+   topics/faq.rst
 
 Indices and tables
 ==================

docs/requirements.txt

@@ -0,0 +1,2 @@
+sphinx>=2
+sphinx-rtd-theme>=0.5.2

docs/samples/job_conf.yml

@@ -20,7 +20,7 @@ execution:
       function: map_tool_to_destination
       rules_module: tpv.rules
       tpv_config_files:
-        - https://github.com/galaxyproject/total-perspective-vortex/raw/main/tpv/tests/fixtures/mapping-rules.yml
+        - https://gxy.io/tpv/db.yml
         - config/tpv_rules_local.yml
     local:
       runner: local