diff --git a/contribute/index.html b/contribute/index.html index 39a4f849..62a5a8fa 100644 --- a/contribute/index.html +++ b/contribute/index.html @@ -310,6 +310,15 @@ + + +
To create a release, increase the version in pyproject.toml
, prepare changelog.md
,
+commit everything and run release.py
. This script will
stable
branch and push it to GitHubThe only manual step that remains is creating a release on GitHub. +To do so, you can paste the changelog section of the release and create a new release on GitHub using the tag that was created.
diff --git a/index.html b/index.html index 48ca1063..585d23d1 100644 --- a/index.html +++ b/index.html @@ -241,6 +241,15 @@ + + +If you installed with pipx
, you can update the package to the most recent release with the following command:
pipx upgrade bfabric
+
Create a file as follows: (note: the password is not your login password, but the web service password available on your profile page)
# ~/.bfabricpy.yml
diff --git a/search/search_index.json b/search/search_index.json
index ee5dd6b3..7cfb0fb8 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":"This package connects the bfabric system to the python and R world while providing a JSON and REST interface using Flask. The bfabricShiny R package is an extension and provides code snippets and sample implementation for a seamless R shiny bfabric integration. For more advanced users the bfabricPy package also provides a powerful query interface on the command-line though using the provided scripts.
Please see below for how to install bfabricPy.
"},{"location":"#installation","title":"Installation","text":"The package is not available on PyPI as of now, but can be installed directly from GitHub and a stable
branch is available for your convenience.
If you are only interested in running the command line scripts, installation with pipx
is recommended as it will create a separate virtual environment for bfabricPy and make it possible to upgrade your installation later easily.
pipx install \"git+https://github.com/fgcz/bfabricPy.git@stable\"\n
Note that pipx
is also useful in scripts, if you want to run a particular version without forcing the global installation of that version (simply replace \"stable\" with a tag of your chosing):
pipx run --spec \"git+https://github.com/fgcz/bfabricPy.git@stable\" bfabric_read.py --help\n
If you want to add it to a pyproject.toml
the syntax for specifying a git dependency is as follows:
[project]\ndependencies = [\n \"bfabric @ git+https://github.com/fgcz/bfabricPy.git@stable\"\n]\n
"},{"location":"#configuration","title":"Configuration","text":"Create a file as follows: (note: the password is not your login password, but the web service password available on your profile page)
# ~/.bfabricpy.yml\n\nGENERAL:\n default_config: PRODUCTION\n\nPRODUCTION:\n login: yourBfabricLogin\n password: yourBfabricWebPassword\n base_url: https://fgcz-bfabric.uzh.ch/bfabric\n
You can also include an additional config for the TEST instance
TEST:\n login: yourBfabricLogin\n password: yourBfabricWebPassword\n base_url: https://fgcz-bfabric-test.uzh.ch/bfabric\n
"},{"location":"changelog/","title":"Changelog","text":"The format is based on Keep a Changelog.
Versioning currently follows X.Y.Z
where
X
is used for major changes, that contain breaking changes Y
should be the current bfabric release Z
is increased for feature releases, that should not break the API
"},{"location":"changelog/#unreleased","title":"[Unreleased]","text":""},{"location":"changelog/#1135-2024-08-13","title":"[1.13.5] - 2024-08-13","text":""},{"location":"changelog/#added","title":"Added","text":" - The
Bfabric
instance is now pickleable. - Entities mapping:
- Add
Entity.id
and Entity.web_url
properties. - Add
Entity.__getitem__
and Entity.get
to access fields from the data dictionary directly. - Add
Entity.find_by
to find entities by a query. - More types and relationships
- Relationships defer imports to descriptor call, i.e. circular relationships are possible now.
HasOne
and HasMany
allow defining optional=True
to indicate fields which can be missing under some circumstances. - Add
nodelist
column and application name to bfabric_list_not_available_proteomics_workunits.py
output.
"},{"location":"changelog/#changed","title":"Changed","text":" Entity.find_all
supports more than 100 IDs now by using the experimental MultiQuery API.
"},{"location":"changelog/#1134-2024-08-05","title":"[1.13.4] - 2024-08-05","text":""},{"location":"changelog/#added_1","title":"Added","text":" - Add
Workunit
, Parameter
, and Resource
entities. - Add concept of has_many and has_one relationships to entities.
bfabric_slurm_queue_status.py
to quickly check slurm queue status. Bfabric.save
provides method
which can be set to checkandinsert
for specific use cases.
"},{"location":"changelog/#changed_1","title":"Changed","text":" - Most messages are now logged to debug level.
- The old verbose version information is now always logged, to INFO level, since it could entail useful information for error reporting.
"},{"location":"changelog/#1133-2024-07-18","title":"[1.13.3] - 2024-07-18","text":""},{"location":"changelog/#added_2","title":"Added","text":" - Flask
- New endpoint
GET /config/remote_base_url
for testing
"},{"location":"changelog/#changed_2","title":"Changed","text":" - Flask
- Simplify logging by using loguru only.
- Simplified setup logic since the production use case should use a WSGI server.
"},{"location":"changelog/#fixed","title":"Fixed","text":" bfabric_save_csv2dataset.py
had an undeclared dependency on numpy and a few bugs which was improved.
"},{"location":"changelog/#1132-2024-07-11","title":"[1.13.2] - 2024-07-11","text":""},{"location":"changelog/#added_3","title":"Added","text":" - Add
bfabric.entities.Dataset
to easily read datasets. - Pydantic-based configuration parsing
- The config format did not change.
- The code is easier to maintain now.
- Additionally, there is a lot more validation of the configuration file now, that should catch errors early.
- Make host and port configurable in
bfabric_flask.py
(currently only dev mode).
"},{"location":"changelog/#1131-2024-07-02","title":"[1.13.1] - 2024-07-02","text":""},{"location":"changelog/#changed_3","title":"Changed","text":" - bfabric_save_csv2dataset will raise an error if problematic characters are found in any of the cells
- Correctly define
bfabric_setWorkunitStatus_available.py
, and processing
and failed
variants.
"},{"location":"changelog/#added_4","title":"Added","text":" - Add loguru for future logging refactoring.
- Easily runnable tests with
nox
and standardized formatting using pre-commit
.
"},{"location":"changelog/#removed","title":"Removed","text":" - Pandas is no longer a dependency, and has been replaced by polars.
"},{"location":"changelog/#1130-2024-05-24","title":"[1.13.0] - 2024-05-24","text":"This is a major release refactoring bfabricPy's API.
"},{"location":"changelog/#changed_4","title":"Changed","text":" - The
Bfabric
class operations now return ResultContainer
objects. - These provide a list-like interface to access individual items or iterate over them.
- Individual items are a dictionary, potentially nested, and not specific to suds/zeep anymore.
- Convenience conversions, e.g. to a polars DataFrame, can be provided there.
- Configuration is now defined in
~/.bfabricpy.yml
and supports multiple configurations, which can be selected by the BFABRICPY_CONFIG_ENV
environment variable. Please consult the README for an example configuration. - Use
pyproject.toml
for package configuration. - Scripts have been refactored on a case-by-case basis.
"},{"location":"changelog/#added_5","title":"Added","text":" - Zeep can be used instead of suds for SOAP communication.
Bfabric
can be instantiated without authentication, that can be provided later. This is useful in a server setup. - Pagination support in
Bfabric
, specify the number of max_results and a potential offset. Pages handling is abstracted away. - Detect errors in responses, e.g. invalid login.
"},{"location":"changelog/#removed_1","title":"Removed","text":" - Several old scripts have been moved into a
deprecated_scripts
folder. - Wrapper creator related code is currently not updated but has been extracted into a dedicated folder
wrapper_creator
as well.
"},{"location":"contribute/","title":"Contribute","text":"This page describes some information relevant for contributing to bfabricPy.
"},{"location":"contribute/#development-install","title":"Development install","text":"You should install the dev
group as it contains some extra packages for running the tests.
pip install -e \".[dev]\"\n
"},{"location":"contribute/#running-tests","title":"Running tests","text":"With nox
and uv
installed, it is as simple as running nox
in the project root without any arguments.
"},{"location":"contribute/#integration-tests","title":"Integration tests","text":"Note that integration tests have been moved to a separate repository. Please contact us if you are interested.
"},{"location":"contribute/#documentation","title":"Documentation","text":"We currently do not have a versioning solution for the documentation, but we can add that later once it is more mature.
# To preview while you write it\nmkdocs serve\n\n# To publish after changes\nmkdocs gh-deploy\n
"},{"location":"entities/","title":"Entities","text":""},{"location":"entities/#high-level-bfabricentities-api","title":"High-level bfabric.entities
API","text":"The bfabric.entities
module offers a high-level, read-only API for retrieval of entities in the B-Fabric system.
This API is designed to simplify entity access by providing lazy-loading capabilities for associated entities. Entities within this module can maintain a reference to a B-Fabric client instance, enabling seamless integration and data retrieval.
"},{"location":"entities/#key-features","title":"Key Features","text":" - Read-Only Access: This module is strictly for read-only operations. None of the classes or methods should be used to modify the database.
- Lazy-Loading: Entities are initialized with basic data as returned by a .read operation on its endpoint. However, associated entities are not loaded until explicitly requested, enhancing performance and reducing unnecessary data processing.
- Modular Design: Only include generic functionality within this module. Avoid adding highly specific logic to ensure the module remains maintainable over time.
- No Circular Imports: Ensure there are no circular dependencies between entities by deferring imports of other entities inside modules until all relevant modules are processed.
"},{"location":"entities/#entity","title":"Entity","text":"Each entity has to provide the name of its B-Fabric entity classname as ENDPOINT
. An entity is defined uniquely by its classname
and id
.
Entities are initialized with the full result structure obtained from a read
operation on its particular endpoint. A return_id_only
result is generally not supposed to be used to initialize an entity instance and behavior is undefined as of now. In cases where it makes sense to operate on the ids, it's better to directly call the Bfabric
client methods.
Entities can be retrieved with the methods Entity.find
, Entity.find_all
and Entity.find_by
. When loading multiple entities of the same type, Entity.find_all
and Entity.find_by
should be preferred. These two methods return a dictionary mapping entity ID to instance and this often helps avoid redundant API calls when expanding relationships over multiple entities.
"},{"location":"entities/#relationships","title":"Relationships","text":"Custom descriptors are available which are used to represent relationships and enable lazy-loading of data. Currently the following descriptors are available:
HasOne
: Defines a one-to-one relationship between entities. HasMany
: Defines a one-to-many relationship between entities.
Both support setting an additional argument optional=True
which will ensure None
(in the case of HasOne
) and an empty collection (in the case of HasMany
) are returned when there is no field at all in the result returned by the API.
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":"This package connects the bfabric system to the python and R world while providing a JSON and REST interface using Flask. The bfabricShiny R package is an extension and provides code snippets and sample implementation for a seamless R shiny bfabric integration. For more advanced users the bfabricPy package also provides a powerful query interface on the command-line though using the provided scripts.
Please see below for how to install bfabricPy.
"},{"location":"#installation","title":"Installation","text":"The package is not available on PyPI as of now, but can be installed directly from GitHub and a stable
branch is available for your convenience.
If you are only interested in running the command line scripts, installation with pipx
is recommended as it will create a separate virtual environment for bfabricPy and make it possible to upgrade your installation later easily.
pipx install \"git+https://github.com/fgcz/bfabricPy.git@stable\"\n
Note that pipx
is also useful in scripts, if you want to run a particular version without forcing the global installation of that version (simply replace \"stable\" with a tag of your chosing):
pipx run --spec \"git+https://github.com/fgcz/bfabricPy.git@stable\" bfabric_read.py --help\n
If you want to add it to a pyproject.toml
the syntax for specifying a git dependency is as follows:
[project]\ndependencies = [\n \"bfabric @ git+https://github.com/fgcz/bfabricPy.git@stable\"\n]\n
"},{"location":"#updating","title":"Updating","text":"If you installed with pipx
, you can update the package to the most recent release with the following command:
pipx upgrade bfabric\n
"},{"location":"#configuration","title":"Configuration","text":"Create a file as follows: (note: the password is not your login password, but the web service password available on your profile page)
# ~/.bfabricpy.yml\n\nGENERAL:\n default_config: PRODUCTION\n\nPRODUCTION:\n login: yourBfabricLogin\n password: yourBfabricWebPassword\n base_url: https://fgcz-bfabric.uzh.ch/bfabric\n
You can also include an additional config for the TEST instance
TEST:\n login: yourBfabricLogin\n password: yourBfabricWebPassword\n base_url: https://fgcz-bfabric-test.uzh.ch/bfabric\n
"},{"location":"changelog/","title":"Changelog","text":"The format is based on Keep a Changelog.
Versioning currently follows X.Y.Z
where
X
is used for major changes, that contain breaking changes Y
should be the current bfabric release Z
is increased for feature releases, that should not break the API
"},{"location":"changelog/#unreleased","title":"[Unreleased]","text":""},{"location":"changelog/#1135-2024-08-13","title":"[1.13.5] - 2024-08-13","text":""},{"location":"changelog/#added","title":"Added","text":" - The
Bfabric
instance is now pickleable. - Entities mapping:
- Add
Entity.id
and Entity.web_url
properties. - Add
Entity.__getitem__
and Entity.get
to access fields from the data dictionary directly. - Add
Entity.find_by
to find entities by a query. - More types and relationships
- Relationships defer imports to descriptor call, i.e. circular relationships are possible now.
HasOne
and HasMany
allow defining optional=True
to indicate fields which can be missing under some circumstances. - Add
nodelist
column and application name to bfabric_list_not_available_proteomics_workunits.py
output.
"},{"location":"changelog/#changed","title":"Changed","text":" Entity.find_all
supports more than 100 IDs now by using the experimental MultiQuery API.
"},{"location":"changelog/#1134-2024-08-05","title":"[1.13.4] - 2024-08-05","text":""},{"location":"changelog/#added_1","title":"Added","text":" - Add
Workunit
, Parameter
, and Resource
entities. - Add concept of has_many and has_one relationships to entities.
bfabric_slurm_queue_status.py
to quickly check slurm queue status. Bfabric.save
provides method
which can be set to checkandinsert
for specific use cases.
"},{"location":"changelog/#changed_1","title":"Changed","text":" - Most messages are now logged to debug level.
- The old verbose version information is now always logged, to INFO level, since it could entail useful information for error reporting.
"},{"location":"changelog/#1133-2024-07-18","title":"[1.13.3] - 2024-07-18","text":""},{"location":"changelog/#added_2","title":"Added","text":" - Flask
- New endpoint
GET /config/remote_base_url
for testing
"},{"location":"changelog/#changed_2","title":"Changed","text":" - Flask
- Simplify logging by using loguru only.
- Simplified setup logic since the production use case should use a WSGI server.
"},{"location":"changelog/#fixed","title":"Fixed","text":" bfabric_save_csv2dataset.py
had an undeclared dependency on numpy and a few bugs which was improved.
"},{"location":"changelog/#1132-2024-07-11","title":"[1.13.2] - 2024-07-11","text":""},{"location":"changelog/#added_3","title":"Added","text":" - Add
bfabric.entities.Dataset
to easily read datasets. - Pydantic-based configuration parsing
- The config format did not change.
- The code is easier to maintain now.
- Additionally, there is a lot more validation of the configuration file now, that should catch errors early.
- Make host and port configurable in
bfabric_flask.py
(currently only dev mode).
"},{"location":"changelog/#1131-2024-07-02","title":"[1.13.1] - 2024-07-02","text":""},{"location":"changelog/#changed_3","title":"Changed","text":" - bfabric_save_csv2dataset will raise an error if problematic characters are found in any of the cells
- Correctly define
bfabric_setWorkunitStatus_available.py
, and processing
and failed
variants.
"},{"location":"changelog/#added_4","title":"Added","text":" - Add loguru for future logging refactoring.
- Easily runnable tests with
nox
and standardized formatting using pre-commit
.
"},{"location":"changelog/#removed","title":"Removed","text":" - Pandas is no longer a dependency, and has been replaced by polars.
"},{"location":"changelog/#1130-2024-05-24","title":"[1.13.0] - 2024-05-24","text":"This is a major release refactoring bfabricPy's API.
"},{"location":"changelog/#changed_4","title":"Changed","text":" - The
Bfabric
class operations now return ResultContainer
objects. - These provide a list-like interface to access individual items or iterate over them.
- Individual items are a dictionary, potentially nested, and not specific to suds/zeep anymore.
- Convenience conversions, e.g. to a polars DataFrame, can be provided there.
- Configuration is now defined in
~/.bfabricpy.yml
and supports multiple configurations, which can be selected by the BFABRICPY_CONFIG_ENV
environment variable. Please consult the README for an example configuration. - Use
pyproject.toml
for package configuration. - Scripts have been refactored on a case-by-case basis.
"},{"location":"changelog/#added_5","title":"Added","text":" - Zeep can be used instead of suds for SOAP communication.
Bfabric
can be instantiated without authentication, that can be provided later. This is useful in a server setup. - Pagination support in
Bfabric
, specify the number of max_results and a potential offset. Pages handling is abstracted away. - Detect errors in responses, e.g. invalid login.
"},{"location":"changelog/#removed_1","title":"Removed","text":" - Several old scripts have been moved into a
deprecated_scripts
folder. - Wrapper creator related code is currently not updated but has been extracted into a dedicated folder
wrapper_creator
as well.
"},{"location":"contribute/","title":"Contribute","text":"This page describes some information relevant for contributing to bfabricPy.
"},{"location":"contribute/#development-install","title":"Development install","text":"You should install the dev
group as it contains some extra packages for running the tests.
pip install -e \".[dev]\"\n
"},{"location":"contribute/#running-tests","title":"Running tests","text":"With nox
and uv
installed, it is as simple as running nox
in the project root without any arguments.
"},{"location":"contribute/#integration-tests","title":"Integration tests","text":"Note that integration tests have been moved to a separate repository. Please contact us if you are interested.
"},{"location":"contribute/#documentation","title":"Documentation","text":"We currently do not have a versioning solution for the documentation, but we can add that later once it is more mature.
# To preview while you write it\nmkdocs serve\n\n# To publish after changes\nmkdocs gh-deploy\n
"},{"location":"contribute/#release","title":"Release","text":"To create a release, increase the version in pyproject.toml
, prepare changelog.md
, commit everything and run release.py
. This script will
- Create a tag and push it to GitHub
- Merge the changes to the
stable
branch and push it to GitHub - Build the documentation and push it to GitHub Pages
The only manual step that remains is creating a release on GitHub. To do so, you can paste the changelog section of the release and create a new release on GitHub using the tag that was created.
"},{"location":"entities/","title":"Entities","text":""},{"location":"entities/#high-level-bfabricentities-api","title":"High-level bfabric.entities
API","text":"The bfabric.entities
module offers a high-level, read-only API for retrieval of entities in the B-Fabric system.
This API is designed to simplify entity access by providing lazy-loading capabilities for associated entities. Entities within this module can maintain a reference to a B-Fabric client instance, enabling seamless integration and data retrieval.
"},{"location":"entities/#key-features","title":"Key Features","text":" - Read-Only Access: This module is strictly for read-only operations. None of the classes or methods should be used to modify the database.
- Lazy-Loading: Entities are initialized with basic data as returned by a .read operation on its endpoint. However, associated entities are not loaded until explicitly requested, enhancing performance and reducing unnecessary data processing.
- Modular Design: Only include generic functionality within this module. Avoid adding highly specific logic to ensure the module remains maintainable over time.
- No Circular Imports: Ensure there are no circular dependencies between entities by deferring imports of other entities inside modules until all relevant modules are processed.
"},{"location":"entities/#entity","title":"Entity","text":"Each entity has to provide the name of its B-Fabric entity classname as ENDPOINT
. An entity is defined uniquely by its classname
and id
.
Entities are initialized with the full result structure obtained from a read
operation on its particular endpoint. A return_id_only
result is generally not supposed to be used to initialize an entity instance and behavior is undefined as of now. In cases where it makes sense to operate on the ids, it's better to directly call the Bfabric
client methods.
Entities can be retrieved with the methods Entity.find
, Entity.find_all
and Entity.find_by
. When loading multiple entities of the same type, Entity.find_all
and Entity.find_by
should be preferred. These two methods return a dictionary mapping entity ID to instance and this often helps avoid redundant API calls when expanding relationships over multiple entities.
"},{"location":"entities/#relationships","title":"Relationships","text":"Custom descriptors are available which are used to represent relationships and enable lazy-loading of data. Currently the following descriptors are available:
HasOne
: Defines a one-to-one relationship between entities. HasMany
: Defines a one-to-many relationship between entities.
Both support setting an additional argument optional=True
which will ensure None
(in the case of HasOne
) and an empty collection (in the case of HasMany
) are returned when there is no field at all in the result returned by the API.
"}]}
\ No newline at end of file