Merge pull request #191 from int-brain-lab/v3.0.0

V3.0.0

.github/workflows/main.yaml

@@ -16,7 +16,7 @@ jobs:
       fail-fast: false  # Whether to stop execution of other instances
       max-parallel: 2
       matrix:
-        python-version: ["3.8", "3.11"]
+        python-version: ["3.10", "3.12"]
         os: ["ubuntu-latest", "windows-latest"]  #  , "macos-latest"
     steps:
     - uses: actions/checkout@v2
@@ -34,7 +34,7 @@ jobs:
         pip install -e .
     - name: run tests
       run: |
-        flake8 .
+        ruff check .
         coverage run --omit=one/tests/* -m unittest discover
     - name: Upload coverage data to coveralls.io
       run: coveralls --service=github

.github/workflows/python-publish.yaml

@@ -11,27 +11,49 @@ permissions:
   contents: read
 
 jobs:
-  deploy:
-    name: Build and publish Python distributions to PyPI
+  build:
+    name: Build Python distributions
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
 
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: '3.x'
 
       - name: Install dependencies
-        run: |
-          python -m pip install --upgrade pip
-          pip install setuptools wheel
+        run: python -m pip install build
 
       - name: Build package
-        run: python setup.py sdist bdist_wheel
+        run: python -m build
 
-      - name: Publish package
-        # GitHub recommends pinning 3rd party actions to a commit SHA.
-        uses: pypa/gh-action-pypi-publish@37f50c210e3d2f9450da2cd423303d6a14a6e29f
+      - name: Store the distribution packages
+        uses: actions/upload-artifact@v4
         with:
-          user: __token__
-          password: ${{ secrets.PYPI_API_TOKEN }}
+          name: python-package-distributions
+          path: dist/
+
+  publish:
+    name: >-
+      Publish Python distributions to PyPI
+    if: startsWith(github.ref, 'refs/tags/')  # only publish to PyPI on tag pushes
+    needs:
+    - build
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/ONE-api
+    permissions:
+      id-token: write  # IMPORTANT: mandatory for trusted publishing
+    steps:
+    - name: Download all the dists
+      uses: actions/download-artifact@v4
+      with:
+        name: python-package-distributions
+        path: dist/
+    - name: Publish distribution to PyPI
+      # GitHub recommends pinning 3rd party actions to a commit SHA.
+      uses: pypa/gh-action-pypi-publish@release/v1
+      with:
+        user: __token__
+        password: ${{ secrets.PYPI_API_TOKEN }}

CHANGELOG.md

@@ -1,5 +1,57 @@
 # Changelog
-## [Latest](https://github.com/int-brain-lab/ONE/commits/main) [2.11.2]
+## [Latest](https://github.com/int-brain-lab/ONE/commits/main) [3.0.0]
+This version drops support for python 3.9 and below, and ONE is now in remote mode by default.
+Also adds a new ALFPath class to replace alf path functions and now returns UUID objects instead of str.
+
+### Modified
+
+- supports python >= 3.10 only
+- OneAlyx uses remote mode by default, instead of auto
+- OneAlyx.search now updates the cache tables in remote mode as paginated sessions are accessed
+- datasets table file_size column nullable by default
+- one.alf.io.save_metadata now returns the saved filepath
+- paths returned by One methods and functions in one.alf.io are now ALFPath instances
+- bugfix: one.alf.path.full_path_parts didn't always raise when invalid path passed
+- one.alf.path module containing ALFPath class
+- ALF cache table generation has lower memory footprint
+- setup in silent mode now uses defaults if base url matches default one
+- bugfix: error downloading from http server with keep_uuids=True
+- one.alf.spec.readableALF and one.alf.spec._dromedary preserve plural acronyms, e.g. 'ROIs'
+- UUID objects returned by to_eid, search, and search_insertions instead of str
+- LazyId objects now handled correctly by One converter methods
+- LazyId object only returned when greater than 1 page of results
+- OneAlyx.search_insertions now updates local cache with query results
+- bugfix: OneAlyx.list_datasets behaves the same as One.list_datasets w.r.t. revisions and default datasets
+- default REST cache expiry reduced from 24 hours to 5 minutes
+- One._update_cache_from_records -> one.alf.cache.merge_tables
+- One.save_cache now updates any tables on disk unless clobber is True
+- one.util.patch_cache -> one.alf.cache.patch_tables
+- One object will save modified tables to disk upon delete
+- bugfix: ONE_HTTP_DL_THREADS environment variable now works as intended
+
+### Added
+
+- one.alf.cache.remove_table_files and One.\_remove_table_files for deleting cache table files
+- one.alf.cache.EMPTY_DATASETS_FRAME and EMPTY_SESSION_FRAME vars for table column, index, and dtype template
+- pyproject.toml replaces deprecated setup file
+- one.alf.exceptions.InvalidALF exception
+- one.params.delete_params
+- One._search_insertions for offline query of insertions when cache table present
+- one.alf.cache.load_tables function to load cache table files with correct index, etc.
+
+### Removed
+
+- setup.py
+- one.alf.files; use one.alf.path instead
+- one.alf.io.remove_uuid_file
+- one.alf.io.remove_uuid_recursive
+- one.util.ensure_list; use iblutil.util.ensure_list instead
+- one.remote.globus.create_globus_client; use one.remote.globus.Globus class instead
+- 'auto' and 'refresh' cache modes have been removed
+- One.refresh_cache
+- `dataset` parameter removed from One.search. Use `datasets` instead.
+
+## [2.11.2]
 
 ### Modified
 
@@ -29,7 +81,7 @@ This version deprecates one.alf.files in preparation for replacing with one.alf.
 - HOTFIX: include Subject/lab part in destination path when downloading from S3
 
 ## [2.10.0]
-This version improves behaviour of loading revisions and loading datasets from list_datasets output.
+This version fixes issues with Alyx authentication in silent mode, and improves behaviour of loading revisions.
 
 ### Modified
 
@@ -44,10 +96,12 @@ This version improves behaviour of loading revisions and loading datasets from l
 - always force authentication when password passed, even when token cached
 - bugfix: negative indexing of paginated response objects now functions correctly
 - deprecate one.util.ensure_list; moved to iblutil.util.ensure_list
+- OneAlyx uses remote mode by default, instead of auto
 
 ### Added
 
 - one.alf.exceptions.ALFWarning category allows users to filter warnings relating to mixed revisions
+- one.alf.cache.remove_cache_table_files and One._remove_cache_table_files for deleting cache table files
 
 ## [2.9.1]
 

CONTRIBUTING.md

@@ -0,0 +1,89 @@
+# Contributing to code
+
+Always branch off branch `main` before commiting changes, then push to remote and open a PR into `main`.
+A developer will then approve the PR and release.
+
+## Before making a pull request
+
+### Commit history
+
+Commit messages should reference a specific issue wherever possible.
+
+### Linting and formatting
+
+We use [ruff](https://docs.astral.sh/ruff/) to lint and format this repository. The ruff settings are defined in [pyproject.toml](./pyproject.toml).
+You can check for linting errors by running `ruff check .`. This should be run within the ONE root directory for the pyproject configuration to be used.
+The linter also checks for docstring formatting.
+
+### Testing and coverage
+
+You should ensure all tests are running locally.  We run tests using python unittest: `python -m unittest discover`. This should be run within the ONE root directory.  Coverage should also be checked as we keep coverage at 100%. Testing with coverage can be run with the following: `coverage run --omit=one/tests/* -m unittest discover`.
+
+## Releasing (maintainers only)
+
+Before merging to main, the CI checks that all unit tests pass and that coverage does not decrease.  Merges should not occur until all checks pass.
+The Github testing workflow can be in [.github/workflows/main.yaml](.github/workflows/main.yaml) and is testing against Python version 3.10 and 3.12 on Ubuntu and Windows.
+The divergent changes to the `docs` branch should be pulled into your release branch as the [.github/workflows/docs.yaml](.github/workflows/docs.yaml) workflow is triggered by both pushes to main and docs.
+
+### Release notes and version
+On a release or feature branch, update the [CHANGELOG.md](./CHANGELOG.md) file with a terse list of changes for the release, oriented for users.  The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).  We organize changes by 'added' - new functions/modules/packages; 'modified' - changes to current functions and modules; 'removed' - deleted functions/modules/packages.  A short one or two sentence description below the version heading should describe the principle changes.  For example:
+
+```markdown
+## [Latest](https://github.com/int-brain-lab/ONE/commits/main) [X.X.X]
+Short sentence describing major changes.
+
+### Modified
+
+- ...
+
+### Added
+
+- ...
+
+### Removed
+
+- ...
+
+```
+
+The ONE version should be iterated. The version is defined in [one/\_\_init\_\_.py](./one/__init__.py) and follows the
+[Python packaging semver specification](https://packaging.python.org/en/latest/specifications/version-specifiers/#version-specifiers).
+
+* Patches (X.X.1) are small or urgent bug fixes or changes that don't affect compatibility.
+* Minor releases (X.1.X) are new features such as added functions or small changes that don't cause major compatibility issues.
+* Major releases (1.X.X) are major new features or changes that break backward compatibility in a big way.
+
+### Releasing to PyPi
+
+Once merged to main you can make a release to PyPi using the [.github/workflows/python-publish.yaml](.github/workflows/python-publish.yaml) workflow:
+
+1. Draft a [release on Github](https://github.com/int-brain-lab/ONE/releases).
+2. Title the release as the version number, starting with 'v'.
+3. Copy and paste the release notes from [CHANGELOG.md](./CHANGELOG.md).
+4. Create a tag with the version. Note that in order to trigger a PyPi release the tag must begin with 'v', e.g. `v2.8.0`.
+
+
+# Contributing to documentation
+## Structure
+The main contents file is found in `docs/index.rst`.  Some pages are written in markdown (.md), and 
+are in the `docs/` folder.  The rest are Jupyter notebooks (.ipynb), placed in the `docs/notebooks/` folder.
+The API reference is automatically generated from the docstrings in the code.  Docstrings should follow
+the NumPy style.  Examples of the NumPy docstring format can be found [here](https://www.sphinx-doc.org/en/master/usage/extensions/example_numpy.html).
+
+## Committing code to GitHub repo
+Commits to the 'docs' branch will trigger the documentation to compile and build automatically.  
+External users should open a pull request to this branch.
+
+## Running locally
+To build the docs locally, first ensure all the requirements are installed:
+```
+pip install -r requirements-docs.txt
+```
+
+Then run the make-script.py file from within the `docs/` folder with the -d flag:
+```
+cd docs/
+python ./make-script.py -d
+```
+
+The HTML files are placed in `docs/_build/html/`.

README.md

@@ -6,10 +6,10 @@ The Open Neurophysiology Environment is a scheme for sharing neurophysiology dat
 
 Please [Click here](https://int-brain-lab.github.io/ONE/) for the main documentation page.  For a quick primer on the file naming convention we use, [click here](https://github.com/int-brain-lab/ONE/blob/main/docs/Open_Neurophysiology_Environment_Filename_Convention.pdf).
 
-**NB**: The API and backend database are still under active development, for the best experience please regularly update the package by running `pip install -U ONE-api`. 
+**NB**: The API and backend database are still under active development, for the best experience please regularly update the package by running `pip install -U ONE-api`.
 
 ## Requirements
-ONE runs on Python 3.8 or later, and is tested on the latest Ubuntu and Windows (3.8 and 3.11 only).
+ONE runs on Python 3.10 or later, and is tested on the latest Ubuntu and Windows (3.10 and 3.12 only).
 
 ## Installing
 Installing the package via pip typically takes a few seconds.  To install, run