[WIP] Rio tiler v4 (#528)

* update rasterio requirement (#517) * update rasterio requirement * remove python 3.7 * update changelog * reader functions returns ImageData object (#515) * reader functions returns ImageData object * update changelog * remove async base reader (#520) * remove gcp reader (#521) Co-authored-by: vincentsarago <[email protected]> * avoid warning in tests * generalize buffer option (#519) * generalize buffer option * refactor buffer calculation * better tests * remove min/max zoom from baseclass attribute (#522) * remove min/max zoom from baseclass attribute * fix benchmark * split zooms methods and add tests * rename function * move apply_expression into ImageData class and use `b{ix}` for band names (#523) * move apply_expression into ImageData class and use `b{ix}` for band names * fix test * make sure we have the good number of bands * Apply expression part2 (#525) * ImageData.apply_expression return a new ImageData object * refactor Points method and deprecate asset_expression * remove unused option * Update rio_tiler/models.py * fix regex * add PointData class (#526) * add PointData class * fix test * remove deprecated * fix test 2 * better expression parsing for stac * docstring fixes * only use WarpedVRT when doing reprojection or nodata overwride (#529) * update baseclass to remove useless kwargs * update changelog * forward dataset statistics to ImageClass (#531) * forward dataset statistics to ImageClass * better type and autorescale * one more test * rename `COGReader` to `Reader` and remove relative import (#534) * rename COGReader to Reader and remove relative import * rasterio * debug * merge from branch * add note for backwards compatibility of COGReader * remove individual options attributes (#535) * remove individual options attributes * add typeDict for options * allow empty options * make sure to not change the input array * add ImageReader for non-geo images (#536) * add ImageReader for non-geo images * update changelog * Xarray reader (#530) * sketch out Xarray support * Update rio_tiler/io/xarray.py Co-authored-by: Ryan Abernathey <[email protected]> * read tile * assume crs is in the dataset * upside down * dependencies * cleanup * add more methods and cleanup * Update rio_tiler/io/xarray.py Co-authored-by: Tom Augspurger <[email protected]> * enumerate from 1 * fix * add dataset_statistics and notebook * add XarrayReader in top level import * add tests * update notebook * update changelog Co-authored-by: Ryan Abernathey <[email protected]> Co-authored-by: Tom Augspurger <[email protected]> * add migration docs * update readme * update notebooks * update docs Co-authored-by: Jeff Albrecht <[email protected]> Co-authored-by: Ryan Abernathey <[email protected]> Co-authored-by: Tom Augspurger <[email protected]>
cogeotiff · Oct 19, 2022 · 4baf505 · 4baf505
1 parent c6b097a
commit 4baf505
Show file tree

Hide file tree

Showing 60 changed files with 5,356 additions and 1,807 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -16,7 +16,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ['3.7', '3.8', '3.9']
+        python-version: ['3.8', '3.9', '3.10']
 
     steps:
       - uses: actions/checkout@v2
@@ -28,17 +28,16 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          python -m pip install pre-commit codecov
-
-      - name: Install module
-        run: python -m pip install .["test"]
+          python -m pip install -e .["test"]
 
       - name: Run pre-commit
         if: ${{ matrix.python-version == env.LATEST_PY_VERSION }}
-        run: pre-commit run --all-files
+        run: |
+          python -m pip install pre-commit
+          pre-commit run --all-files
 
       - name: Run tests
-        run: python -m pytest --cov rio_tiler --cov-report xml --cov-report term-missing --benchmark-skip
+        run: python -m pytest --cov rio_tiler --cov-report xml --cov-report term-missing --benchmark-skip -s -vv
 
       - name: Upload Results
         if: ${{ matrix.python-version == env.LATEST_PY_VERSION }}
@@ -73,20 +72,20 @@ jobs:
       - name: Run Benchmark
         run: python -m pytest --benchmark-only --benchmark-autosave --benchmark-columns 'min, max, mean, median' --benchmark-sort 'min' --benchmark-json output.json
 
-      - name: Store and Compare benchmark result
-        uses: benchmark-action/github-action-benchmark@v1
-        with:
-          name: rio-tiler Benchmarks
-          tool: 'pytest'
-          output-file-path: output.json
-          alert-threshold: '130%'
-          comment-on-alert: true
-          fail-on-alert: true
-          # GitHub API token to make a commit comment
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          # Make a commit on `gh-pages` only if master
-          auto-push: ${{ github.ref == 'refs/heads/master' }}
-          benchmark-data-dir-path: benchmarks
+      # - name: Store and Compare benchmark result
+      #   uses: benchmark-action/github-action-benchmark@v1
+      #   with:
+      #     name: rio-tiler Benchmarks
+      #     tool: 'pytest'
+      #     output-file-path: output.json
+      #     alert-threshold: '130%'
+      #     comment-on-alert: true
+      #     fail-on-alert: true
+      #     # GitHub API token to make a commit comment
+      #     github-token: ${{ secrets.GITHUB_TOKEN }}
+      #     # Make a commit on `gh-pages` only if master
+      #     auto-push: ${{ github.ref == 'refs/heads/master' }}
+      #     benchmark-data-dir-path: benchmarks
 
   publish:
     needs: [tests]

diff --git a/.github/workflows/deploy_mkdocs.yml b/.github/workflows/deploy_mkdocs.yml
@@ -44,8 +44,9 @@ jobs:
             rio_tiler.expression \
             rio_tiler.models \
             rio_tiler.io.base \
-            rio_tiler.io.cogeo \
+            rio_tiler.io.rasterio \
             rio_tiler.io.stac \
+            rio_tiler.io.xarray \
             rio_tiler.mosaic.methods.base \
             rio_tiler.mosaic.methods.defaults \
             rio_tiler.mosaic.reader \

diff --git a/.gitignore b/.gitignore
@@ -108,4 +108,4 @@ tests/benchmarks/data/*
 tests/fixtures/mask*
 .vscode/settings.json
 
-docs/api*
+docs/src/api/*
diff --git a/CHANGES.md b/CHANGES.md
@@ -1,4 +1,162 @@
 
+# 4.0.0 (TBD)
+
+* add python 3.10 support
+* add `apply_expression` method in `rio_tiler.models.ImageData` class
+* update `rio-tiler.reader.read/part` to avoid using WarpedVRT when no *reprojection* or *nodata override* is needed
+* add `rio_tiler.io.rasterio.ImageReader` to work either with Non-geo or Geo images in a Non-geo manner (a.k.a: in the pixel coordinates system)
+    ```python
+    with ImageReader("image.jpg") as src:
+        im = src.part((0, 100, 100, 0))
+
+    with ImageReader("image.jpg") as src:
+        im = src.tile(0, 0, src.maxzoom)
+        print(im.bounds)
+
+    >>> BoundingBox(left=0.0, bottom=256.0, right=256.0, top=0.0)
+    ```
+
+* add `rio_tiler.io.xarray.XarrayReader` to work with `xarray.DataArray`
+    ```python
+    import xarray
+    from rio_tiler.io import XarrayReader
+
+    with xarray.open_dataset(
+        "https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/noaa-coastwatch-geopolar-sst-feedstock/noaa-coastwatch-geopolar-sst.zarr",
+        engine="zarr",
+        decode_coords="all"
+    ) as src:
+        ds = src["analysed_sst"][:1]
+        ds.rio.write_crs("epsg:4326", inplace=True)
+
+        with XarrayReader(ds) as dst:
+            img = dst.tile(1, 1, 2)
+    ```
+    note: `xarray` and `rioxarray` optional dependencies are needed for the reader
+
+**breaking changes**
+
+* remove python 3.7 support
+* update rasterio requirement to `>=1.3` to allow python 3.10 support
+* rename `rio_tiler.io.cogeo` to `rio_tiler.io.rasterio`
+* rename `COGReader` to `Reader`. We added `rio_tiler.io.COGReader` alias to `rio_tiler.io.Reader` backwards compatibility
+    ```python
+    # before
+    from rio_tiler.io import COGReader
+    from rio_tiler.io.cogeo import COGReader
+
+    # now
+    from rio_tiler.io import Reader
+    from rio_tiler.io.rasterio import Reader
+    ```
+
+* `rio_tiler.readers.read()`, `rio_tiler.readers.part()`, `rio_tiler.readers.preview()` now return a ImageData object
+* remove `minzoom` and `maxzoom` attribute in `rio_tiler.io.SpatialMixin` base class
+* remove `minzoom` and `maxzoom` attribute in `rio_tiler.io.Reader` (now defined as properties)
+* use `b` prefix for band names in `rio_tiler.models.ImageData` class (and in rio-tiler's readers)
+    ```python
+    # before
+    with COGReader("cog.tif") as cog:
+        img = cog.read()
+        print(cog.band_names)
+        >>> ["1", "2", "3"]
+
+        print(cog.info().band_metadata)
+        >>> [("1", {}), ("2", {}), ("3", {})]
+
+        print(cog.info().band_descriptions)
+        >>> [("1", ""), ("2", ""), ("3", "")]
+
+        print(list(cog.statistics()))
+        >>> ["1", "2", "3"]
+
+    # now
+    with Reader("cog.tif") as cog:
+        img = cog.read()
+        print(img.band_names)
+        >>> ["b1", "b2", "b3"]
+
+        print(cog.info().band_metadata)
+        >>> [("b1", {}), ("b2", {}), ("b3", {})]
+
+        print(cog.info().band_descriptions)
+        >>> [("b1", ""), ("b2", ""), ("b3", "")]
+
+        print(list(cog.statistics()))
+        >>> ["b1", "b2", "b3"]
+
+    with STACReader("stac.json") as stac:
+        print(stac.tile(701, 102, 8, assets=("green", "red")).band_names)
+        >>> ["green_b1", "red_b1"]
+    ```
+
+* depreciate `asset_expression` in MultiBaseReader. Use of expression is now possible
+* `expression` for MultiBaseReader must be in form of `{asset}_b{index}`
+
+    ```python
+    # before
+    with STACReader("stac.json") as stac:
+        stac.tile(701, 102, 8, expression="green/red")
+
+    # now
+    with STACReader("stac.json") as stac:
+        stac.tile(701, 102, 8, expression="green_b1/red_b1")
+    ```
+
+* `rio_tiler.reader.point()` (and all Reader's point methods) now return a **rio_tiler.models.PointData** object
+
+    ```python
+    # before
+    with rasterio.open("cog.tif") as src::
+        v = rio_tiler.reader.point(10.20, -42.0)
+        print(v)
+        >>> [0, 0, 0]
+
+    with COGReader("cog.tif") as cog:
+        print(cog.point(10.20, -42.0))
+        >>> [0, 0, 0]
+
+    # now
+    with rasterio.open("cog.tif") as src::
+        v = rio_tiler.reader.point(src, (10.20, -42))
+        print(v)
+        >>> PointData(
+            data=array([3744], dtype=uint16),
+            mask=array([255], dtype=uint8),
+            band_names=['b1'],
+            coordinates=(10.20, -42),
+            crs=CRS.from_epsg(4326),
+            assets=['cog.tif'],
+            metadata={}
+        )
+
+    with Reader("cog.tif") as cog:
+        print(cog.point(10.20, -42.0))
+        >>> PointData(
+            data=array([3744], dtype=uint16),
+            mask=array([255], dtype=uint8),
+            band_names=['b1'],
+            coordinates=(10.20, -42),
+            crs=CRS.from_epsg(4326),
+            assets=['cog.tif'],
+            metadata={}
+        )
+    ```
+
+* deleted `rio_tiler.reader.preview` function and updated `rio_tiler.reader.read` to allow width/height/max_size options
+* reordered keyword options in all `rio_tiler.reader` function for consistency
+* removed `AlphaBandWarning` warning when automatically excluding alpha band from data
+* remove `nodata`, `unscale`, `resampling_method`, `vrt_options` and `post_process` options to `Reader` init method and replaced with `options`
+    ```python
+    # before
+    with COGReader("cog.tif", nodata=1, resampling_method="bilinear") as cog:
+        data = cog.preview()
+
+    # now
+    with Reader(COGEO, options={"nodata": 1, "resampling_method": "bilinear"}) as cog:
+        data = cog.preview()
+    ```
+
 # 3.1.6 (2022-07-22)
 
 * Hide `NotGeoreferencedWarning` warnings in `utils.render` and `utils.resize_array`

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -56,8 +56,9 @@ pdocs as_markdown \
    rio_tiler.expression \
    rio_tiler.models \
    rio_tiler.io.base \
-   rio_tiler.io.cogeo \
+   rio_tiler.io.rasterio \
    rio_tiler.io.stac \
+   rio_tiler.io.xarray \
    rio_tiler.mosaic.methods.base \
    rio_tiler.mosaic.methods.defaults \
    rio_tiler.mosaic.reader \

diff --git a/README.md b/README.md
@@ -47,26 +47,26 @@ data and metadata from any raster source supported by Rasterio/GDAL.
 This includes local and remote files via HTTP, AWS S3, Google Cloud Storage,
 etc.
 
-At the low level, `rio-tiler` is *just* a wrapper around the [rasterio.vrt.WarpedVRT](https://github.com/rasterio/rasterio/blob/5b76d05fb374e64602166d6cd880c38424fad39b/rasterio/vrt.py#L15) class, which can be useful for doing re-projection and/or property overriding (e.g nodata value).
+At the low level, `rio-tiler` is *just* a wrapper around the [rasterio](https://github.com/rasterio/rasterio) and [GDAL](https://github.com/osgeo/gdal) libraries.
 
 ## Features
 
 - Read any dataset supported by GDAL/Rasterio
 
     ```python
-    from rio_tiler.io import COGReader
+    from rio_tiler.io import Reader
 
-    with COGReader("my.tif") as image:
+    with Reader("my.tif") as image:
         print(image.dataset)  # rasterio opened dataset
         img = image.read()    # similar to rasterio.open("my.tif").read() but returns a rio_tiler.models.ImageData object
     ```
 
 - User friendly `tile`, `part`, `feature`, `point` reading methods
 
     ```python
-    from rio_tiler.io import COGReader
+    from rio_tiler.io import Reader
 
-    with COGReader("my.tif") as image:
+    with Reader("my.tif") as image:
         img = image.tile(x, y, z)            # read mercator tile z-x-y
         img = image.part(bbox)               # read the data intersecting a bounding box
         img = image.feature(geojson_feature) # read the data intersecting a geojson feature
@@ -76,9 +76,9 @@ At the low level, `rio-tiler` is *just* a wrapper around the [rasterio.vrt.Warpe
 - Enable property assignment (e.g nodata) on data reading
 
     ```python
-    from rio_tiler.io import COGReader
+    from rio_tiler.io import Reader
 
-    with COGReader("my.tif") as image:
+    with Reader("my.tif") as image:
         img = image.tile(x, y, z, nodata=-9999) # read mercator tile z-x-y
     ```
 
@@ -107,14 +107,46 @@ At the low level, `rio-tiler` is *just* a wrapper around the [rasterio.vrt.Warpe
         )
     ```
 
+- [Xarray](https://xarray.dev) support **(>=4.0)**
+
+    ```python
+    import xarray
+    from rio_tiler.io import XarrayReader
+
+    ds = xarray.open_dataset(
+        "https://pangeo.blob.core.windows.net/pangeo-public/daymet-rio-tiler/na-wgs84.zarr/",
+        engine="zarr",
+        decode_coords="all",
+        consolidated=True,
+    )
+    da = ds["tmax"]
+    with XarrayReader(da) as dst:
+        print(dst.info())
+        img = dst.tile(1, 1, 2)
+    ```
+    *Note: The XarrayReader needs optional dependencies to be installed `pip install rio-tiler["xarray"]`.*
+
+- Non-Geo Image support **(>=4.0)**
+
+    ```python
+    from rio_tiler.io import ImageReader
+
+    with ImageReader("image.jpeg") as src:
+        im = src.tile(0, 0, src.maxzoom)  # read top-left `tile`
+        im = src.part((0, 100, 100, 0))  # read top-left 100x100 pixels
+        pt = src.point(0, 0)  # read pixel value
+    ```
+
+    *Note: `ImageReader` is also compatible with proper geo-referenced raster datasets.*
+
 - [Mosaic](https://cogeotiff.github.io/rio-tiler/mosaic/) (merging or stacking)
 
     ```python
-    from rio_tiler.io import COGReader
+    from rio_tiler.io import Reader
     from rio_tiler.mosaic import mosaic_reader
 
     def reader(file, x, y, z, **kwargs):
-        with COGReader(file) as image:
+        with Reader(file) as image:
             return image.tile(x, y, z, **kwargs)
 
     img, assets = mosaic_reader(["image1.tif", "image2.tif"], reader, x, y, z)
@@ -124,11 +156,11 @@ At the low level, `rio-tiler` is *just* a wrapper around the [rasterio.vrt.Warpe
 
     ```python
     import morecantile
-    from rio_tiler.io import COGReader
+    from rio_tiler.io import Reader
 
     # Use EPSG:4326 (WGS84) grid
     wgs84_grid = morecantile.tms.get("WorldCRS84Quad")
-    with COGReader("my.tif", tms=wgs84_grid) as cog:
+    with Reader("my.tif", tms=wgs84_grid) as cog:
         img = cog.tile(1, 1, 1)
     ```
 

diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml
@@ -51,8 +51,9 @@ nav:
     - rio_tiler.models: api/rio_tiler/models.md
     - rio_tiler.io:
       - rio_tiler.io.base: api/rio_tiler/io/base.md
-      - rio_tiler.io.cogeo: api/rio_tiler/io/cogeo.md
+      - rio_tiler.io.rasterio: api/rio_tiler/io/rasterio.md
       - rio_tiler.io.stac: api/rio_tiler/io/stac.md
+      - rio_tiler.io.xarray: api/rio_tiler/io/xarray.md
     - rio_tiler.mosaic:
       - rio_tiler.mosaic.reader: api/rio_tiler/mosaic/reader.md
       - rio_tiler.mosaic.methods:
@@ -64,6 +65,7 @@ nav:
     - rio_tiler.utils: api/rio_tiler/utils.md
   - Migration from v1.0 to v2.0: 'v2_migration.md'
   - Migration from v2.0 to v3.0: 'v3_migration.md'
+  - Migration from v3.0 to v4.0: 'v4_migration.md'
   - Development - Contributing: 'contributing.md'
   - Release Notes: 'release-notes.md'