Filter cells by region #66
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Skipped Deployment
|
BielStela
force-pushed
the
tile-subsets
branch
from
d1bed9b to
6b9bd42
Compare
alexeh
approved these changes
Sep 26, 2024
mbarrenechea
added a commit
that referenced
this pull request
Nov 5, 2024
* sin up dev instance * point to dev host in cd * Grid prototype (#54) * Add router for h3 tiler * Add router to main app and fix dependencies * WIP script for converting raster to tiles * script for base case * remove progress context manager * WIP compute overviews * WIP compute overviews * small refactor and clenup * Adds low memory filtering * Set loggin levl to INFO * Rename script and wip combinator for merging multiple tile sources to single folder * some improvements, still not there * Add metadata to tiled dataset creation and format to 100 columns * Improves dataset combination * Fix dtype conversion * Add tests for grid enpoint and fixes bad index handling * Fix test env vars in docker * Add metadata model and endpoint for it * add testito for meta endpoint and fix deprecation in model config * Testing (#55) * Header: tests * Card: loader and no data * Testing: Formats * Testing: husky * Testing: population * Images * Testing: utils * Fix auth in grid and update metadata model (#56) * Wopsie * Adds one for all test to check unauthorized accesses * adds tests and removes dead code * Make use of fastapi tools for auth (#57) * Makes use of fastapis tooling to check tokens instead of our own * Handle token errors in house * Add comment * Compact * Table endpoint (#59) * WIP table endpoint with filters * set to use polars instead of duckdb. it is faster * query builder and model for table filters json. * add a couple of tests for the query builder * add missing __init__ 🤦 * model tidy up models * fix tests to new model * add more test cases for the all the ops and changes order by to use '-' as desc indicator in column name * tidy up names and tests * remove duckdb from requirements * add test * set tests settings env vars in conftest * Add column filter to tile endpoint (#60) * query builder and model for table filters json. * add a couple of tests for the query builder * add missing __init__ 🤦 * fix tests to new model * add more test cases for the all the ops and changes order by to use '-' as desc indicator in column name * tidy up names and tests * remove duckdb from requirements * Adds columns query param to grid tile endpoint * fix column param default and tests * tidyup api documentation * set correct status code for incorrect columns in filters * amazonia360+ rewording * updates metamodel (#64) * add missing env var to compose setup * add env vars for testing flow * remove env vars from service for tests * Remove zoom limit * Search improvements * Search: fix reset * Fullstory * Fullstory * Filter cells by region and use hex representation of h3 cell id (#66) * update dataset to use hexadecimal representation of cell id * Add enpoint to get tiles filtered by geojson * Speedup request by caching geojson filling result and improve queries * removes all outputs from notebooks * Adds nbstrip to pre-commit * update deps and filter table by geojson * Adds test for table results with geojson filter * Tidy up the table response model * Improve response documentation in OpenAPI * Update compose call to use the compose subcommand in the ci runner * small comment * improve benchmark by using multiple geojson so we can check caching * Fix polars arrow string compatibility to use large string instead of string view so js implementation can parse it (#67) * Set h3 index type to `string` in IPC table (#69) * Notebook to convert grid csv to arrow files * Cast IPC table to string * Fix python docker image no longer ships with libexpat a requirement for fiona and gdal * remove test router * Add (again) the restart policy for docker compose. Upsidoo! * platform renamed to AmazoniaForever360+ --------- Co-authored-by: alexeh <[email protected]> Co-authored-by: Biel Stela <[email protected]> Co-authored-by: Miguel Barrenechea Sánchez <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new tile endpoint with the functionality of filtering the tile cells by the are of a polygon. This responds to the user need to only focus on a small subset of results.
Implementation details
To filter the tile cells we first need to create the list of cells that fill the polygon. To do so I'm using a cached function so the filling operation does not have to be repeated for each requested tile.
This PR also changes the number representation of the h3 cell ids to use hexadecimal since the client needs the data in hex and was doing the conversion. We do not really need it in int64 since we only operate with the index once, and it is in a cached function so any overhead will be dissipated quickly.