Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support joins from sqlite tables in tile-join #308

Draft
wants to merge 21 commits into
base: main
Choose a base branch
from
Draft

Support joins from sqlite tables in tile-join #308

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
5c57d5a
Sketching out sqlite options for tile-join
e-n-f Nov 4, 2024
f2d1490
Merge remote-tracking branch 'origin/main' into tile-join-sqlite
e-n-f Dec 2, 2024
a07aad9
Enable sqlite3 serialized multithreading
e-n-f Dec 2, 2024
36b207c
Fix some unnecessary round trips from std::string to char * and back
e-n-f Dec 3, 2024
0ae73c1
Gathering join keys for sql query
e-n-f Dec 4, 2024
eab74d4
Merge remote-tracking branch 'origin/main' into tile-join-sqlite
e-n-f Dec 5, 2024
264665b
Make a query
e-n-f Dec 5, 2024
ad47b93
Actually open the gpkg. Fix the query quoting.
e-n-f Dec 5, 2024
cbcdbc5
Actually do the query and get results back
e-n-f Dec 6, 2024
a4b8668
Join the attributes onto the feature
e-n-f Dec 6, 2024
c88f8c0
Set matched if the sql join matches
e-n-f Dec 17, 2024
03daaef
Add a flag to get the feature ID from the query
e-n-f Dec 17, 2024
fcc611a
Observe attribute exclusion when joining from sql queries
e-n-f Dec 17, 2024
638a761
Merge remote-tracking branch 'origin/main' into tile-join-sqlite
e-n-f Dec 17, 2024
5c8f154
Add a flag to exclude all attributes from the tile side of the join
e-n-f Dec 17, 2024
d6ce924
Make tile-join bounding boxes reflect feature bounds, not tile bounds
e-n-f Dec 20, 2024
acffe23
More tests
e-n-f Dec 20, 2024
7338c4b
An empty tileset has empty bounds at null island
e-n-f Dec 20, 2024
4788b67
Gather the results from each thread *after* the thread finishes
e-n-f Dec 20, 2024
632d031
Missed a test
e-n-f Dec 20, 2024
b87d8e4
Fix accidental inclusion of the top left of the tile in the bbox
e-n-f Dec 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions tests/feature-filter/out/filtered.json.standard
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "-180.000000,-85.051129,180.000000,85.051129",
"bounds": "-180.000000,-85.051129,180.000000,85.051129",
"antimeridian_adjusted_bounds": "-100.019531,0.000000,0.966797,0.966751",
"bounds": "-100.019531,0.000000,0.966797,0.966751",
"center": "0.000000,0.000000,0",
"description": "tests/feature-filter/out/all.mbtiles",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/feature-filter/out/places-filter.mbtiles.json.standard
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "-175.781250,-42.032974,180.000000,64.168107",
"bounds": "-175.781250,-42.032974,180.000000,64.168107",
"antimeridian_adjusted_bounds": "-175.220947,-41.302571,179.217224,64.168107",
"bounds": "-175.220947,-41.302571,179.217224,64.168107",
"center": "-62.578125,17.307462,8",
"description": "tests/feature-filter/out/places.mbtiles",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/join-population/concat.mbtiles.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "-180.000000,-85.051129,180.000000,85.051129",
"bounds": "-180.000000,-85.051129,180.000000,85.051129",
"antimeridian_adjusted_bounds": "-122.255859,37.718590,-122.080078,37.857507",
"bounds": "-122.255859,37.718590,-122.080078,37.857507",
"center": "-122.104097,37.695438,0",
"description": "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/join-population/joined-i.mbtiles.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "-122.343750,37.857507,-122.255859,37.926868",
"bounds": "-122.343750,37.857507,-122.255859,37.926868",
"antimeridian_adjusted_bounds": "-122.309418,37.881357,-122.280579,37.900865",
"bounds": "-122.309418,37.881357,-122.280579,37.900865",
"center": "-122.299805,37.892187,12",
"description": "tests/join-population/tabblock_06001420.mbtiles",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/join-population/joined-no-tile-stats.mbtiles.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "-122.343750,37.857507,-122.255859,37.926868",
"bounds": "-122.343750,37.857507,-122.255859,37.926868",
"antimeridian_adjusted_bounds": "-122.343750,37.874853,-122.280579,37.900865",
"bounds": "-122.343750,37.874853,-122.280579,37.900865",
"center": "-122.299805,37.892187,12",
"description": "tests/join-population/tabblock_06001420.mbtiles",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/join-population/joined-null.mbtiles.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "-122.343750,37.857507,-122.255859,37.926868",
"bounds": "-122.343750,37.857507,-122.255859,37.926868",
"antimeridian_adjusted_bounds": "-122.343750,37.874853,-122.280579,37.900865",
"bounds": "-122.343750,37.874853,-122.280579,37.900865",
"center": "-122.299805,37.892187,12",
"description": "tests/join-population/tabblock_06001420.mbtiles",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/join-population/joined-tile-stats-attributes-limit.mbtiles.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "-122.343750,37.857507,-122.255859,37.926868",
"bounds": "-122.343750,37.857507,-122.255859,37.926868",
"antimeridian_adjusted_bounds": "-122.343750,37.874853,-122.280579,37.900865",
"bounds": "-122.343750,37.874853,-122.280579,37.900865",
"center": "-122.299805,37.892187,12",
"description": "tests/join-population/tabblock_06001420.mbtiles",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/join-population/joined-tile-stats-sample-values-limit.mbtiles.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "-122.343750,37.857507,-122.255859,37.926868",
"bounds": "-122.343750,37.857507,-122.255859,37.926868",
"antimeridian_adjusted_bounds": "-122.343750,37.874853,-122.280579,37.900865",
"bounds": "-122.343750,37.874853,-122.280579,37.900865",
"center": "-122.299805,37.892187,12",
"description": "tests/join-population/tabblock_06001420.mbtiles",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/join-population/joined-tile-stats-values-limit.mbtiles.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "-122.343750,37.857507,-122.255859,37.926868",
"bounds": "-122.343750,37.857507,-122.255859,37.926868",
"antimeridian_adjusted_bounds": "-122.343750,37.874853,-122.280579,37.900865",
"bounds": "-122.343750,37.874853,-122.280579,37.900865",
"center": "-122.299805,37.892187,12",
"description": "tests/join-population/tabblock_06001420.mbtiles",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/join-population/joined.mbtiles.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "-122.343750,37.857507,-122.255859,37.926868",
"bounds": "-122.343750,37.857507,-122.255859,37.926868",
"antimeridian_adjusted_bounds": "-122.343750,37.874853,-122.280579,37.900865",
"bounds": "-122.343750,37.874853,-122.280579,37.900865",
"center": "-122.299805,37.892187,12",
"description": "tests/join-population/tabblock_06001420.mbtiles",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/join-population/just-macarthur.mbtiles.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "237.656250,37.857507,237.744141,37.926868",
"antimeridian_adjusted_bounds": "-122.294655,37.694688,-122.103424,37.833649",
"attribution": "macarthur's attribution",
"bounds": "-122.343750,37.695438,-122.104097,37.926868",
"bounds": "-122.294655,37.694688,-122.103424,37.833649",
"center": "-122.299805,37.892187,11",
"description": "macarthur description",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/join-population/macarthur-6-9-exclude.mbtiles.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "-122.343750,37.439974,-121.992188,37.996163",
"bounds": "-122.343750,37.439974,-121.992188,37.996163",
"antimeridian_adjusted_bounds": "-122.294655,37.695231,-122.103424,37.833107",
"bounds": "-122.294655,37.695231,-122.103424,37.833107",
"center": "-122.167969,37.833010,9",
"description": "tests/join-population/macarthur.mbtiles",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/join-population/macarthur-6-9.mbtiles.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "-122.343750,37.439974,-121.992188,37.996163",
"bounds": "-122.343750,37.439974,-121.992188,37.996163",
"antimeridian_adjusted_bounds": "-122.294655,37.695231,-122.103424,37.833107",
"bounds": "-122.294655,37.695231,-122.103424,37.833107",
"center": "-122.167969,37.833010,9",
"description": "tests/join-population/macarthur.mbtiles",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/join-population/merged-folder.mbtiles.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "237.656250,37.857507,237.744141,37.926868",
"bounds": "-122.343750,37.695438,-122.104097,37.926868",
"antimeridian_adjusted_bounds": "-122.343750,37.694688,-122.103424,37.900865",
"bounds": "-122.343750,37.694688,-122.103424,37.900865",
"center": "-122.299805,37.892187,12",
"description": "tests/join-population/tabblock_06001420-folder",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/join-population/merged.mbtiles.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "237.656250,37.857507,237.744141,37.926868",
"bounds": "-122.343750,37.695438,-122.104097,37.926868",
"antimeridian_adjusted_bounds": "-122.343750,37.694688,-122.103424,37.900865",
"bounds": "-122.343750,37.694688,-122.103424,37.900865",
"center": "-122.299805,37.892187,12",
"description": "tests/join-population/tabblock_06001420.mbtiles",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/join-population/no-macarthur.mbtiles.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "237.656250,37.857507,237.744141,37.926868",
"bounds": "-122.343750,37.695438,-122.104097,37.926868",
"antimeridian_adjusted_bounds": "-122.343750,37.874853,-122.280579,37.900865",
"bounds": "-122.343750,37.874853,-122.280579,37.900865",
"center": "-122.299805,37.892187,12",
"description": "tests/join-population/tabblock_06001420.mbtiles",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/join-population/raw-merged-folder.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "237.656250,37.857507,237.744141,37.926868",
"bounds": "-122.343750,37.695438,-122.104097,37.926868",
"antimeridian_adjusted_bounds": "-122.343750,37.694688,-122.103424,37.900865",
"bounds": "-122.343750,37.694688,-122.103424,37.900865",
"center": "-122.299805,37.892187,12",
"description": "tests/join-population/tabblock_06001420.mbtiles",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/join-population/renamed.mbtiles.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "-122.343750,37.439974,-121.992188,37.996163",
"bounds": "-122.343750,37.439974,-121.992188,37.996163",
"antimeridian_adjusted_bounds": "-122.294655,37.694688,-122.103424,37.833649",
"bounds": "-122.294655,37.694688,-122.103424,37.833649",
"center": "-122.167969,37.828608,10",
"description": "tests/join-population/macarthur2.mbtiles",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/join-population/windows.mbtiles.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "-122.343750,37.439974,-121.992188,37.996163",
"bounds": "-122.343750,37.439974,-121.992188,37.996163",
"antimeridian_adjusted_bounds": "-122.294655,37.694688,-122.103424,37.833649",
"bounds": "-122.294655,37.694688,-122.103424,37.833649",
"center": "-122.167969,37.833010,10",
"description": "tests/join-population/macarthur.mbtiles",
"format": "pbf",
Expand Down
4 changes: 2 additions & 2 deletions tests/raw-tiles/raw-tiles-z67-join.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ "type": "FeatureCollection", "properties": {
"antimeridian_adjusted_bounds": "-123.750000,45.089036,-120.937500,47.040182",
"bounds": "-123.750000,45.089036,-120.937500,47.040182",
"antimeridian_adjusted_bounds": "-122.682953,45.512121,-122.654800,45.569832",
"bounds": "-122.682953,45.512121,-122.654800,45.569832",
"center": "-122.662354,45.514045,7",
"description": "tests/raw-tiles/raw-tiles",
"format": "pbf",
Expand Down
Loading
Loading