Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs(blog): ibis, duckdb and lonboard for overture maps #10143

Closed
wants to merge 107 commits into from
Closed

docs(blog): ibis, duckdb and lonboard for overture maps #10143

wants to merge 107 commits into from

Conversation

ncclementi
Copy link
Contributor

This is still a draft, there are a few comments and things to decide

  • Unfortunately there is a bug in the new duckdb 1.1. release where writing to_parquet geoparquet files, there is a PR open to fix this Fix data race when writing GeoParquet duckdb/duckdb#13962, not sure if there will be a bug fix release or if we will encourage downloading the nightly when it goes out.
  • Unfortunately I had to use screenshots of the visualizations otherwise the html file gets huge to host.
  • @kylebarron I know it's been a long time since the original issue, if you are still interested in co-authoring there are couple of places where we can fit in some lonboard content.

@ncclementi ncclementi added the docs-preview Add this label to trigger a docs preview label Sep 16, 2024
@github-actions GitHub Actions
Copy link
Contributor

ACTION NEEDED

Ibis follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message.

Please update your PR title and description to match the specification.

@ibis-docs-bot ibis-docs-bot bot removed the docs-preview Add this label to trigger a docs preview label Sep 16, 2024
@ncclementi ncclementi changed the title docs(blog): Ibis, duckdb and lonboard for overture maps docs(blog): ibis, duckdb and lonboard for overture maps Sep 16, 2024
@Maxxen
Copy link

Maxxen commented Sep 16, 2024

Bug fix release is scheduled for monday if everything goes as planned!

@ncclementi
Copy link
Contributor Author

Thanks @Maxxen that would be awesome. If you are interested in being part of the blog too, I'd be happy to get include a paragraph or something related to the duckdb geoparquet support that you added and list you as a co-author.

At the moment is a very rough draft, but I'll leave the invitation open in case you are interested.

kylebarron
kylebarron reviewed Sep 16, 2024
View reviewed changes
docs/posts/ibis-overturemaps/index.qmd Outdated Show resolved Hide resolved
docs/posts/ibis-overturemaps/index.qmd Outdated Show resolved Hide resolved
docs/posts/ibis-overturemaps/index.qmd Outdated Show resolved Hide resolved
docs/posts/ibis-overturemaps/index.qmd
```

## Motivation

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd start with a sentence intro about what Overture Maps is. Like

Overture Maps is a project to build map data products on top of a variety of sources, like OpenStreetMap.

(not sure that's the best one-liner description of what Overture Maps is, any zingers @jwass ?)

docs/posts/ibis-overturemaps/index.qmd Outdated Show resolved Hide resolved
docs/posts/ibis-overturemaps/index.qmd Outdated Show resolved Hide resolved
docs/posts/ibis-overturemaps/index.qmd Outdated Show resolved Hide resolved
docs/posts/ibis-overturemaps/index.qmd Outdated
Comment on lines 163 to 164
plants_CA = plants.filter(_.bbox.xmin.between(-118.6, -117.9),
_.bbox.ymin.between(34.5, 35.3))[_.names.primary, _.geometry]
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I'd suggest using standard black/ruff code formatting for all Python code blocks

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, I'll fix this.

docs/posts/ibis-overturemaps/index.qmd Outdated
and that's how you can visualize ~7 million points from the comfort of
your laptop.

Note: I got the ~7M by adding the number of points in power_lines and minor_lines. I'm not sure if plotting the lines that connect these points add points to this.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is more effort for the GPU to render lines than to render points, but a line isn't rendered as a collection of points.

It might be clearer to say 7 million coordinates rather than points.

docs/posts/ibis-overturemaps/index.qmd Outdated Show resolved Hide resolved
@ncclementi
Copy link
Contributor Author

duckdb 1.1.1 is going out on Monday, I'll revisit changes, cleanup code and update text accordingly once the release is out.

ncclementi and others added 18 commits September 24, 2024 18:58
@ncclementi @cpcloud
fix(datafusion): raise when attempting to create temp table (ibis-pro…
f9b905b 
…ject#10072)

Co-authored-by: Phillip Cloud <[email protected]>
@jcrist @ncclementi
depr(selectors): deprecate c and r selectors in favor of cols a…
f64a7c6 
…nd `index`
@jcrist @ncclementi
depr(api): deprecate bool_val.negate()/-bool_val in favor of `~bo…
f329636 
…ol_val`
@cpcloud @ncclementi
test(backends): use slightly more useful assertions for call count
bd06b3f
@cpcloud @ncclementi
test(bigquery): make test non-strict xfail (ibis-project#10081)
d8b2510
@IndexSeek @cpcloud @ncclementi
feat(mssql): add lpad and rpad ops (ibis-project#10060)
9050b5c 
Co-authored-by: Phillip Cloud <[email protected]>
@IndexSeek @ncclementi
refactor(mssql): simplify lpad and rpad ops (ibis-project#10085)
bcee24f 
## Description of changes

Simplifying the previous implementation from `sge.Case(ifs...)` per
ibis-project#10060 (comment).
@jcrist @ncclementi
doc: add docstring example for topk
4ec6cac
@jcrist @ncclementi
feat(api): add name argument to topk
abf0863
@jcrist @ncclementi
feat(api): add name argument to value_counts
521958b
@IndexSeek @ncclementi
test(polars): replace register with create_table
0ae44aa
@IndexSeek @ncclementi
fix(docs): update invalid read_parquet link
2b22433
@cpcloud @ncclementi
perf(backends): speed up most memtable existence checks (ibis-project…
20e0bb7 
…#10067)
@cpcloud @gmcrocetti @ncclementi
ci(backends): run backend doctests in CI (ibis-project#9970)
11fb399 
Co-authored-by: Guilherme Martins Crocetti <[email protected]>
@gforsyth @ncclementi
revert: fix(datafusion): raise when attempting to create temp table (i…
c3a7481 
…bis-project#10072) (ibis-project#10099)
@cpcloud @ncclementi
ci: test examples (ibis-project#10098)
39ebbb5
@semantic-release-bot @ncclementi
chore(release): 9.5.0
a507754
@jcrist @ncclementi
refactor(sql): simplify paren handling for binary ops
c1fa0b5
@github-actions github-actions bot added docs Documentation related issues or PRs tests Issues or PRs related to tests impala The Apache Impala backend postgres The PostgreSQL backend sqlite The SQLite backend ci Continuous Integration issues or PRs clickhouse The ClickHouse backend mysql The MySQL backend pyspark The Apache PySpark backend datafusion The Apache DataFusion backend dependencies Issues or PRs related to dependencies bigquery The BigQuery backend datatypes Issues relating to ibis's datatypes (under `ibis.expr.datatypes`) duckdb The DuckDB backend polars The polars backend snowflake The Snowflake backend mssql The Microsoft SQL Server backend nix Issues or PRs related to nix trino The Trino backend druid The Apache Druid backend oracle The Oracle backend flink Issues or PRs related to Flink exasol Issues related to the exasol backend risingwave The RisingWave backend labels Sep 24, 2024
@ncclementi
Copy link
Contributor Author

Oh no, I have no clue what git shenanigans happen that this got so messed up. I will close this in favor of a new one.

@ncclementi
Copy link
Contributor Author

closing in favor of #10215

Apologies for the mess, no clue what happened.

@ncclementi ncclementi closed this Sep 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kylebarron kylebarron kylebarron left review comments

@gforsyth gforsyth gforsyth left review comments

@cpcloud cpcloud cpcloud left review comments

Assignees
No one assigned
Labels
bigquery The BigQuery backend ci Continuous Integration issues or PRs clickhouse The ClickHouse backend datafusion The Apache DataFusion backend datatypes Issues relating to ibis's datatypes (under `ibis.expr.datatypes`) dependencies Issues or PRs related to dependencies docs Documentation related issues or PRs druid The Apache Druid backend duckdb The DuckDB backend exasol Issues related to the exasol backend flink Issues or PRs related to Flink impala The Apache Impala backend mssql The Microsoft SQL Server backend mysql The MySQL backend nix Issues or PRs related to nix oracle The Oracle backend polars The polars backend postgres The PostgreSQL backend pyspark The Apache PySpark backend risingwave The RisingWave backend snowflake The Snowflake backend sqlite The SQLite backend tests Issues or PRs related to tests trino The Trino backend
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

docs(blog/how to): Ibis + duckdb spatial + geoparquet + lonboard