Enables Schemas in DuckDB

[rlancer]

Fixes issue where using a DuckDB database with multiple schemas breaks

Current result when running with a DB that has multiple schemas:

[jonmmease]

Thanks for the fix @rlancer! I'd like to work out how to test this for the single and multi schema case (and I'm happy to work on adding tests). Do you know of an easy way to construct a DuckDB connection with multiple schemas?

[jonmmease]

Looks like the tests that use a single schema are failing: https://github.com/hex-inc/vegafusion/actions/runs/8179198596/job/22368156069?pr=470

[rlancer]

@jonmmease thank you! Will look at getting the tests worked out also noticing the same issue the tests are highlighting

[jonmmease]

Hi @rlancer, anything I can do to help get this over the finish line? Do you know of a self-contained way to construct a DuckDB connection with multiple schemas that triggers this error?

[rlancer]

Hi @rlancer, anything I can do to help get this over the finish line? Do you know of a self-contained way to construct a DuckDB connection with multiple schemas that triggers this error?

Any connection to MotherDuck should trigger the error

As far as fixing it I think we would need to change how table paths are resolved ie table://movies to support schemas

[jonmmease]

Any connection to MotherDuck should trigger the error

I don't have a MotherDuck account, and I wouldn't want to introduce this as a dependency for testing. I was hoping there would be a way to create this situation with a local DuckDB connection.

[rlancer]

Any connection to MotherDuck should trigger the error

I don't have a MotherDuck account, and I wouldn't want to introduce this as a dependency for testing. I was hoping there would be a way to create this situation with a local DuckDB connection.

Sure I'll do a repo with just DuckDB

[rlancer]

@jonmmease this will repo the issue using only DuckDB:

import vegafusion as vf
import pandas as pd
import altair as alt
import duckdb

conn = duckdb.connect()
conn.sql("create or replace schema schema1; create or replace table schema1.table1 (a int, b int);")

vf.runtime.set_connection(conn)

[jonmmease]

this will repo the issue using only DuckDB:

perfect!

[jonmmease]

Looking at this a bit, and I think the approach of referencing tables by URL with the tables:// is going to be a bit messy in the multi-schema case.

The newer alternative, that I haven't documented yet, is to wrap an individual DuckDBPyRelation object in a VegaFusion DuckDbDataset wrapper, then pass this to Altair as if it were a pandas DataFrame. And also enable the "vegafusion" data transformer in Altair.

It looks like this approach works fine with the schema case:

For example:

import vegafusion as vf
from vegafusion.dataset.duckdb import DuckDbDataset
import pandas as pd
import altair as alt
import duckdb

conn = duckdb.connect()
conn.sql("""
create or replace schema schema1;
create or replace table schema1.table1 (a int, b int);
INSERT INTO schema1.table1 VALUES (1, 5);
INSERT INTO schema1.table1 VALUES (2, 8);
""")

rel = conn.sql("select * from schema1.table1")

# Create VegaFusion duckdb dataset with verbose=True so the query gets printed out
ds = DuckDbDataset(rel, verbose=True)

alt.data_transformers.enable("vegafusion")

alt.Chart(DuckDbDataset(rel)).mark_bar().encode(x="a", y="b")

DuckDB Query:
WITH tbl_0 AS (SELECT "a", "b" FROM "tbl"), tbl_1 AS (SELECT row_number() OVER () AS "_vf_order", * FROM tbl_0), tbl_2 AS (SELECT "_vf_order" AS "_vf_order", "a" AS "a", "b" AS "b" FROM tbl_1), tbl_3 AS (SELECT count(0) AS "__count", min("_vf_order") AS "_vf_order", "a" FROM tbl_2 GROUP BY "a"), tbl_4 AS (SELECT "_vf_order", "a", "__count" FROM tbl_3), tbl_5 AS (SELECT * FROM tbl_4 WHERE coalesce("a" IS NOT NULL, false)), tbl_6 AS (SELECT "_vf_order", "__count", "a" FROM tbl_5), tbl_7 AS (SELECT * FROM tbl_6 ORDER BY "_vf_order" ASC NULLS LAST) SELECT "__count", "a" FROM tbl_7

See if this approach works for you

[rlancer]

Thank you, that does solve it!

[jonmmease]

Thank you, that does solve it!

Awesome. I'm thinking long term we may want to deprecate the special "table://" syntax for referencing a table in a connection since this doesn't scale to multiple schema.