How to use DuckDbClient with Arrow table (not file)?

[davisford]

The docs for DuckDbClient always reference taking a FileAttachment https://observablehq.com/framework/lib/duckdb but it says it can be expressed as an arrow table:

To get a DuckDB client, pass zero or more named tables to DuckDBClient.of. Each table can be expressed as a FileAttachment, Arquero table, Arrow table, an array of objects, or a promise to the same.

We use a basic fetch to grab the arrow data from our backend... example:

    const response = await fetch(`${baseUrl}/org/1/data/${report}?startDate=${startDate}&endDate=${endDate}`);
    const data = await response.arrayBuffer();
    const table = await arrow.tableFromIPC(fetchData); 

    // i now have an in-memory arrow table but this does not work... ?
   const db = DuckCbClient.of({table: table})

How can I leverage the db client in this way?

[mbostock]

There are several inconsistencies in your code snippet that could be causing the problem:

• Did you mean data instead of fetchData?
• Did you mean DuckDBClient instead of DuckCbClient?
• Did you mean Arrow instead of arrow (for the implicit import of npm:apache-arrow)?
• Did you need to await the result of DuckDBClient.of, since it returns a Promise?
• The name table is reserved, so you’ll need to use 'table' to query it.

Here’s a complete snippet that works for me:

```js
const response = await fetch("https://observablehq.com/framework/_file/lib/flights-200k.eedcb010.arrow");
const buffer = await response.arrayBuffer();
const table = await Arrow.tableFromIPC(buffer);
const db = await DuckDBClient.of({flights: table});
```

You can then query your database and display the results like so:

```js
const results = await db.sql`SELECT * FROM flights`;
display(Inputs.table(results));
```

[davisford]

Hi @mbostock - indeed there are typos. I suffered an injury a couple months ago and lost the use of one arm so typing with one hand now isn't ideal. It's great news that presumably it should work directly with the arrow table, When testing it now, and using flights for the table name (assuming the name is arbitrary if not using a reserved word like table as you pointed out) - it generates the following error

const db = await DuckDBClient.of({flights: table});
const results = await db.sql`SELECT * FROM flights`;

Error: Catalog Error: Table with name flights does not exist!
Did you mean "sqlite_schema"?
LINE 1: SELECT * FROM 'flights'

so we are looking at the structural diff between our arrow table and the flight data loaded from that url may cause such an issue. i appreciate your prompt response and all the great work you do on this and d3!

[mbostock]

That error suggests that the value of table in your code maybe isn’t a valid Arrow table object. I’d consider using a debugger and seeing what the value of table is that you’re passing into DuckDBClient.of.

One possible cause is loading an old version of Apache Arrow instead of using the built-in version that Framework provides as an implicit import (Arrow); Framework’s DuckDB client requires Apache Arrow 7.0.0 or later.

framework/src/client/stdlib/duckdb.js

Lines 388 to 400 in 82412a4

	// Returns true if the value is an Apache Arrow table. This uses a “duck” test
	// (instead of strict instanceof) because we want it to work with a range of
	// Apache Arrow versions at least 7.0.0 or above.
	// https://arrow.apache.org/docs/7.0/js/classes/Arrow_dom.Table.html
	function isArrowTable(value) {
	return (
	value &&
	typeof value.getChild === "function" &&
	typeof value.toArray === "function" &&
	value.schema &&
	Array.isArray(value.schema.fields)
	);
	}

[davisford]

We discovered the issue. It was the difference between this:

import { tableFromIPC } from "apache-arrow";
const table = await tableFromIPC(buffer);
// does not work

vs this

const table = await Arrow.tableFromIPC(buffer);
// works

Unsure why, but using the static function from the Arrow class works. We introspected the table object in devtools and nothing in particular stuck out. npm dependencies:

├── @easepick/[email protected]
├── @easepick/[email protected]
├── @observablehq/[email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
└── [email protected]

[Fil]

You could try with import { tableFromIPC } from "npm:apache-arrow"; to see if loading Arrow from npm works better than loading it from your local modules.