Hooks to respond to changes in underlying data

[willium]

We currently use Mosaic's selections for interaction, but we also allow users to change the query defining the underlying view (synced with a remote data warehouse). This approach is necessary because it is infeasible for us to load all the data into the browser.

There is no clear way to trigger Mosaic to refresh the coordinator, indices, and clients.

Proposal

1. Short Term: Expose Endpoints for Resetting Indices:

   • Provide a mechanism to manually notify the coordinator of data changes, prompting it to refresh indices and update all registered clients.

2. Long Term: Support for Live Data Updates:

   • Implement hooks that listen to changes in the underlying DuckDB data, allowing for automatic updates to indices and clients.

3. Stretch Goal: Support Calculating Data Differences:

   • Implement a method to calculate a 'diff' between local data and a remote table, requesting only the necessary remote data as needed.

[domoritz]

Just to note that you can already manually run queries against the database to clear indexes etc.

You will also want to clear the caches on the server and browser (ideally not wiping it entirely but selectively flush only what is not consistent anymore).

[willium]

What is the best way to run queries such that indices are reset?

I think adding this is and marrying it to selections to a degree would be really nice. In practice that would allow for progressive visualization from a backend, and not needing to redraw points intermittently.

[jheer]

If you are updating the backing table that is being queried, you need to reset both data cube indexes and (in general) also Mosaic client stats gathered via field info.

To reset the clients, you can disconnect and reconnect them all:

const clients = Array.from(coordinator.clients); // extract current client set
coordinator.clear(); // disconnect clients and clear query cache
clients.forEach(client => coordinator.connect(client)); // reconnect clients, pulling new field info

For the data cubes, all Mosaic data cube indexes include the same table prefix (cube_index_), so you could drop all such tables via a custom exec query. There is not really a better way at present, as the Mosaic coordinator does not necessarily know what all the existing indexes are. For example, you could generate them in a previous session and reuse them later. If more precision is required (e.g., dropping some but not all indexes), we might need to evolve the naming scheme to incorporate the source table/view.

Hope that helps! If the above is not what you're asking about, please share more specifics of your use case, so we know a bit more precisely what is changing and what is not.

[willium]

Thanks Jeff - that makes sense. I'll give this is a whirl.

[jaanli]

Mind sharing any code for this @willium ? We need the same feature :)

[willium]

[jaanli]

We @onefact need this as well @willium ! Specifically, to work with https://openvoicenetwork.org/interoperability-initiative/ and re-validation of the data every time a transcript changes (e.g. using https://github.com/ggerganov/whisper.cpp/tree/master/examples/whisper.wasm).