v0.12.1

Changelog

• Add waffle mark. (Example)
• Add selection activation to inputs.
• Fix hexbin query y-coordinate calculation.
• Fix examples for better dark mode viewing. (#600)

Full Changelog: v0.12.0...v0.12.1

v0.12.0

Highlights

• Overhaul SQL utilities for greater coverage and flexibility.
• Expand and rename preaggregation optimizations.
• Add vgplot region interactor for point-typed screen-space selections. (Example)
• Add support for Param-valued table references. (Example)
• Add support for Param-valued column references. By default, Param values map to literals within SQL expressions. For example, a string-valued param will map to a SQL string literal. However, sometimes users want to dynamically reference a table column instead. This version adds functionality to interpret a param as a SQL column reference, using either the column transform method (e.g., vg.column($paramName) or a special $$ syntax within declarative spec SQL expression strings (e.g., { sql: $$paramName ** 2 }. (Example)

Changelog

• Breaking: Rewrite mosaic-sql package to use a complete AST formulation. Add additional SQL helper methods.
• Breaking: Drop agg SQL expression helper. We now perform automatic analysis to determine if an expression is an aggregate expression.
• Breaking: Remove automatic internal casts, such as casting count() aggregates to 32-bit integers. As we dropped Arrow-JS in favor of Flechette, Arrow parsing now gracefully handles 64-bit integers, decimals, and other type conversions.
• Add more SQL operators, math functions, isIn function, conditionals, and more.
• Move SQL transforms for binning and M4 to mosaic-sql package in a new transforms folder.
• Update plots, inputs, and spec parsing to use new SQL utilities. SQL expressions no longer provide column dependency lists, param listeners, or idiosyncratic annotations. Instead, needed metadata is now extracted using visitor methods that walk the SQL expression AST. All query generator methods now create full, analyzable ASTs using our updated SQL builders.
• Add Param-value support for from table names in vgplot marks and table views. This change permits dynamic swapping of backing data tables at runtime, including among differently sized samples.
• Add column param node type to SQL AST helpers.
• Add column transform to declarative spec language.
• Add $$param syntax to declarative spec SQL expressions to indicate a column-valued param.
• Breaking: Rename coordinator dataCubeIndexer to preaggregator.
• Breaking: Rename coordinator indexes option key to preagg.
• Breaking: Rename pre-aggregation index method to request.
• Breaking: Rename pre-aggregation dropIndexTables method to dropPreAggregateTables.
• Rename other internal methods and comments to use the terminology of "pre-aggregation" and "pre-aggregated materialized views".
• Update documentation to use the terminology of "pre-aggregation" and "pre-aggregated materialized views".
• Update pre-aggregation optimizations to support arbitrary aggregate expressions, not just individual aggregate function calls. We use the SQL facilities to rewrite expressions for more complex optimized aggregations.
• Update pre-aggregation optimizations to support aggregates with FILTER clauses.
• Refactor pre-aggregation optimizations, moving them to the new mosaic-core/preagg folder.
• Add region interactor for 2D selection of individual point values.
• Add use of SQL IN operator for single-value point selections.
• Add channels mark property to spec types.
• Fix empty selection resolution to use false literal.
• Fix interactor facet handling and datum extraction.
• Fix plot label generator handling of * character.
• Refactor BrushStyles types to dedicated file.
• Update facet handling to follow changes in Observable Plot.
• Add local test spec loading in dev runner.
• Update top-level Plot attributes: remove label, add clip, ariaLabel, ariaDescription
• Remove deprecated Coordinator query recorder support.
• Improve TypeScript types and tests in mosaic-sql and mosaic-core.
• Basic documentation updates.

v0.11.0

Changelog:

• Breaking: Replace apache-arrow with @uwdata/flechette to access Apache Arrow IPC data with better performance and smaller footprint. While the API for Flechette tables mirrors many aspects of the Arrow JS reference implementation, it is not a perfect one-to-one replacement. It is possible that some Mosaic users may need to adjust query result processing in response.
• Breaking: Redesign the data cube indexer to write index tables into a named schema (default 'mosaic'). This allows data cube index tables to be managed in a largely isolated environment and to be reused across users and sessions. Temporary index tables are no longer supported, as temporary tables can not be created within a named schema.
• Breaking: Remove temp option for data cube index tables.
• Breaking: Remove data cube indexer enabled method, instead use get/set properties.
• Add data cube indexer schema get/set properties.
• Add data cube indexer dropIndexTables() method. This method issues a query to remove the entire data cube index table schema. It should be called if base tables are updated, causing index tables to become stale and inaccurate.
• Add Selection include to define a Selection that includes the clauses from one or more upstream selections. This addition allows selections to be composed to build up larger clause sets from sub-selections.
• Add client initialize method. This method supports dynamic client re-initialization, issuing new field info and client query requests.
• Fix throttle utility to be resilient to callback rejections. (thanks @andyrooger!)
• Fix canceled queries by rejecting them, rather than leaving them pending. (thanks @andyrooger!)
• Fix index creation queries to no longer be canceled when clearing the data cube indexes. These indexes may already be used in other queued requests. (thanks @andyrooger!)
• Fix Plot class to re-initialize mark clients upon dynamic field updates.
• Fix data cube indexer to generate correct averages in variance and covariance calculations. (#487)
• Fix broken usage of json_format and add loadJSON test. (thanks @geoffreyd!)
• Fix management of cancelled queries, remove race condition. (thanks @andyrooger!)
• Fix query consolidation to flag aggregate queries with no groupby. (#479, thanks @andyrooger!)
• Fix variable access typo in DuckDBStatement within node.js DuckDB lib.
• Fix time interval year span calculation. (#484)
• Fix nearest interactor with param-compatible clause for single-field selections. (#502)
• Fix nearest interactor for consistent param and selection updates. (#508)
• Fix Coordinator queryRequest to always return a Promise. (#507)
• Update Jupyter widget, remove temp_indexes property and add dataCubeSchema property.
• Update types, method signatures in Coordinator and DataCubeIndexer.
• Update test runner from mocha to vitest. (thanks @domoritz!)
• Update Coordinator API documentation.
• Update athletes example to use Selection include. We now use an upstream selection to limit text search options to the current sport and sex.

v0.10.0

Changelog:

• Breaking: Update bin transform declarative spec to support top-level option properties, remove options object as second argument.
• Breaking: Remove coordinator configure method. It was confusing and involved interactions among settings.
• Breaking: Refactor clause generators for better clarity (clauseInterval instead of just interval).
• Add default client set construction to selection clause helpers.
• Add date interval support to bin transform. A new interval option specifies a desired date/time unit (hour, day, month, etc) or forces standard numerical binning (number). If not specified, a reasonable date/time interval and step size will be determined automatically for temporal data. (#427)
• Add dateBin SQL helper method.
• Add empty flag to selections. If true, selections with no clauses will not select any values. If false (the default), selections with no clauses select all data values.
• Add as selection option to Table input, for clauses corresponding to selected table row values.
• Add rLabel plot attribute.
• Add support for plot aspectRatio property when auto-calculating heights at the Mosaic level.
• Add groupby and flexible column reference support to hexbin mark queries.
• Add more flexible groupby handling in data cube index column analysis.
• Add pending() method to AsyncDispatch to track param/selection event processing.
• Fix plot legend to check for plot label attributes.
• Fix table offset reset upon external updates. (#363)
• Fix empty data handling in regression mark.
• Fix query error handling in coordinator and clients.
• Update data cube indexer to operate over selection-client pairs directly. Use a single indexer per coordinator. Remove coordinator-internal FilterGroup class.
• Update toDataColumns utility, move from mosaic-plot to mosaic-core. This method maps Apache Arrow data to native JS values.
• Update internal query result promise to a proper class.
• Update test cases for client and selection lifecycle methods. (h/t @matys1)
• Update python libs to enable uv via installer key. (thanks @peter-gy)
• Update dependencies, use DuckDB 1.0.0.

v0.9.0

Changelog:

Features:

• Breaking: Rename selection clause schema property to meta.
• Breaking: Change nearest interactor options to support multidimensional point selection and separate pointer targets from selected data channels.
• Add nearest interactor for selecting nearest 2D point.
• Add errorbarX and errorbarY marks for parametric confidence intervals over unaggregated data. The existing ruleX/Y marks should be used for pre-computed interval boundaries or custom aggregate expressions.
• Add mark-level select option for internal filtering of mark data, for example first or last points in a series or the current nearest mark to the pointer. Mark-level select simplifies creating various hover effects and annotations, though may query data in an unoptimized fashion, and does not interoperate with params or selections.
• Add covariance aggregate method.
• Add data cube support in cases where only some filtered clients are indexable.
• Add stddev, variance, covariance, and regression aggregates to data cube indexer.
• Add stddev, variance, covariance transforms in declarative specs.
• Add binning method hints for data cube indexer.
• Add selection clause generation methods (point, interval, etc).
• Add selection clause typings.
• Add Selection valueFor method to look up source-specific values.
• Add selection clause methods to inputs and interactors.
• Add value label to slider input.
• Add interval selection support to slider input.
• Add field option to inputs (menu, slider, search) to control the column referenced in selection clause predicates. (#406)
• Add toggleZ interactor directive to select line or area series using the z channel.
• Add QueryManager logQueries method to toggle logging of queries submitted to connector.
• Add FilterGroup index method to toggle data cube indexing.
• Add SQLExpression export to mosaic-sql package.
• Add Query.select column override support. Newer as values overwrite prior ones, and a null expression value removes any prior statements.

Examples:

• Add linear-regression-10m example with dynamic regression calculation over 10m points.
• Add athlete-height example with error bars and slider-based selection.
• Add line-multi-series example with interactive hover highlights and annotations.

Developer Support:

• Add options for logging and optimization to the dev (npm run dev) example runner.
• Add JSON schema publishing (e.g., to https://uwdata.github.io/mosaic/schema/latest.json).
• Add project vscode settings for YAML schemas (requires the YAML extension)

Bug Fixes, etc.:

• Fix bin transform to support extents for rule marks.
• Fix data type lookup in plot renderer.
• Fix SQL generation escaping of strings with single quotes.
• Fix Menu value initialization and updates. (#401)
• Fix legend interactor peers and selection value extraction.
• Fix interval 1D brush extent sort order.
• Fix raster mark color domain sorting.
• Fix data lookup from SVG in toggle and highlight interactors to support lines/areas.
• Update dependencies, documentation, and types.

v0.8.0

Changelog:

• Add use of columnar data throughout vgplot internals.
• Add interactive brushing for continuous color and opacity legends.
• Add meta-key input support to toggle interactor.
• Add symbol scale directives to vgplot.
• Add legends test example spec.
• Add preamble option to astToESM() to insert custom code.
• Add data cube index support for ARG_MIN, ARG_MAX aggregates.
• Add data cube index support for PRODUCT, BIT_AND, BIT_OR, BIT_XOR, BOOL_AND, BOOL_OR aggregates.
• Add xLabelArrow, yLabelArrow plot attributes.
• Add (x|y|color|opacity|r|length)Percent plot attributes.
• Remove invalid fxNice and fyNice plot attributes.
• Fix legend sizing based on the presence of a label (legend title).
• Fix regression mark tip option to pass to error band only, for a single informative tooltip.
• Fix plot directive to call update, which is needed for plots without marks.
• Fix raster mark transient scale domain updates to get appropriate domain definitions.
• Fix contour mark fill/stroke/opacity encoding channel generation for density thresholds.
• Fix hspace/vspace codegen to properly stringify values.
• Fix namedPlots.clear() to also clear queued requests.
• Fix proper context use in AST node instantiate() calls.
• Fix JSON DuckDB type handling for field info.
• Refactor handleParam mark utility for better clarity and type-friendliness.
• Update dev example runner.
• Update some jsdoc typings throughout Mosaic packages.
• Add and publish TypeScript types for declarative JSON specs.
• Add and publish JSON schema for declarative JSON specs.
• Add test TypeScript files for all specs, run tsc on them as part of @uwdata/mosaic-spec tests.
• Add JSON schema validation to spec tests.
• Update dependencies, including latest version of TypeScript.

v0.7.1

Changelog

• Fix astToDOM() to respect pre-defined Params/Selections passed via the params options.
• Add tests for arrow value conversions.
• Bump dependencies.

v0.7.0

Changelog:

• Add Query.describe(query) to generate DESCRIBE ... queries for metadata.
• Add support for arbitrary SQL expressions (not just column names) as field info metadata. Passing direct SQL expressions to vgplot marks such as {x: vg.sql`-ra`} should now work!
• Add support to (lossily) convert Apache Arrow DECIMAL values to JavaScript Number values.
• Update query consolidator to handle DESCRIBE queries, which require filtering rather than projection.
• Remove table catalog from Coordinator, rely on cache and consolidation instead.
• Refactor fieldInfo metadata management within vgplot Mark implementations.
• Refactor Apache Arrow utilities to be exports of @uwdata/mosaic-core.
• Update dependencies.

As a result of the above, the catalog option of Coordinator.clear is no longer supported. Clients can still pass the option, but it will have no effect. Instead, clearing the cache will now also drop any cached table metadata.

v0.6.1

Changelog:

• Add raster mark interpolation options nearest, barycentric, and random-walk.
• Add initial plot auto-height support, fixes case of raster dimensions without a specific plot height.
• Add aeromagnetic survey example to test raster interpolation and smoothing.
• Update example preparation script, consolidate processing for both tests and docs.
• Fix error reduction in initialization of Deriche convolution.
• Bump dependencies.

v0.6.0

This release adds support for non-linear scales across binning methods, both in gridded marks like raster, and for the bin transform. It also improves marks that use gridded data, such as density, raster, and the new heatmap convenience mark.

Breaking Changes:

This release renames some of the options for density/contour/raster marks and adjusts their defaults. This simplifies some naming and usage, and better aligns with the underlying Observable Plot library.

• binType is now interpolate. The "normal" option value is now "none".
• binWidth is now pixelSize.
• binsX is now width.
• binsY is now height.

Changelog:

• Breaking: Rename options for density/contour/raster marks, change defaults.
• Add support for non-linear scales (log, symlog, sqrt) to pixel-level binning methods.
• Add support for non-linear scales (log, symlog, sqrt) to the bin transform.
• Add SQL scale transform utilities to the mosaic-sql package.
• Add richer option support to raster mark, including gridded ordinal data alongside numeric data.
• Add heatmap mark, a raster mark with convenient defaults for density estimation.
• Add additional plot attributes for scale transforms.
• Add more SQL transforms to the declarative specs.
• Update mark implementations to use improved binning.
• Update density1d example spec to use a log-scaled domain.
• Update examples, including new Observable latency example.
• Update documentation.
• Update dependencies, use DuckDB v0.10.0.
• Fix range transformation bug in DataCubeIndexer.
• Fix filter range bug in DensityLineMark.
• Fix M4 optimization bug in ConnectedMark (line and area).