v0.11.0-rc11

[jonmmease]

Release candidate for 0.11.0-rc11

This version introduces a brand new mime-type based renderer that is compatible with many notebook and compute environments without requiring custom extensions. It also includes a new transformed_data() function that makes it possible to evaluate the data transforms associated with an Altair Chart and return the result as a pandas DataFrame. Finally, support was added for several additional Vega transforms including pivot and impute.

Installation

To try out the release candidate:

pip install vegafusion-python-embed==0.11.0rc11 vegafusion==0.11.0rc11 vegafusion-jupyter==0.11.0rc11 vl-convert-python==0.6.0

---

This discussion was created from the release v0.11.0-rc11.

[jonmmease]

Here is some of the draft documentation for the website that will be updated along with the release of 0.11.0.

VegaFusion

VegaFusion provides serverside scaling for the Vega visualization library. While not limited to Python, an initial application of VegaFusion is the scaling of the Altair Python interface to Vega-Lite.

The core VegaFusion algorithms are implemented in Rust. Python integration is provided using PyO3 and JavaScript integration is provided using wasm-bindgen.

Quickstart 1: Overcome MaxRowsError with VegaFusion

The VegaFusion mime renderer can be used to overcome the Altair MaxRowsError by performing data-intensive aggregations on the server and pruning unused columns from the source dataset.

Open a Jupyter notebook (either the classic notebook or a notebook inside JupyterLab), and create an Altair histogram of a 1 million row flights dataset

import pandas as pd
import altair as alt

flights = pd.read_parquet(
    "https://vegafusion-datasets.s3.amazonaws.com/vega/flights_1m.parquet"
)

delay_hist = alt.Chart(flights).mark_bar().encode(
    alt.X("delay", bin=alt.Bin(maxbins=30)),
    alt.Y("count()")
)
delay_hist

---------------------------------------------------------------------------
MaxRowsError                              Traceback (most recent call last)
...
MaxRowsError: The number of rows in your dataset is greater than the maximum allowed (5000). For information on how to plot larger datasets in Altair, see the documentation

This results in an Altair MaxRowsError, as by default Altair is configured to allow no more than 5,000 rows of data to be sent to the browser. This is a safety measure to avoid crashing the user's browser. The VegaFusion mime renderer can be used to overcome this limitation by performing data intensive transforms (e.g. filtering, binning, aggregation, etc.) in the Python kernel before the resulting data is sent to the web browser.

Run these two lines to import and enable the VegaFusion mime renderer

import vegafusion as vf
vf.enable_mime(mimetype="html")

Now the chart displays quickly without errors

delay_hist

Quickstart 2: Extract transformed data

By default, data transforms in an Altair chart (e.g. filtering, binning, aggregation, etc.) are performed by the Vega JavaScript library running in the browser. This has the advantage of making the charts produced by Altair fully standalone, not requiring access to a running Python kernel to render properly. But it has the disadvantage of making it difficult to access the transformed data (e.g. the histogram bin edges and count values) from Python. Since VegaFusion evaluates these transforms in the Python kernel, it's possible to access then from Python using the vegafusion.transformed_data() function.

For example, the following code demonstrates how to access the histogram bin edges and counts for the example above:

import pandas as pd
import altair as alt
import vegafusion as vf

flights = pd.read_parquet(
    "https://vegafusion-datasets.s3.amazonaws.com/vega/flights_1m.parquet"
)

delay_hist = alt.Chart(flights).mark_bar().encode(
    alt.X("delay", bin=alt.Bin(maxbins=30)),
    alt.Y("count()")
)
vf.transformed_data(delay_hist)

	bin_maxbins_30_delay	bin_maxbins_30_delay_end	__count
0	-20	0	419400
1	80	100	11000
2	0	20	392700
3	40	60	38400
4	60	80	21800
5	20	40	92700
6	100	120	5300
7	-40	-20	9900
8	120	140	3300
9	140	160	2000
10	160	180	1800
11	320	340	100
12	180	200	900
13	240	260	100
14	-60	-40	100
15	260	280	100
16	200	220	300
17	360	380	100

Quickstart 3: Accelerate interactive charts

While the VegaFusion mime renderer works great for static Altair charts, it's not as well suited for interactive charts visualizing large datasets. This is because the mime renderer does not maintain a live connection between the browser and the python kernel, so all the data that participates in an interaction must be sent to the browser.

To address this situation, VegaFusion provides a Jupyter Widget based renderer that does maintain a live connection between the chart in the browser and the Python kernel. In this configuration, selection operations (e.g. filtering to the extents of a brush selection) can be evaluated interactively in the Python kernel, which eliminates the need to transfer the full dataset to the client in order to maintain interactivity.

The VegaFusion widget renderer is provided by the vegafusion-jupyter package.

Instead of enabling the mime render with vf.enable_mime(), the widget renderer is enabled with vf.enable_widget(). Here is a full example that uses the widget renderer to display an interactive Altair chart that implements linked histogram brushing for a 1 million row flights dataset.

import pandas as pd
import altair as alt
import vegafusion as vf

vf.enable_widget()

flights = pd.read_parquet(
    "https://vegafusion-datasets.s3.amazonaws.com/vega/flights_1m.parquet"
)

brush = alt.selection(type='interval', encodings=['x'])

# Define the base chart, with the common parts of the
# background and highlights
base = alt.Chart().mark_bar().encode(
    x=alt.X(alt.repeat('column'), type='quantitative', bin=alt.Bin(maxbins=20)),
    y='count()'
).properties(
    width=160,
    height=130
)

# gray background with selection
background = base.encode(
    color=alt.value('#ddd')
).add_selection(brush)

# blue highlights on the selected data
highlight = base.transform_filter(brush)

# layer the two charts & repeat
chart = alt.layer(
    background,
    highlight,
    data=flights
).transform_calculate(
    "time",
    "hours(datum.date)"
).repeat(column=["distance", "delay", "time"])
chart

flights_brush_histogram.mov

Histogram binning, aggregation, and selection filtering are now evaluated in the Python kernel process with efficient parallelization, and only the aggregated data (one row per histogram bar) is sent to the browser.

You can see that the VegaFusion widget renderer maintains a live connection to the Python kernel by noticing that the Python kernel is running as the selection region is created or moved. You can also notice the VegaFusion logo in the dropdown menu button.

Supported Environments

VegaFusion can display Altair visualizations is the following contexts

Classic Notebook

Both the VegaFusion mime and widget renderers are compatible with version 6 of the Jupyter Notebook (Sometimes referred to as the Classic Notebook since it predates JupyterLab). The Classic Notebook extension containing the VegaFusion widget is included in the vegafusion-jupyter Python package, and it is enabled automatically when the package is installed. The mime renderer should be enabled with the "html" mimetype:

import vegafusion as vf
vf.enable_mime(mimetype="html")
...

JupyterLab

Both the VegaFusion mime and widget renderers are compatible with version 3+ of JupyterLab. The JupyterLab extension containing the VegaFusion widget is included in the vegafusion-jupyter Python package, and it is enabled automatically when the package is installed.

Voila

Both the VegaFusion mime and widget renderers are compatible with the Voila dashboard toolkit. This works with or without the enable_nbextensions flag.

Visual Studio Code

Both the VegaFusion mime and widget renderers are compatible with Visual Studio Code Python notebooks. The widget extension will be downloaded automatically from a CDN location, so an active internet connection is required to use the widget renderer.

Hex

The VegaFusion mime renderer is compatible with the Hex notebook. VegaFusion comes pre-installed and the mime renderer can be enabled as usual.

import vegafusion as vf
vf.enable_mime()
...

The widget renderer is not compatible with Hex.

Colab

The VegaFusion mime renderer is compatible with Google Colab when the "html-colab" mimetype is enabled:

%pip install vegafusion[embed]

import vegafusion as vf
vf.enable_mime(mimetype="html-colab")
...

The VegaFusion widget renderer is also compatible with Google Colab when the custom widget manager is enabled:

%pip install vegafusion-jupyter[embed]

from google.colab import output
output.enable_custom_widget_manager()

import vegafusion as vf
vf.enable_widget()
...

Kaggle

The VegaFusion mime renderer is compatible with Kaggle Notebooks when the "html-kaggle" mimetype is enabled:

%pip install vegafusion[embed]

import vegafusion as vf
vf.enable_mime(mimetype="html-kaggle")
...

The widget renderer is not currently compatible with Kaggle.

[jonmmease]

cc @binste @joelostblom @mattijn @nicolaskruchten @domoritz @dhirschfeld as this release candidate includes some features I've mentioned to you all over the past 4 months 😄

If anyone has a chance to give the release candidate a try, feel free to drop feedback in this thread. Thanks!

[jonmmease]

Let me know if anyone has opinions on which mime bundle type should be the default. I'm waffling between the text/html and application/vnd.vega.v5+json mimetypes.

text/html has the broadest compatibility (classic notebook, JupyterLab, Colab, etc.), but it requires an active internet connection to access a CDN. application/vnd.vega.v5+json is also pretty widely supported, except for the classic notebook and Colab, and it doesn't require an internet connection.

[binste]

Maybe it's more intuitive for users if it's the same as in Altair, i.e. html. This way, if a user never had to change the renderer in Altair then there is also no need to do it if VegaFusion is enabled. If the user works in an environment where the html mime bundle does not work due to for example a missing internet connection, they need to figure it out anyway and so are probably aware of the concepts and that they need to switch.

If I find some time this week to try out the release candidate, I'll make sure to let you know in case I'd find something. Thank you in any case, really looking forward to the release!

[binste]

I like it! But I see your point regarding the differentiation. Is it easier if mode refers to either "mime" or "widget" but still in the same function? Maybe there is a more intuitive name instead of "mime" ("altair", "default", "vegalite") but in any case, that would allow to split the documentation into two and reflect that the difference between "html" and "widget" is much bigger then "html" and "vega"?

vf.enable()  # Same as next line
vf.enable(mode="mime", mimetype="html")  # Default
vf.enable(mode="mime", mimetype="vega")
vf.enable(mode="mime", mimetype="svg")
vf.enable(mode="mime", mimetype="png")
vf.enable(mode="widget")

[jonmmease]

Looking at the details some more, enable_widget has several distinct configuration options that don't apply to the mime renderer case, so that would make the vf.enable signature a bit messy.

I think I'm going to rename enable_mime() to just enable() and keep enable_widget() as is. In the documentation I'll still use the terms "Mime renderer" and "Widget renderer", but the API doesn't need to force a user to deal with the term "mime" right off the bat.

Thanks for the feedback!

[joelostblom]

I also think it makes sense to match what altair is having as the default so that that user can expect the same things from both libraries. Although I must say that I also like the idea of offline capable by default... maybe we should explore if there should be an offline-enabled default renderer in altair? I am not entirely clear on under which combinations of renderers and frontends altair chart are currently displayed when offline.

[jonmmease]

I think currently "html", "colab", and "kaggle" renderers are the ones that require an internet connection to load from a CDN.

It's possible to create an offline capable HTML renderer by inlining the full JavaScript bundles in the HTML. plotly and bokeh are able to do this, but it's a pain because you don't want to inline the JS libraries for every chart, so you end up needing some kind of global (init_offline_mode()) function that dumps the JS dependencies into the notebook once.

I'd probably recommend keeping this as is for now, and then once Jupyter Notebook 7 is out (the classic notebook based on JupyterLab) switch to using the vega mimetype renderer as it's already pretty well supported everywhere outside of the classic notebook and Colab (and I think Colab already overrides the default renderer).

[joelostblom]

I'd probably recommend keeping this as is for now, and then once Jupyter Notebook 7 is out (the classic notebook based on JupyterLab) switch to using the vega mimetype renderer as it's already pretty well supported everywhere outside of the classic notebook and Colab (and I think Colab already overrides the default renderer).

I think that sounds like a good idea. It seems like the v7 version of the notebook is well on its way https://github.com/jupyter/notebook/milestone/45 so it might make sense to consider this for v5 of Altair since it is a change to the defaults that would make sense to have in a major release (depending on when we actually release the next altair version, there are still about 50 or so open issues for v7 of the notebook). I tested all the renderers and in addition to the three you mention, there is also "'zeppelin'" that requires an online connection.

I opened vega/altair#2797 to discuss this further

[jonmmease]

New release candidate over in #213

[jonmmease]

Latest RC in #225.

Note that I decided to skip 0.11.0 and go right to 1.0.0 since this release is the most significant one since VegaFusion was first announced (in particular, the mime renderer, transformed_data function). In terms of timeline, I'm aiming to release 1.0.0 final with docs and announcement material before the end of the month (hopefully sooner!).

[mattijn]

I finally had some time to play with vegafusion in the last two weeks and I've to say its really neat. Before we wrote custom aggregation query using duckdb-sql syntax and pipe it into the chart for really large tables, but now one can stick to the vega-altair/lite syntax without writing additional queries and still handle large tables.
I did run into one issue though. I'm not sure if there is yet support for the geo-interface. Let me open a issue for that.

[jonmmease]

That's great to hear! Yeah, VegaFusion doesn't implement any of the geo transforms yet. It shouldn't break a geo visualization (if it does that's definitely bug), but those transforms will still be performed with the Vega JavaScript library in the browser.

Thanks, it would be helpful to have an issue that includes the general geo use case(s) you're interested in having support for.