You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We would like to have alignment between our indicator and ETL APIs, however the flattening of dimensions causes a misalignment and means that there is nowhere in the ETL APIs that can give you the same data that's in a chart.
Proposal
We should serialise and publish the generated data frames for grapher://grapher steps, as well as shipping that data to MySQL. It might need a new channel.
The text was updated successfully, but these errors were encountered:
Thanks @larsyencken, just a quick note: By construction, the reason data://grapher steps exist (and that we don't publish garden steps directly) is to adapt our curated data to the neds of our grapher tool. So, if data://grapher are actually different from grapher://grapher steps, then they are not really fulfilling that promise, and therefore are a bit misleading (or redundant).
The data team does not need to be aware of the technical difference between data://grapher and grapher://grapher steps. In fact, we usually just speak of "grapher steps". However, with the current implementation, data managers need to know that, for some reason, mdim steps depend on grapher://grapher steps (instead of the usual data://grapher).
In my view, the ideal solution would be to have just one kind of grapher step (data://grapher). And grapher://grapher steps would simply be an implementation helper that data managers don't even need to be aware of.
We can discuss this alongside the mdim discussion (#3635) on the next weekly data call.
larsyencken
changed the title
Serialise the output of grapher://grapher steps to disk
Serialise wide format tables at the end of data://grapher steps by default
Dec 5, 2024
Motivation
We would like to have alignment between our indicator and ETL APIs, however the flattening of dimensions causes a misalignment and means that there is nowhere in the ETL APIs that can give you the same data that's in a chart.
Proposal
We should serialise and publish the generated data frames for
grapher://grapher
steps, as well as shipping that data to MySQL. It might need a new channel.The text was updated successfully, but these errors were encountered: