Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serialise wide format tables at the end of data://grapher steps by default #3608

Open
larsyencken opened this issue Nov 25, 2024 · 4 comments

Comments

@larsyencken
Copy link
Collaborator

Motivation

We would like to have alignment between our indicator and ETL APIs, however the flattening of dimensions causes a misalignment and means that there is nowhere in the ETL APIs that can give you the same data that's in a chart.

Proposal

We should serialise and publish the generated data frames for grapher://grapher steps, as well as shipping that data to MySQL. It might need a new channel.

@larsyencken
Copy link
Collaborator Author

/cc @danyx23 @Marigold @pabloarosado

@pabloarosado
Copy link
Contributor

Thanks @larsyencken, just a quick note: By construction, the reason data://grapher steps exist (and that we don't publish garden steps directly) is to adapt our curated data to the neds of our grapher tool. So, if data://grapher are actually different from grapher://grapher steps, then they are not really fulfilling that promise, and therefore are a bit misleading (or redundant).
The data team does not need to be aware of the technical difference between data://grapher and grapher://grapher steps. In fact, we usually just speak of "grapher steps". However, with the current implementation, data managers need to know that, for some reason, mdim steps depend on grapher://grapher steps (instead of the usual data://grapher).
In my view, the ideal solution would be to have just one kind of grapher step (data://grapher). And grapher://grapher steps would simply be an implementation helper that data managers don't even need to be aware of.

@pabloarosado
Copy link
Contributor

We can discuss this alongside the mdim discussion (#3635) on the next weekly data call.

@larsyencken larsyencken changed the title Serialise the output of grapher://grapher steps to disk Serialise wide format tables at the end of data://grapher steps by default Dec 5, 2024
@larsyencken
Copy link
Collaborator Author

Renamed this now to reflect the approach that @Marigold is exploring, which seems promising.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants