Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DON'T MERGE] PoC of recording stats during kedro run #1465

Closed
wants to merge 1 commit into from

Conversation

noklam
Copy link
Contributor

@noklam noklam commented Jul 27, 2023

Description

#662

If you run

cd demo_project
kedro run

You should see a stats.json created.

Development notes

Pros:

  • No extra cost to read data when users fire up viz to see the statistics.
  • Need better way to manage the version, or maybe it is ok because the current pipeline view also only display the latest pipeline. If viz saw this file then it can optionally skip the read.

Cons:

  • It can only show the stats for the last run, if it's the first time user fire up Viz it won't work.
  • Requires an upgrade of kedro-viz & a new kedro run to generate this stats.json

Notes:
It's a PoC, if we actually want to implement this we need to be more careful about the type checking, maybe we can check the type of the dataset instead of data. We may also need to take care of the remote storage (or it could be a next step)

QA notes

Checklist

  • Read the contributing guidelines
  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added new entries to the RELEASE.md file
  • Added tests to cover my changes

@noklam noklam changed the title POC of recording stats during kedro run [DON'T MERGE] PoC of recording stats during kedro run Jul 27, 2023
@noklam
Copy link
Contributor Author

noklam commented Aug 16, 2023

Closed in favor of #1472

@noklam noklam closed this Aug 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants