Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Read properties file only once for all reports #121

Closed
Martin1887 opened this issue Dec 13, 2023 · 1 comment · May be fixed by #133
Closed

Read properties file only once for all reports #121

Martin1887 opened this issue Dec 13, 2023 · 1 comment · May be fixed by #133

Comments

@Martin1887
Copy link
Contributor

Fetchers with many executions and attributes can produce giant properties files. Read a properties file of this size can take around 1 minute, and this can suppose several hours when many plots are needed.

So, would be possible run a range of steps at once to read the properties file in memory only once?

The CLI interface could be something like

python fetcher.py 2-2000

This would dramatically reduce the total time of generating reports, since each plot takes around 2 seconds when the properties file is in memory.

Thanks!
@jendrikseipp
Copy link
Collaborator

Thanks for the idea! The approach of using one report per step is currently baked deeply into Lab. As such it will be quite difficult to do this for all kinds of reports. For your specific use case, however, you can consider copying the ScatterPlotReport class and instead of requiring that there's exactly one attribute and one pair of algorithms, loop over multiple of them.

@Martin1887 Martin1887 mentioned this issue Jul 18, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

One read multiple report steps Martin1887/lab
2 participants
@jendrikseipp @Martin1887