Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature request]: Have gempyor been able to measure its pickled memory to setup batch run #360

Open
jcblemai opened this issue Oct 25, 2024 · 0 comments
Open

[Feature request]: Have gempyor been able to measure its pickled memory to setup batch run #360

jcblemai opened this issue Oct 25, 2024 · 0 comments
Labels
batch Relating to batch processing. gempyor Concerns the Python core. inference Concerns the parameter inference framework. medium priority Medium priority.

Comments

@jcblemai
Copy link
Collaborator

Label

batch, gempyor, inference

Priority Label

medium priority

Is your feature request related to a problem? Please describe.

When submitting a cluster job, we need to specify some memory usage. During the beta testing of emcee, a lot folks (@saraloo, @jcblemai, @anjalika-nande) got into trouble because not enough memory was allocated, and guessing this number is hard.

Is your feature request related to a new application, scenario round, pathogen? Please describe.

This is for all rounds, a big time saver.

Describe the solution you'd like

In the submission script we want to build for emcee, we might want to build the pickable object that is distributed across cores, measure its memory footprint, and use that to inform the SLURM memory request
@jcblemai jcblemai changed the title [Feature request]: Have gempyor been able to measure it's pickled memory [Feature request]: Have gempyor been able to measure its pickled memory to setup batch run Oct 25, 2024
@jcblemai jcblemai mentioned this issue Oct 28, 2024
Open
TimothyWillard added a commit that referenced this issue Oct 31, 2024
@TimothyWillard
Add JobSize.size_from_jobs_sims_blocks
2e9b66a 
Added a class method to `gempyor.batch.JobSize` class that is slightly
more convenient to use from a CLI script. Provides a good home for
future code realted to GH-360. Corresponding tests and docs.
TimothyWillard added a commit that referenced this issue Nov 4, 2024
@TimothyWillard
Add JobSize.size_from_jobs_sims_blocks
6be1a62 
Added a class method to `gempyor.batch.JobSize` class that is slightly
more convenient to use from a CLI script. Provides a good home for
future code realted to GH-360. Corresponding tests and docs.
@TimothyWillard TimothyWillard added gempyor Concerns the Python core. batch Relating to batch processing. medium priority Medium priority. inference Concerns the parameter inference framework. labels Nov 4, 2024
TimothyWillard added a commit that referenced this issue Nov 7, 2024
@TimothyWillard
Greatly simplify JobSize for first draft
c8c0723 
Mainly reduced the functionality of the `size_from_jobs_sims_blocks`
class method. Best to leave that to more feedback and GH-360.
TimothyWillard added a commit that referenced this issue Nov 8, 2024
@TimothyWillard
Add JobSize.size_from_jobs_sims_blocks
10faf77 
Added a class method to `gempyor.batch.JobSize` class that is slightly
more convenient to use from a CLI script. Provides a good home for
future code realted to GH-360. Corresponding tests and docs.
TimothyWillard added a commit that referenced this issue Nov 8, 2024
@TimothyWillard
Greatly simplify JobSize for first draft
b0d1852 
Mainly reduced the functionality of the `size_from_jobs_sims_blocks`
class method. Best to leave that to more feedback and GH-360.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
batch Relating to batch processing. gempyor Concerns the Python core. inference Concerns the parameter inference framework. medium priority Medium priority.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jcblemai @TimothyWillard