Parallel pyiron table #1050
Closed
Parallel pyiron table #1050
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jan-janssen
commented
Feb 27, 2023
| @@ -432,7 +420,7 @@ def _collect_job_update_lst(self, job_status_list, job_stored_ids=None): | |||
| and job.status in job_status_list | |||
| and self.filter_function(job) | |||
| ): | |||
| job_update_lst.append(job) | |||
| job_update_lst.append(job_id) | |||
There was a problem hiding this comment.
This is the tricky part. To apply the filter_function the job is already loaded in inspect mode, but as the job cannot be communicated to the subprocess, it has to be loaded again inside the subprocess.
|
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
|
@ligerzero-ai Does this help your needs for a parallel pyiron table version? |
jan-janssen
commented
Jul 24, 2023
|
This is great - I've got a separate contribution opening up as a draft in contrib later tonight. That one is completely individualised outside of the pyiron ecosystem. I am hoping that it will allow users to create dataframes for ML potentials easily (a. la. TrainingContainer with a little bit of fiddling) I will ping you there. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
jan-janssen
added
format_black
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
# Conflicts: # .ci_support/environment.yml # setup.py
|
Based on the performance analysis from @ligerzero-ai in https://github.com/orgs/pyiron/discussions/211#discussioncomment-8034046 and the general implementation of adding support for executors in #1155 - I removed the direct dependence on |
|
On hand I like that one can customize the executor. On the other hand having to pass the instance in is a bit clunky and won't work anymore once I submit a table to the queue (or does it somehow?). I think it's ok leave the option to pass the instance in, but it would be nice if the table just creates a |
jan-janssen
added
format_black
jan-janssen
added
format_black
|
The issue here is similar to what we discuss in #1296 . Basically, we can have an executor attached to a single job object, in that case the job object is submitted to the executor and executed on one of the workers of the executor. Still in the case of jobs with multiple job objects inside them, like GenericMaster jobs or alternatively with pyiron tables which have multiple function calls inside them we want assign a single executor which is then used to execute the individual tasks within this job. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently the initialisation for the individual subprocesses is too expensive, so it does not make a lot of sense to focus on a parallel pyiron table implementation. Still it is maybe a good point to start. Example code: