Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write submission script to temp #501

Closed
wants to merge 1 commit into from
Closed

Write submission script to temp #501

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions executorlib/cache/queue_spawner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import os
import subprocess
import tempfile
from typing import List, Optional, Tuple, Union

from pysqa import QueueAdapter
Expand Down Expand Up @@ -53,6 +54,7 @@ def execute_with_pysqa(
"command": " ".join(command),
"dependency_list": [str(qid) for qid in task_dependent_lst],
"working_directory": os.path.abspath(cwd),
"submission_script_path": tempfile.NamedTemporaryFile().name,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Critical: Improve temporary file handling

The current implementation has several issues:

  1. The temporary file might be deleted before the queue system can access it
  2. No proper resource cleanup
  3. Missing error handling
  4. File permissions aren't explicitly set

Here's a safer implementation:

-            "submission_script_path": tempfile.NamedTemporaryFile().name,
+            "submission_script_path": tempfile.mkstemp(prefix="job_", suffix=".sh", text=True)[1],

Additionally, consider implementing proper cleanup:

def execute_with_pysqa(...):
    temp_files = []
    try:
        # Create temp file with mkstemp
        fd, temp_path = tempfile.mkstemp(prefix="job_", suffix=".sh", text=True)
        os.close(fd)  # Close the file descriptor
        temp_files.append(temp_path)
        
        submit_kwargs = {
            # ... other kwargs ...
            "submission_script_path": temp_path,
        }
        # ... rest of the function ...
    finally:
        # Cleanup temp files after job submission
        for temp_file in temp_files:
            try:
                if os.path.exists(temp_file):
                    os.remove(temp_file)
            except OSError:
                pass  # Best effort cleanup

This solution:

  1. Uses mkstemp to create a persistent temporary file
  2. Implements proper cleanup after job submission
  3. Uses appropriate file permissions
  4. Includes error handling
🧰 Tools
🪛 Ruff

57-57: Use a context manager for opening files

(SIM115)

}
if "cwd" in resource_dict:
del resource_dict["cwd"]
Expand Down
Loading