Skip to content

Commit

Permalink
Fix bug uploading empty directories for deepmodeling/dpgen#1180 (#322)
Browse files Browse the repository at this point in the history
Items in `directory_list` are dealt with in the same way as `file_list`
during `_walk_directory()`. However, items in it were not converted back
to relavent path pattern while `file_list` did, causing bug when
uploading an empty directory. In DP-GEN process, each Lammps task
contains an empty `traj` directory, and thus, it will be passed into
`tar.add()` with wrong `ii` and moreover, wrong `arcname`.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
  • Loading branch information
Cloudac7 and pre-commit-ci[bot] authored Apr 13, 2023
1 parent f1b710f commit 4a5564e
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion dpdispatcher/ssh_context.py
Original file line number Diff line number Diff line change
Expand Up @@ -565,7 +565,7 @@ def upload(
file_list = []
directory_list = []
for task in submission.belonging_tasks:
directory_list.append(task.task_work_path)
directory_list.append(os.path.join(self.local_root, task.task_work_path))
# file_list.append(ii)
self._walk_directory(
task.forward_files,
Expand All @@ -577,6 +577,9 @@ def upload(
submission.forward_common_files, self.local_root, file_list, directory_list
)

# convert to relative path to local_root
directory_list = [os.path.relpath(jj, self.local_root) for jj in directory_list]

# check if the same file exists on the remote file
# only check sha256 when the job is recovered
if recover:
Expand Down

0 comments on commit 4a5564e

Please sign in to comment.