Files: Transfer may log errors when transferring a directory.

[scblack321]

When transferring a directory containing multiple files, the Files service may log errors.
Transferring a directory with a single file does not log any errors.
The transfer does succeed, possibly because Files is re-trying each child transfer.
Here is an example stack trace:

2024-05-02 15:57:09.997 ERROR [pool-6-thread-19] e.u.t.t.f.l.s.ChildTaskTransferService:745 - FILES_CLIENT_SSH_OP_ERR1 Error during operation. OboTenant: dev OboUser: testuser2 Operation: mkdir System: tapisv3-simpletest-exec-fork EffectiveUser: testuser2 Host: 129.114.35.122 Path: /home/testuser2/simpletest/exec/workdir/jobs/5b01f93c-4f83-40f1-a80a-4234d02b5be4-007/file1.txt Error: SFTP error (SSH_FX_FAILURE): Failure
java.util.concurrent.ExecutionException: java.io.IOException: FILES_CLIENT_SSH_OP_ERR1 Error during operation. OboTenant: dev OboUser: testuser2 Operation: mkdir System: tapisv3-simpletest-exec-fork EffectiveUser: testuser2 Host: 129.114.35.122 Path: /home/testuser2/simpletest/exec/workdir/jobs/5b01f93c-4f83-40f1-a80a-4234d02b5be4-007/file1.txt Error: SFTP error (SSH_FX_FAILURE): Failure
    at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
    at edu.utexas.tacc.tapis.files.lib.services.ChildTaskTransferService.doTransfer(ChildTaskTransferService.java:741)
    at edu.utexas.tacc.tapis.files.lib.services.ChildTaskTransferService.handleMessage(ChildTaskTransferService.java:265)
    at edu.utexas.tacc.tapis.files.lib.services.ChildTaskTransferService.handleDelivery(ChildTaskTransferService.java:209)
    at edu.utexas.tacc.tapis.files.lib.services.ChildTaskTransferService$2.handleDelivery(ChildTaskTransferService.java:186)
    at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149)
    at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:111)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.IOException: FILES_CLIENT_SSH_OP_ERR1 Error during operation. OboTenant: dev OboUser: testuser2 Operation: mkdir System: tapisv3-simpletest-exec-fork EffectiveUser: testuser2 Host: 129.114.35.122 Path: /home/testuser2/simpletest/exec/workdir/jobs/5b01f93c-4f83-40f1-a80a-4234d02b5be4-007/file1.txt Error: SFTP error (SSH_FX_FAILURE): Failure
    at edu.utexas.tacc.tapis.files.lib.clients.SSHDataClient.handleSftpException(SSHDataClient.java:1054)
    at edu.utexas.tacc.tapis.files.lib.clients.SSHDataClient.mkdir(SSHDataClient.java:281)
    at edu.utexas.tacc.tapis.files.lib.clients.SSHDataClient.createFile(SSHDataClient.java:668)
    at edu.utexas.tacc.tapis.files.lib.clients.SSHDataClient.upload(SSHDataClient.java:289)
    at edu.utexas.tacc.tapis.files.lib.services.ChildTaskTransferService.performSynchFileTransfer(ChildTaskTransferService.java:920)
    at edu.utexas.tacc.tapis.files.lib.services.ChildTaskTransferService.processTransfer(ChildTaskTransferService.java:536)
    at edu.utexas.tacc.tapis.files.lib.services.ChildTaskTransferService$3.call(ChildTaskTransferService.java:708)
    at edu.utexas.tacc.tapis.files.lib.services.ChildTaskTransferService$3.call(ChildTaskTransferService.java:705)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    ... 3 common frames omitted
Caused by: org.apache.sshd.sftp.common.SftpException: Failure
    at org.apache.sshd.sftp.client.impl.AbstractSftpClient.throwStatusException(AbstractSftpClient.java:277)
    at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkResponseStatus(AbstractSftpClient.java:264)
    at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkResponseStatus(AbstractSftpClient.java:246)
    at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkCommandStatus(AbstractSftpClient.java:233)
    at org.apache.sshd.sftp.client.impl.AbstractSftpClient.mkdir(AbstractSftpClient.java:810)
    at edu.utexas.tacc.tapis.shared.ssh.apache.SSHSftpClient.mkdir(SSHSftpClient.java:275)
    at edu.utexas.tacc.tapis.files.lib.clients.SSHDataClient.mkdir(SSHDataClient.java:278)
    ... 10 common frames omitted
2024-05-02 15:57:10.024 ERROR [pool-6-thread-19] e.u.t.t.f.l.s.ChildTaskTransferService:300 - FILES_TXFR_SVC_ERR1 Tenant: dev User: testuser2 Operation: handleMessage ID: 161,412 Tag: null UUID: f7bd42d3-ce40-420b-9ea6-0de3ba57e6c1 Error: FILES_CLIENT_SSH_OP_ERR1 Error during operation. OboTenant: dev OboUser: testuser2 Operation: mkdir System: tapisv3-simpletest-exec-fork EffectiveUser: testuser2 Host: 129.114.35.122 Path: /home/testuser2/simpletest/exec/workdir/jobs/5b01f93c-4f83-40f1-a80a-4234d02b5be4-007/file1.txt Error: SFTP error (SSH_FX_FAILURE): Failure
java.io.IOException: FILES_CLIENT_SSH_OP_ERR1 Error during operation. OboTenant: dev OboUser: testuser2 Operation: mkdir System: tapisv3-simpletest-exec-fork EffectiveUser: testuser2 Host: 129.114.35.122 Path: /home/testuser2/simpletest/exec/workdir/jobs/5b01f93c-4f83-40f1-a80a-4234d02b5be4-007/file1.txt Error: SFTP error (SSH_FX_FAILURE): Failure
    at edu.utexas.tacc.tapis.files.lib.services.ChildTaskTransferService.doTransfer(ChildTaskTransferService.java:747)
    at edu.utexas.tacc.tapis.files.lib.services.ChildTaskTransferService.handleMessage(ChildTaskTransferService.java:265)
    at edu.utexas.tacc.tapis.files.lib.services.ChildTaskTransferService.handleDelivery(ChildTaskTransferService.java:209)
    at edu.utexas.tacc.tapis.files.lib.services.ChildTaskTransferService$2.handleDelivery(ChildTaskTransferService.java:186)
    at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149)
    at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:111)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.IOException: FILES_CLIENT_SSH_OP_ERR1 Error during operation. OboTenant: dev OboUser: testuser2 Operation: mkdir System: tapisv3-simpletest-exec-fork EffectiveUser: testuser2 Host: 129.114.35.122 Path: /home/testuser2/simpletest/exec/workdir/jobs/5b01f93c-4f83-40f1-a80a-4234d02b5be4-007/file1.txt Error: SFTP error (SSH_FX_FAILURE): Failure
    at edu.utexas.tacc.tapis.files.lib.clients.SSHDataClient.handleSftpException(SSHDataClient.java:1054)
    at edu.utexas.tacc.tapis.files.lib.clients.SSHDataClient.mkdir(SSHDataClient.java:281)
    at edu.utexas.tacc.tapis.files.lib.clients.SSHDataClient.createFile(SSHDataClient.java:668)
    at edu.utexas.tacc.tapis.files.lib.clients.SSHDataClient.upload(SSHDataClient.java:289)
    at edu.utexas.tacc.tapis.files.lib.services.ChildTaskTransferService.performSynchFileTransfer(ChildTaskTransferService.java:920)
    at edu.utexas.tacc.tapis.files.lib.services.ChildTaskTransferService.processTransfer(ChildTaskTransferService.java:536)
    at edu.utexas.tacc.tapis.files.lib.services.ChildTaskTransferService$3.call(ChildTaskTransferService.java:708)
    at edu.utexas.tacc.tapis.files.lib.services.ChildTaskTransferService$3.call(ChildTaskTransferService.java:705)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    ... 3 common frames omitted
Caused by: org.apache.sshd.sftp.common.SftpException: Failure
    at org.apache.sshd.sftp.client.impl.AbstractSftpClient.throwStatusException(AbstractSftpClient.java:277)
    at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkResponseStatus(AbstractSftpClient.java:264)
    at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkResponseStatus(AbstractSftpClient.java:246)
    at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkCommandStatus(AbstractSftpClient.java:233)
    at org.apache.sshd.sftp.client.impl.AbstractSftpClient.mkdir(AbstractSftpClient.java:810)
    at edu.utexas.tacc.tapis.shared.ssh.apache.SSHSftpClient.mkdir(SSHSftpClient.java:275)
    at edu.utexas.tacc.tapis.files.lib.clients.SSHDataClient.mkdir(SSHDataClient.java:278)
    ... 10 common frames omitted