You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi I am trying to download lot of files from S3 to local.
I am sending the command file like below s5cmd run --numworker 256 commands.txt
and my commands file will have below sample of around 500 entries
eg: commands.txt
cp s3://XYZ/alprgagqenm-e-ec-c7kx-enmhost-1/abc.txt /home/test/copy/files/
I have literally like these around 500 entries in single commands.txt file
the same thing will be repeated untill all the files from S3 are downloaded like 10000 iteration.
What I am seeing is when I add records more than 70 its really taking a long to cp the files inspite of adding --numworker to 256 there is no impact.
what am I doing wrong or how does num-worker works?
Every number worker is 1 CPU on the server its running? for eg: i am running this on 80CPU 256GB gaint box.
provide some insight to tweak this pls.
The text was updated successfully, but these errors were encountered:
Hi I am trying to download lot of files from S3 to local.
I am sending the command file like below
s5cmd run --numworker 256 commands.txt
and my commands file will have below sample of around 500 entries
eg: commands.txt
cp s3://XYZ/alprgagqenm-e-ec-c7kx-enmhost-1/abc.txt /home/test/copy/files/
I have literally like these around 500 entries in single commands.txt file
the same thing will be repeated untill all the files from S3 are downloaded like 10000 iteration.
What I am seeing is when I add records more than 70 its really taking a long to cp the files inspite of adding --numworker to 256 there is no impact.
provide some insight to tweak this pls.
The text was updated successfully, but these errors were encountered: