Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S5CMD numworker issue #743

Open
anudina opened this issue Jul 26, 2024 · 1 comment
Open

S5CMD numworker issue #743

anudina opened this issue Jul 26, 2024 · 1 comment

Comments

@anudina
Copy link

anudina commented Jul 26, 2024

Hi I am trying to download lot of files from S3 to local.
I am sending the command file like below
s5cmd run --numworker 256 commands.txt
and my commands file will have below sample of around 500 entries

eg: commands.txt

cp s3://XYZ/alprgagqenm-e-ec-c7kx-enmhost-1/abc.txt /home/test/copy/files/
I have literally like these around 500 entries in single commands.txt file
the same thing will be repeated untill all the files from S3 are downloaded like 10000 iteration.
What I am seeing is when I add records more than 70 its really taking a long to cp the files inspite of adding --numworker to 256 there is no impact.

  1. what am I doing wrong or how does num-worker works?
  2. Every number worker is 1 CPU on the server its running? for eg: i am running this on 80CPU 256GB gaint box.
    provide some insight to tweak this pls.
@rmanus
Copy link

rmanus commented Aug 20, 2024

You might be exhausting your network bandwith with so many workers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants