Killed - Out of Memory

[Bob-emblibrary]

Trying to sync a lot of files and getting
[7258676.421271] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-367512.scope,task=s5cmd,pid=3961979,uid=0
[7258676.421303] Out of memory: Killed process 3961979 (s5cmd) total-vm:2356132kB, anon-rss:1659808kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:3364kB oom_score_adj:0
This is running on a droplet on DigitalOcean. Any ideas on how to mitigate the memory issue?

[rmanus]

I was able to mitigate the issue by tuning the Go garbage collector to be more aggressive when executing s5cmd sync command:

        `GOMEMLIMIT=500MiB GOGC=60 s5cmd sync ....`

I'm also using a modified version of s5cmd where I'm able to reduce the number of objects that are sorted in-memory.

That way I'm able to sync at least a million of files on a system with 2GiB of RAM.