Fine-tunning memory consumption

[dundee]

Hi all,
are you satisfied with memory consumption and performance on huge file systems when using the -g flag?

I am considering if there should some garbage collector tuning applied, e.g. something like https://eng.uber.com/how-we-saved-70k-cores-across-30-mission-critical-services/

It would introduce for example flag --max-memory which will tell gdu what is the limit of memory usage it should not cross.
Gdu will then be able to do garbage collection less often in the beginning of analysis when less memory is used (gaining more performance) and then to do it more often when memory consumption will approach the limit.

What do you think?

//cc @sss123next @daniejstriata

[dundee]

And another question 😄 :

Would you appreciate a possibility to store the analysis into some fast embedded key-value store instead of memory? E.g. BadgerDB. The analysis would be of course slower a lot but it will use very little memory and the result could be opened again very fast.

[dundee]

Yes, it would be not trivial, but it looks like a nice addition to gdu.

[dundee]

Thanks for feedback, I have created #118 for this.

[sss123next]

thx !

[danie-dejager]

If a DB is to be written, can gdu calculate free memory and process into memory until it can no longer and dump the results to the DB?

As a user, I'd find it difficult to know beforehand to turn garbage collection on to prevent / minimise gdu from crashing? I don't know how big a directory is initially and especially if it is a large directory I want to go on with other tasks and not babysit anything.

[dundee]

Wow, that sounds like it would need some quite heavy logic. I will think about it, it would be beneficial to have this.

[dundee]

I have implemented first step as a start :) Gdu will now meassure how much memory is used and will compare with the total free memory on the host. When will be used more than is free, gdu will enable the garbage collection.

The GC will be called more often when more memory is used (and less is free).