lua memory usage

[ghost]

Hello, as I am able now to measure the lua used memory of my loaded mods in minetest,
(for this may look here)
moreblocks is using near 11950 KByte of lua memory (Biggest user on my server). As there are reported OOM crashes, when using LuaJIT is is important to reduce the usage of memory to avoid OOM.

• 3 Memory use optimisations for Lua mapgens
• Wiki - Mapgen memory optimisations
  and also the using of local var, is speeding up lua also
• Lua Performance Tips
• Lua object memory sizes
  as some of this optimization are also speeding up the code itself.

So you know your code better then others, to see what can be changed, as I guess might bring a lot memory saving.

[ghost]

please see Re: [Server] Asia Thailand Minetest 5.0.0 :30001 aktive as detected changes in behaviour with new version.

so what you changed that after all 1.500 KB more mem used now ?

I am working on some automatic lua memory cleaning, better then stop_lj_oom, but still not "able" (lua knowledge) to help you there inside your code

as also twoelk asked: what is happening inside your code
to may understand and find solutions

[Calinou]

@minetest-one Can you test every commit from the latest one (no need to test commits that didn't change any Lua files) to see where the memory usage increase comes from? That would be highly appreciated.

[ghost]

uff, had to search first how to find commit-versions, I just did today my first own mod, and still not sure for what all the links
so the one new is d96f110 I just took, the 1d054ec is 3 days older and 0fea5ee 14.07.18 might be the last one to compare, so 3 of them
give me some time - as I restart the test-server about 10 times again, again, so see if values keep similar

?? so cant download a commit-version, I don't see it - but should be able to reverse the seen changes at the last one, so then have the older version
and the very old should be still here somewhere

[ghost]

here we go:
loading:

• time (mem to 377),
• free_lj_mem (mem to 1004),
• moreblocks (mem to see below),
• unified_iinventory (end about ~600 higher than moreblocks)
  so these together just need 1600 during loading, but also I have seen that unified_inventory then index ? collect and rise, but here the value after was lower at old version, and higher at both new ones.

values are:
1st what might mod load reached mem during start,
2nd after unified_inventory showed loaded size,
3rd after that collectgarbage(), and show again

• old version 0fea5ee 14.07.18, loaded on 9.3.2019 acts at start
  10805 - 8931 - 8931
  10825 - 8857 - 8857
  10868 - 8970 - 8970
  10777 - 8848 - 8848
  10763 - 8879 - 8879

• newer version d96f110 (so far I did right changes back, but I guess I did)
  7078 - 12037 - 12037
  7078 - 12037 - 12037
  7425 - 12376 - 12376
  7529 - 12214 - 12215

• latest version 1d054ec (not much idff)
  7233 - 12186 - 12186
  7363 - 12309 - 12309
  7312 - 12256 - 12263
  7280 - 12225 - 12225
  7240 - 12187 - 12187

like I realised, new version load less at begin, but get huge after, and keep more memory
Why that small change cause so big diff, might be interesting to investigate as we have here less change in code but much in memory behaviour

I have no idea yet, and take a break - let this sink into brain ... or bad luck, get lost
add:
was that really the only change ? in moreblocks code ?
I solved that problem secondary - using free_lj_mem to keep memory low.
But as a fact, this is one of the most memory using mods I know, end understand code too less yet to understand why.
Good luck

[fluxionary]

i don't have anything concrete to add here, but more recent versions of minetest have plugged a number of memory leaks. moreblocks currently creates a huge quantity of new node ids, and the client generally doesn't handle nodeblock/mesh drawtypes as well as full nodes. this is probably an intractable problem to fix within the scope of this mod, though i think we should give links to relevant engine issues before closing this.

[fluxionary]

in my experience - which is broad but far from total across the minetest mod ecosystem - luajit OOM issues are mostly related to worldgen (caverealms, other_worlds, underch, etc.).

#191 should help reduce the impact of moreblocks on memory in the future, but overall, this is an architectural issue w/ minetest, and moreso luajit, and so is outside the scope of this mod.