-
Notifications
You must be signed in to change notification settings - Fork 284
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ModSecurity memory leak #236
Comments
Hi @vncloudsco, What is your ModSecurity version? |
We run the system for about 2 weeks as it runs out of memory. |
@vncloudsco is that version 3.0.4? are you happens to perform webserver reloads? Did you ever observed if the leaks are related to the reload? |
Hi @vncloudsco , There are some known memory leaks that have already been fixed in 3.1 development code, but are not yet in v3/master or in an official release. Per @zimmerle's question if you're on 3.0.4, please mention whether you mean the official release (from Jan. 10, 2020) or if you mean current v3/master. |
Similar problem here. Problem occurs with v1.0.1 connector, modsecurity version 3.0.4 on nginx 1.16.1.
and nginx -s reload command was run, x (M) * number of nginx servers (with modsecurity module attached) is the amount of memory that nginx is taking permanently. Without connecting the module to any of nginx servers, the problem does not occur. |
Running any particular ruleset? custom rules? |
Yes I got some custom ruleset and I'm basing on owasp-modsecurity-crs-3.2.0. I also just checked that after disabling the custom ruleset, problem still seems to exist. |
@martinhsv My problem is the same with @Michal256 . When start running, the system has no problem. But after a period of use, the memory is overflowed specifically after 2 weeks of operation. I have tested it with many rule (comodo modsecurity and owasp-modsecurity-crs-3.2.0 ) sets but to no avail. |
If your memory leaks are not associated with rule reload, In v3.0.4 I am aware of the following less-used ModSecurity features that will leak memory: setenv (action) To my knowledge CRS does not use any of these, but depending on which rule sets you are using you (All three of the above have been corrected in 3.1-experimental code -- i.e. not yet available in v3/master) It may well be, however, that you are experiencing something not previously identified. |
Dear: @martinhsv Something is wrong, let me ask how can I build the library?
|
Hi @vncloudsco , I have not seen that particular compile error. One thing to pay attention to is the output of the ./configure step. The output may tell you about some critical dependency that is missing (although in this case I suspect that it won't). One possibility is that what you are seeing is related to compiler version. What is your compiler and version? (And what O/S and distribution are you on?) If you are using gcc and the versioning seems to match, you may be hitting this compiler bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58541 Which is asserted to have been fixed in 7.1.0. If that all seems to line up, you could try upgrading your compiler version. |
@martinhsv my os:
I cannot compile it. though can run in v3.0 version with no problems |
Hi @vncloudsco , Did you try checking your compiler version? E.g.
|
@martinhsv |
Hi @martinhsv @vncloudsco, i'm expirencing same issues with modsecurity enabled in nginx version 1.16.1 with libmodsecurity 3.0.4 and nginx-modesecurity module 1.0.1:
Number of vhost with memory usage after ~6 days of nginx uptime reloading ~once per 4 hours. I have no idea why only 2 of these servers have acceptable 1.2GB memory usage (with 0 modsec vhosts) but others doesn't.
Screenshots from monitoring: View of manual nginx stop / nginx reloading: View of automatic reloads (~once per 4h, clear increase of memory usage): |
@qjavax thanks for the detailed report. is lmdb enabled? if so, may be related to owasp-modsecurity/ModSecurity#2520 |
@zimmerle no I don't, i use nginx as a reverse proxy without lmdb, Only modules i have enabled are: headers-more-nginx-module, naxsi and ModSecurity-nginx |
@qjavax I meant to ask if you had lmdb enabled during the ModSecurity build. You can choose different backends for collections, one of the backends is the lmdb. There is this pull request on owasp-modsecurity/ModSecurity#2520 that aiming to fix a memory leak. |
@zimmerle oh sorry i didn't get your question. But still I don't have LMDB enabled in my modsecurity build I use almost default configuration):
|
@zimmerle do you want me to provide more information about this leak? |
@zimmerle Do you need any more information? we still get the memory leak. |
@qjavax @vncloudsco It would be amazing to have a valgrind output of this particular issue. It will speed up the investigation process. |
Hi @zimmerle, yes I could provide some valgrind output but i'm having trouble building nginx with debug symbols. I tried adding: |
@zimmerle Do you need any more information about leak? |
JFTR: (there is a reproducible way to observe a number of leaks not necessary related to any regex processing - see the above comments; just came across to this ticket while searching for any existing unresolved memory leaks reports) |
@vncloudsco @defanator See #277 and owasp-modsecurity/ModSecurity#2710. I have fixed in #277 by a workaround. |
@liudongmiao If I apply the patch to the current master branch the memory leak gets fixed but starts to block a basic login page with TX:ANOMALY_SCORE + audit log stop working. Looks like this leak was reported really long time ago - in the meantime new version was also released - NOT everybody is affected by this memory leak problem? Anybody has any instructions how I can install with no memory leak ? |
may be your WAF has started to work? :)
Now I checked, and it runs since only few minutes, I sent few requests, but there nothing changed in the memory usage of Nginx.
I'm using Debian, and installed packages from here. Note: the libmodsecurity package does not use the LMDB. |
Still memory leaks in current v3/master branch. I compiled modsec using base configuration and memory isn't freed on |
@baudneo The main memory leak should be fixed in owasp-modsecurity/ModSecurity@e9a7ba4. And for discussion, you can read this issue: owasp-modsecurity/ModSecurity#2728 |
@liudongmiao hi! I've built modsec v3/master branch with those fixes applied and still have the memory leak. I have opened a new issue about it but the maintainer seemed to take offense to me opening the issue. If I only enabled modsec once in the root http {} block and modsec isn't compiled with lmdb the memory leak is only 10-12MB per reload. With db I get 200+MB leak per reload. I have a user that is loading modsec directives and different rule files in server or location blocks and they are reporting up to 300MB leak on every reload. |
After I updated to the latest ubuntu the memory leak issue got fixed .... (22.04) |
I just tried what @szilard6 proposed, unfortunately, I cannot observe any meaningful difference between the two setups (Ubuntu 20.04 vs 22.04). I am not even sure that the Operating system is actually making any significant difference. |
I would be quite surprised upgrading the OS version would eliminate issues with reload-without-restart. |
I never did see you answer zimmerle's question "are you happens to perform webserver reloads?" If this report is related to nginx reload, I would prefer to close this as a duplicate. |
Dear: @martinhsv I also update my OS 20.04 and 22.04 and didn't see any change as @szilard6 mentioned |
Closing as duplicate. |
Dear: dev team
I have to use modsec with nginx (nginx version 1.16.1) However, it takes about 2 weeks for us to run out of memory, the system did not self-release the ram. How can I fix it?
lsof command show
The text was updated successfully, but these errors were encountered: