Segmentation fault in ModSecurity 2.9 for Apache #3300
Comments
|
Hi @vizovitin, thanks for provided info. We try to investigate the root cause soon, but probably the given patch would be enough at first. @marcstern what do you think? |
|
JFYI: There might be a few other places in the same file with similar issue, but since I'm not acquainted with the code and they don't seem to trigger any issues, I chose to keep the patch to a minimum (i.e. patch only one usage). |
|
I wonder if the cause is not that modsecurity_request_body_end_raw() is called with a body that's not chunked-encoded. @vizovitin: can you show us an example of problematic request? |
|
Unfortunately, I don't think I can. It is from a production server I don't control. The request itself is rather large and includes various authentication and other potentially private data. If you need to check something specific I may still be able to check it. |
|
Can we have the request headers (you can remove any specific one, like cookies)? |
|
Sure. I've redacted some stuff (mostly marked as curl 'https://abcde.example.net/wp-json/wplms/v2/create/13377?post' \
-H 'accept: */*' \
-H 'accept-language: en-US,en;q=0.9' \
-H 'content-type: text/plain;charset=UTF-8' \
-H 'cookie: wordpress_test_cookie=WP%20Cookie%20check; wp_lang=en_US; other_cookies=REDACTED' \
-H 'origin: https://abcde.example.net' \
-H 'priority: u=1, i' \
-H 'referer: https://abcde.example.net/' \
-H 'sec-ch-ua: "Not/A)Brand";v="8", "Chromium";v="123", "Google Chrome";v="123"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'sec-ch-ua-platform: "Windows"' \
-H 'sec-fetch-dest: empty' \
-H 'sec-fetch-mode: cors' \
-H 'sec-fetch-site: same-origin' \
-H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/512.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/512.36' \
--data-raw '{"course":"REDACTED","object":["REDACTED"],"token":"REDACTED","course_id":13377}' |
|
Can we have the headers "Content-Type" and "Transfer-Encoding" from the original request? |
|
Content-Type is listed above. There was no Transfer-Encoding, apparently. |
|
Sorry, I meant Content-Length. |
|
I just copied the request data from a browser developer console. So if the header is not there - I suppose it wasn't present. There are no specified directives in the configuration:
You can also see above in the issue, which might be useful: |
|
I can confirm the issue on a production server with thousands of requests per minute. Unfortunately, because there are so many requests, I can't assign the segmentation faults to specific calls. I had to disable modsecurity until the version is fixed. The current Version is not usable for large production servers with thousands of requests. |
|
I feel your pain @Orgoth, but ModSec 2.9 is definitely suitable for production. Which does not mean it's free from bugs. What you are encountering is tough. Have not seen such a bad behavior in years. On Apache, you have mod_forensic that writes a request to disk the moment it starts - and when it finishes. When you get a segfault, the culprit is in the log, in the delta to be precise. Maybe you can enable that together with ModSec in the off hours for a couple of minutes. Or if you have a loadbalancer in front of multiple instances, you direct but a small portion of the traffic to a server with mod_forensic + ModSec so the impact is lower. |
Thank you for the hint with mod_forensic, I will test it when the load is not this high. |
|
Very good. Unfortunately, I can not help you fix the code - @marcstern is in the best position to do that. But I can help you dig down on this. Which will ideally bring up a minimal config and a minimal curl call to trigger the segfault. Which will apparently support a fast solution for your problem. So mod_forensic should help you isolate the request. The next step is putting it under the microscope. Identify the request via URI, IP address or some other pattern. Then create a rule that disables ModSec for everybody else (ctl rule engine off). Ideally very few SegFaults going forward now and you can run this in prod under normal load. Then you remove all the rules outside the ctl statement rule and hope there is no more segfault. Now you add a rule that triggers the full audit log for the request in question. You now have a full dump of the request. Reproduce it with curl and try to trigger the segfault (reenable ModSec and the rules for your client apparently). If that works, try to reproduce in the lab. If that works, gradually reduce the rules and the config until you have a minimal config and the individual rule that triggers the problem. I am not ruling out a situation where it's a combination of rules and requests interfering with one another. But I suggest you try out the recipe above. If this does not work, we need to brainstorm based on your findings. |
This is strange no? |
|
Yes. Strange but while I have seen application servers segfault on this, ModSec is usually coping. Or am I wrong? |
|
Can you give at least the exception from the core dump? |
These values are technically from 2 different requests (although I imagine they should be pretty close) - one from initial gdb session over a core dump triggered by browser request, another one from a
If you're asking me, no, I cannot. This was a production server and I don't have access to it anymore. |
|
In my Case, the segmentation faults did not happen again since the restart this morning with forensic and security2 enabled. I have another 7 servers on which ModSecurity will also be used in the near future. Thanks again for the tip with log_forensic and how I can best proceed. Have a nice week. |
|
Ah no, a Heisenbug! But good luck, keeping my fingers crossed for you! |
|
@vizovitin / @Orgoth, can you give the exact exception from the core dump? |
It is a large production server, and core dumps are not enabled by default. |
Describe the bug
Segmentation fault in ModSecurity 2.9.7 for Apache on very specific POST requests.
Logs and dumps
I cannot provide full core dump or request parameters due to potential inclusion of information from a production server. However, here's the stack trace:
To Reproduce
N/A
Reproduced only on production server with very specific steps in a very customized WordPress.
ModSecurity for Apache, blocking, with Comodo (free) ruleset, in Fast mode, with 210710 and 222212 security rule IDs switched off.
Actual behavior
Request fails with 502 Bad Gateway served by nginx (reverse proxy before Apache). In the backend the following is logged:
Expected behavior
Apache children should not crash due to ModSecurity.
Server (please complete the following information):
Rule Set (please complete the following information):
Additional context
Unfortunately, I cannot test the issue on 2.9.8 (as it is production server I don't own).
The following patch alleviates the issue, although it is most certainly "incorrect":
The text was updated successfully, but these errors were encountered: