False positive when charset=utf-8\x0d\x0a in content-type header #3046
Comments
|
When APP calls PATCH with some data inside then its fine, but when it POST empty data to endpoint |
|
It definitely looks like an illegal request to me. |
|
+1 to what @theseion said. It would be interesting to compare one of your working |
|
@theseion working on it, hopefully will get data soon |
|
So this is request what pass: this trigger modsec rule: In second PATCH there is no string DAILY but only |
|
Interesting! It looks like this could be a bug in ModSecurity. ModSecurity has a dedicated multipart parser that does some magic to detect / handle parts with |
|
Thank you @theseion. Happy to hear its not our application issue. Would you have some smarter idea how to temporarily exclude it without removing whole rule with |
|
You could write a new rule that looks for |
theseion
added
⚠️ ModSec
|
@theseion sorry to bother you again, but when I write down this exception it does not work: is this what you meant ? Similar condition is used by 922110 Rule, so I dont understand why it does not match in exception |
|
Your new rule must be processed before the rule you want to remove (see linked documentation). Is that the case? That means it has to run in the same phase or an earlier one and it needs to come before the other rule physically (in the same file or in a file that is read before BTW, to be safe I would match the header without case sensitivity, e.g. If you have the option, you can enable debug logging and you will see when and how your new rule is processed. |
|
@theseion thanks a lot for a hint. I've been all the time using phase 1, but in this case it has to be phase:2 as it is already part of the body. Now exception works well :) |
|
Great to hear. |
|
Sorry for the late answer, I could check this issue just now. So as I see the first comment isn't the issue, that's the correct behavior, right? @chladic's 4th comment contains two requests, but I couldn't reproduce the request which triggers rule 922210. Note, that in the second request's first line is a very long line, could you check that, is it correct? Also to @chladic: could you give us a |
|
Hi @airween. This very long line was probably caused by copy/paste issue. Didn't notice, sorry. What we got from Postman is such a curl: or So whenever we put in |
|
Thanks @chladic. You said above that the request contained |
|
@theseion before I took curl from postman I was not aware either, just assumed its null. But in this case it's empty string |
|
Thanks for the clarification. |
|
@chladic, thanks for the examples. Unfortunately I still can't reproduce the behavior what you mentioned. See my log lines: As you can see, I tried with two requests: and The only difference is between the two requests that in the second case I sent an empty string, while in first case I sent a "none" value. The triggered rule is the same in both cases: 920273, and the cause is a well-known libmodsecurity3 bug. Could you attach your relevant log lines after you sent those curl requests above? |
|
@chladic is it justified to keep this issue open? Is there anything else we can help with? |
|
@airween Ervin sorry for late reply, Im back on this issue. When doing curl Im also not able to reproduce this. When doing it from mobile application then Im triggering rule |
|
For what its worth, I get this error when a simple multipart form is posted with a text document in it. I can't give you a curl that will reproduce the issue, but this is a mildly redacted capture of a request that's triggering the rule. (Apologies if this is unrelated, especially if this is a scenario that the rule is meant to capture!) |
|
Hi @mikegoatly. Please open a separate issue for your question. It would also be very helpful if you could provide log output from ModSecurity. I don't immediately see why your request would be blocked, the log would tell us that. |
|
Hi @theseion Sorry, very happy to raise a separate issue - the reason I mentioned it here is that the ModSecurity log output is exactly same as this issue:
My observation is that my and the original request that @chladic posted both contain multi-part form data in the body, and the false positive trigger seems to be something to do with the fact that a The warning seems to suggest that the parsing of that additional content type header is not excluding the trailing CR/LF from the string it's trying to match (e.g. Edit: Actually, just a thought: At what point are the CR/LF being escaped to |
|
I see. Let's keep it here then. You might be right that We haven't been able to recreate the issue so far, so it looks like it has to be a specific request format. Given that you are able to capture the request that triggers the issue, could you try and capture the actual bytes in the request (with |
|
Ok, this request fails whenever I throw it against an API fronted by ModSecurity: POST https://YOURURLGOESHERE/
Content-Type: multipart/form-data; boundary="----x"
------x
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data; name=title
Test
------x
Content-Disposition: form-data; name=file; filename="New Text Document.txt"; filename*=utf-8''New%20Text%20Document.txt
Some sample text
------x--
(It's in the I've tried the request with both CRLF and LF line endings, both fail. Stripping the POST https://YOURURLGOESHERE/
Content-Type: multipart/form-data; boundary="----x"
------x
Content-Type: text/plain;
Content-Disposition: form-data; name=title
Test
------x
Content-Disposition: form-data; name=file; filename="New Text Document.txt"; filename*=utf-8''New%20Text%20Document.txt
Some sample text
------x--
I've attached the failing request so you can see the exact content that's being sent from my machine: request.zip Does that help? |
|
That's awesome! Thanks a lot @mikegoatly. Well try to reproduce the issue on our end. |
Let me check that it soon. |
|
@mikegoatly, I tried to reproduce your issue, also with your suggested way (REST client for VSCode - thanks for the tip), but no luck.
now I made a small Python script which sends the raw format of your request, you can see that here. First, please check that, and let me know if there is something not well. Then could you try it against your application? (I see you use HTTPS, but actually in my dev env I do not have any HTTPS, and it's more convenient to try the raw requests like this without HTTPS). This is what I see when I run the script: and this is what I see in my log: or with Nginx+libmodsecurity3: So as you can see there is only one rule triggered in phase 2 with id 200002 (with both implementations): and I'm using CRS with Apache 2.9.7, and Nginx 1.18 and libmodsecurity3 3.0.8. Could you check that script with your application, and align it to reproduce the issue? |
|
Thanks @airween - I was unable to reproduce the problem using variations of your script, so I figured it must be something to do with the way that the request is being formed. I had a play with a different way of building the request using Python's The script I ended up with is: import requests
files = {'file': ("New Text Document.txt", "Test", "text/plain; charset=utf-8")}
response = requests.post("http://testapi/test", files=files)
print("REQUEST:")
print(response.request.body[:200])
print("")
print("RESPONSE:")
print("STATUS: %s %s" % (response.status_code, response.reason))
print("CONTENT:")
print(response.text)Running this for me gives the output: The logs show the exact errors and warnings that have been reported here. And if I remove What happens in your environment if you make a request using that script? |
|
@mikegoatly - thanks for script. One of the reason why I didn't want to use Python's request module that it triggers the rule 913101: The other one is that request formats the HTTP request as well, therefore I can't send any invalid request :). So I changed the ... files=files, headers={'User-Agent': 'PyTest Client v0.1'})and run the script. No rule was triggered. None of them. Note: in my dev env I have CRS v4.0/dev. Which version of CRS do you use? |
|
OK, silly question time - I'm running ModSecurity as part of ingress-nginx. What's the best way to find that version number for you? |
|
|
|
@mikegoatly - thanks. I was able to reproduce the issue on Nginx + libmodsecurty3 3.0.8 and CRS 3.3.4. Let me investigate the reasons, why happens this (and why does not with Apache). |
|
Thank you for your patience. Looks like libmodsecurity3 handles buggy the multipart content types (too?), because it keeps the trailing white space chars, namely What can you do now? I've found a workaround, this patch worked for me: diff --git a/rules/REQUEST-922-MULTIPART-ATTACK.conf b/rules/REQUEST-922-MULTIPART-ATTACK.conf
index 1bf5a03..7c2fd36 100644
--- a/rules/REQUEST-922-MULTIPART-ATTACK.conf
+++ b/rules/REQUEST-922-MULTIPART-ATTACK.conf
@@ -53,7 +53,7 @@ SecRule MULTIPART_PART_HEADERS "@rx ^content-type\s*+:\s*+(.*)$" \
phase:2,\
block,\
capture,\
- t:none,t:lowercase,\
+ t:none,t:lowercase,t:removeWhiteSpace,\
msg:'Illegal MIME Multipart Header content-type: charset parameter',\
logdata:'Matched Data: %{TX.1} found within Content-Type multipart form',\
tag:'application-multi',\
@@ -67,7 +67,7 @@ SecRule MULTIPART_PART_HEADERS "@rx ^content-type\s*+:\s*+(.*)$" \
severity:'CRITICAL',\
chain"
SecRule TX:1 "!@rx ^(?:(?:\*|[^\"(),\/:;<=>?![\x5c\]{}]+)\/(?:\*|[^\"(),\/:;<=>?![\x5c\]{}]+))(?:\s*+;\s*+(?:(?:charset\s*+=\s*+(?:\"?(?:iso-8859-15?|windows-1252|utf-8)\b\"?))|(?:(?:c(?:h(?:a(?:r(?:s(?:e[^t\"(),\/:;<=>?![\x5c\]{}]|[^e\"(),/:;<=>?![\x5c\]{}])|[^s\"(),/:;<=>?![\x5c\]{}])|[^r\"(),/:;<=>?![\x5c\]{}])|[^a\"(),/:;<=>?![\x5c\]{}])|[^h\"(),/:;<=>?![\x5c\]{}])|[^c\"(),/:;<=>?![\x5c\]{}])[^\"(),/:;<=>?![\x5c\]{}]*(?:)\s*+=\s*+[^(),/:;<=>?![\x5c\]{}]+)|;?))*(?:\s*+,\s*+(?:(?:\*|[^\"(),\/:;<=>?![\x5c\]{}]+)\/(?:\*|[^\"(),\/:;<=>?![\x5c\]{}]+))(?:\s*+;\s*+(?:(?:charset\s*+=\s*+(?:\"?(?:iso-8859-15?|windows-1252|utf-8)\b\"?))|(?:(?:c(?:h(?:a(?:r(?:s(?:e[^t\"(),\/:;<=>?![\x5c\]{}]|[^e\"(),/:;<=>?![\x5c\]{}])|[^s\"(),/:;<=>?![\x5c\]{}])|[^r\"(),/:;<=>?![\x5c\]{}])|[^a\"(),/:;<=>?![\x5c\]{}])|[^h\"(),/:;<=>?![\x5c\]{}])|[^c\"(),/:;<=>?![\x5c\]{}])[^\"(),/:;<=>?![\x5c\]{}]*(?:)\s*+=\s*+[^(),/:;<=>?![\x5c\]{}]+)|;?))*)*$" \
- "t:lowercase,\
+ "t:lowercase,t:removeWhiteSpace,\
setvar:'tx.anomaly_score_pl1=+%{tx.critical_anomaly_score}'"
# Content-Transfer-Encoding was deprecated by rfc7578 in 2015 and should not be used (see: https://www.rfc-editor.org/rfc/rfc7578#section-4.7)so added the (ping @fzipi as author of the rule: what do you think? Does it make sense to add this permanently?) If you want to avoid the unwanted other |
|
I guess we could use |
|
Thank you everyone for finding solution to this. Really appreciate |
I try to find the bug and send a PR to fix it. If do not have time, I just open an issue. |
|
Just FYI: the sent patch above for libmodsecurity3 has been merged. |
Hello CRS team,
some actions in my mobile application are triggering this rule and block request. I didnt set in APP charset=utf-8\x0d\x0a,
so maybe its coming from android.
When I exclude this rule
SecRuleRemoveById 922110then its fine, but I want to exclude it for everythingand could not figure out any exception:
I tried this:
tx.allowed_request_content_type_charset=|utf-8| |utf-8\x0d\x0a| |iso-8859-1| |iso-8859-15| |windows-1252|'and also this:
I checked 922110 rule and it cant match above with regex defined there.
Anyone can help me to understand this issue please ?
Many thanks
ModSecurity: Warning. Matched "Operator `Rx' with parameter `^(?:(?:\*|[^\"(),\/:;<=>?![\x5c\]{}]+)\/(?:\*|[^\"(),\/:;<=>?![\x5c\]{}]+))(?:\s*+;\s*+(?:(?:charset\s*+=\s*+(?:\"?(?:iso-8859-15?|windows-1252|utf-8)\b\"?))|(?:(?:c(?:h(?:a(?:r(?:s(?:e[^t\"(),\/:;<=> (714 characters omitted)' against variable `TX:1' (Value: `text/plain; charset=utf-8\x0d\x0a' ) [file "/usr/local/coreruleset-3.3.4/rules/REQUEST-922-MULTIPART-ATTACK.conf"] [line "51"] [id "922110"] [rev ""] [msg "Illegal MIME Multipart Header content-type: charset parameter"] [data "Matched Data: text/plain; charset=utf-8\x0d\x0a found within Content-Type multipart form"] [severity "2"] [ver "OWASP_CRS/3.3.4"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "OWASP_CRS"] [tag "capec/272/220"] [tag "paranoia-level/1"] [hostname "10.151.0.2"] [uri "/something.json"] [unique_id "167059009818.061175"] [ref "o0,41o14,27v974,41t:lowercaset:lowercase"],The text was updated successfully, but these errors were encountered: