[Feature Request] pm support more delimiter

[daixiang0]

Describe the bug

OWASP uses pmFromFile file.data to cover static data, I have a case that uses WASM to call ModSecurity.

WASM does not support file actions, so I need to convert static data to a stringlike below:
pmFrom file.data ==> pm '(a)' '\"b\"' ...

file.data:

(a)
"b"

I guess pmFromFile uses line break as the delimiter, so a line in the file.data can include single quote, double quote, escape character and so on, while I need to convert file content to one string, the only way I knew is add escape char, which is hard to handle all cases.

As a rule may be like:

.... "pm 'a' 'b' " ...

Can pm support more delimiter like sed does( sed "s#a#b#", sed "s?a?b?"), it may help me.

As I am not the expert of ModSecurity, is there any better solution to do convert things or no need any convert?

[liudongmiao]

@pm and @rx cannot contains % (% is macro), " (as operator need to be double "). For @rx, we can use \x25 / \x22 to solve the problem.

And, @pm cannot support (space) or |(which will be converted to seperator), and cannot endswith \.

We can turn @pm to @rx to support %, ", (space), |, and \.

[martinhsv]

@daixiang0 ,

It sounds like the feature that you are suggesting is a duplicate of open issue #682 .

In terms of a workaround for your immediate needs, you can do as @liudongmiao suggested in the previous comment and use a regex instead.

[daixiang0]

For example, my data is like:

<title>=[ 1n73ct10n privat shell ]=</title>
>Ajax/PHP Command Shell<
.:: :[ AK-74 Security Team Web-shell ]: ::.
("a" 'b')

I need to use escape char to cover single quote and double quote first, then use escape char and double quote to cover each line like:

"pm \"<title>=[ 1n73ct10n privat shell ]=</title>\" \">Ajax/PHP Command Shell<\" \".:: :[ AK-74 Security Team Web-shell ]: ::.\" \"(\\\"a\\\" \\\'b\\\')\""

Is it what you suggest? Since when matching of a large number of keywords is needed, and my data includes hundreds of lines， @pm performs much better than @rx, I do not want to switch to use @rx.

[liudongmiao]

separate abnormal character %, ", (space), |, and \(followed by ") and normal character to different rules.

[daixiang0]

I want to do convert in all OWASP rules, separating them into different rules is an epic task for me :(

[liudongmiao]

def is_invalid_pm_value(signature):
    if '%' in signature or '"' in signature:
        return True
    elif ' ' in signature or '|' in signature:
        return True
    elif signature.endswith('\\'):
        return True
    return False

def xxx(rule_type, rule_signature):
     if rule_type == '@pm' and is_invalid_pm_value(rule_signature):
        rule_type = '@rx'
        for meta in '\\^$.[|()?*+{' + '%" ':
            rule_signature = rule_signature.replace(meta, '\\x%02x' % ord(meta))
     # then i have a function to combine all the @rx to a single one, crs has some example

[martinhsv]

Hi @daixiang0 ,

Regarding your observation about the superior performance of pm and pmFromFile over rx: you are correct that rx is less performant. However, such performance differences can often be mostly mitigated -- for example through the use of start or end anchors.

Another alternative that you might find more appealing (since it would simplify your rule(s) conversion):

Continue to use pmFromFile but use the alternative way to specify it, that gets the data from a url using http, rather than from a local file. This has been described to you a couple of months ago: #2676

As I said, in terms of the feature, there is already an open issue for it, which makes this a duplicate and should be closed. A couple of alternatives/workarounds have been suggested to you. It is, of course, up to you what you want to do with that.

[daixiang0]

@martinhsv thanks a lot, I would try and find the most suitable way for my case.

[jcchavezs]

Did you get somewhere with this @daixiang0 ?

[daixiang0]

@jcchavezs please check #2699 (comment). I am too busy to dig.

[jcchavezs]

@martinhsv thanks for the suggestion, tho:

Continue to use pmFromFile but use the alternative way to specify it, that gets the data from a url using http, rather than from a local file. This has been described to you a couple of months ago: #2676

About this, in some environments we have no choice other because in proxy-wasm-sdk we can only call envoy clusters (if we are in envoy) so we can freely specify a URL, also not sure what library for HTTP calls do modsecurity uses but most like that isn't allowed in the proxy-wasm environment.

I would really prefer to keep this as an open issue as there is not clear fix for things like proxy-wasm environments.

[martinhsv]

@jcchavezs ,

The reason this was issue was closed is because it is a duplicate of an open item. See earlier comment #2699 (comment) .

[jcchavezs]

Besides the feasibility of expanding the rules from @pmFromFile to @pm something that might be concerning is this post which describes the processing impact from running the entire rule from a string vs from a file. I wonder if we should treat this entirely different in modsecurity for WASM and attempt to do the request to external source to retrieve the data from a HTTP server (like @martinhsv points out) cc @M4tteoP