Search Multiple logical patterns in single ugrep command

[sharrythakur]

I am working on a log analysis task and need to search for multiple logical patterns within log files using ugrep. I want to combine all logical patterns in a single ugrep command and display which pattern was matched in the output. Here's an example of the command I am currently using:

ugrep -HPnrz -o -J=8 --format='%f----%n----%o----%G%~' -e '(?<error_85>version = 9.13.16604.0|version = 9.11.16404|version = 9.11.16405|version = 9.11.16407)' file.log

This command searches for any of the specified versions and outputs the results like this:

However, I have some patterns that need to use logical AND as well. For example, I want to search all files that contain all of these versions:

ugrep -HPnrz -o -J=8 --format='%f----%n----%o----%G%~' -e '(?<error_85>version = 9.13.16604.0 AND version = 9.11.16404 AND version = 9.11.16405 AND version = 9.11.16407)' file.log

How can I create a command that combines multiple logical patterns using -e pattern1 -e pattern2 and so on, and also includes in the output which pattern was matched?

Expecting command something like below
ugrep -HPnrz -o -J=8 --format='%f----%n----%o----%G%~' -e '(?<error_85>"This is error 100" | "This is Error 101")' -e '(?<error_86>"This is error 500" AND "This is Error 501")' file.log

Requirements:

1. Combine multiple logical patterns using -e options.
2. Output should display which pattern was matched.

[genivia-inc]

Use option -%% (--bool --files) and group patterns with AND using parenthesis. This is easier to visualize in the following simplified example to search for A or for B or for both C and D:

$ ugrep -%% 'A | B | (C AND D)' file

The short form is:

$ ugrep -%% 'A | B | (C D)' file

because pattern spacing is the same as AND.

Your query in Boolean form:

$ ugrep -%% -HPnrz -o -J=8 --format='%f----%n----%o----%G%~' '(?<error_85>"This is error 100" | "This is Error 101") | ((?<error_86>"This is error 500" AND "This is Error 501"))' file.log

Notes:

• Beware that option -e with --bool is confusing. It is ambiguous how the patterns are combined (are they combined as OR or as AND?). Actually, options -e typically form an AND when used with --bool to combine more constraints into one. But because it is ambiguous and not clear, this may change in the future.

• Beware that group capturing may not behave as we normally expect with one regex pattern. Boolean queries are normalized to conjunctive normal forms (CNF) shown with --stats:

  $ ugrep -%% 'A | B | (C AND D)' file --stats
  ...
  Files matched if:
    "C|A|B" matches on a line, and
    "D|A|B" matches on a line

  where each of the C|A|B and D|A|B is matched as a separate pattern anywhere in the file, which means we may end up with duplicate group capture indices. Also unnecessary parenthesis are removed in CNF, so (A) | (B) | ((C) AND (D)) is the same because (A) is just A. With option -P we can use (?<name>pattern) group captures instead, but this does not avoid the group capture duplication problem:

  $ ugrep -%% -P '(?<a>A) | (?<b>B) | ((?<c>C) AND (?<d>D))' file
  ugrep: error: error at position 41
  (?<b>B)|(?<d>D)|(?<a>A)|(?<b>B)
                       \___two named subpatterns have the same name (PCRE2_DUPNAMES not set)

  Perhaps PCRE2_DUPNAMES should be considered to work around this limitation.

[genivia-inc]

$ ugrep -%% -P '(?<a>A) | (?<b>B) | ((?<c>C) AND (?<d>D))' file

Perhaps PCRE2_DUPNAMES should be considered to work around this limitation.

I tried this out by setting PCRE2_DUPNAMES in the ugrep source code when Boolean queries are used. Now it works great to find and report named group captures. For example:

$ ugrep -%% -P '(?<a>foo) | (?<b>bar) | ((?<c>foo) AND (?<d>baz))' file?.txt --format='%f----%n----%o----%G%~' --stats
file3.txt----1----foo----c
file3.txt----2----bar----b
file1.txt----1----foo----c
file1.txt----2----baz----d
file2.txt----1----bar----b
file2.txt----2----baz----d

[sharrythakur]

Thank you Robert for all your time and responses to my question. I understood your explanation and it clarified a few points about logical operators.
However, I am still a bit confused about how to use ugrep to solve my log analysis problem. Could you help me figure out the best method to use?

Here’s a sample dictionary that stores all logical patterns. The dictionary key is pattern_id, and the dictionary value contains the pattern and its corresponding solution:
` {

'pattern_1': [("Error 100"|"Error 101"|"Error 102"), solution_1],

'pattern_2': [("Error 200" AND "Error 201" AND "Error 202"), solution_2],

...

'pattern_N': [("Error N00" AND ("Error N01" | "Error N02")), solution_N]

}`

I want to get the filenames where each pattern appears anywhere in the file. Ideally, I would like ugrep to return the pattern_id so that I can map it back to the corresponding solution in the dictionary.

The challenge is that I have a large number of files (around 400MB in total) and hundreds of such patterns to search for. Searching linearly is taking too much time.

Is there a way to construct a single ugrep command that can search for all patterns and return the pattern_id along with the filenames?

Thank you in advance for your help!

[genivia-inc]

Pattern matches can be found with ugrep, but it will be difficult to extract which AND-pattern matched, because group capture naming is associated with a single term that can be expressed as a single regex, such as "Error 200" and not with the full "Error 200" AND "Error 201" AND "Error 202" that is split up in three regex patterns, which means we get three group captures when written as ("Error 200") AND ("Error 201") AND ("Error 202"). Those terms like "Error 200" may also clash (duplicate) when we apply CNF. Or worse, you may get a different group capture name when the same term occurs all over the place, i.e. in many of these AND patterns.

Again, the reason for this difficulty is that regex pattern matching in general doesn't support an AND constraint and certainly not one that applies anywhere in a file. Ugrep uses regex pattern matching using each individual regex term and then combines the result to check the AND constraints.

Furthermore, if the dictionary of these AND-patterns is large, then you may want to implement a custom solution. When there are many AND-patterns then the normalization into AND constraints can be huge and slower than expected.

[genivia-inc]

To create a custom solution, here is what I would do:

• write code to collect all terms from the patterns, e.g. Error 200 is a term, remove duplicates from this set
• enumerate the set of terms so that each term has a unique index
• translate the dictionary to code with if to test which terms were present in a file, such as if (t1 or (t2 and t3)) found("pattern_8") where t1 , t2 and t3 are true if the term is found in the input
• write code to set t1, t2, ... tn to true if a term is in the file, then execute the code

It's more efficient for the last part to check each line in the input for matching terms so you don't scan the file more than once.

Writing these code parts can be automated with a code generator. If the dictionary changes then run the code generator again. For example, I would use RE/flex to write a scanner and parser to extract the patterns from the dictionary and generate code. If the dictionary is JSON, then use a JSON parser to do the same.

[sharrythakur]

Thanks for your time to help with the approach. Much Appreciated.
I am gonna go ahead and combine ugrep with a custom solution to resolve my problem.

[genivia-inc]

The fastest implementation that I can think of is in C++ with RE/flex to find patterns in lines and mark them when found e.g. mark in a bit array. The patterns can be found per line of input using regex (\QError 200\E)|(\QError 201\E)|... where \Q and \E delineate literal strings to match literally (since these pattern strings may contain meta chars such as . that should match literally). Then run the if-then code to check bits in the array. All of this can be auto-generated from the dictionary of patterns. It is also possible to implement this translator/auto-generator with a RE/flex lexer and parser.