Let's talk about the road ahead

[plusvic]

YARA is about to become 15 years old. During all these years a lot of decisions were made, many features got implemented, and many ideas got discarded for one reason or another. The project is now mature and stable, and a great community of users and contributors has blossomed around it. I feel it's time for laying out the plans for the next 15 years of YARA. It's time for dreaming big again.

I'm opening this discussion with the intention of starting a collective brain-dump that helps us shape the future of YARA. This is a place for describing the features and ideas that you would like to see in a future incarnation of YARA, no matter how crazy they sound. I'm particularly interested in the following topics:

• Existing features that you have never used.
• Features that you really miss.
• New modules that should be implemented.
• Pain points, common use-cases where the current solution is awkward or unnatural.
• Design decisions both in the language or the API that you believe are wrong or could be improved.

The main purpose of this discussion is getting a grasp of how the community is interacting with YARA, what they miss and which are they main sources of frustration. Don't be shy, all contributions are welcomed.

[plusvic]

Implement the in operator

Right now YARA allows $a in (0..100), which returns true if pattern $a appears at some offset between 0 and 100 in the file. It also allows #a in (0..100) which returns the number of matches of $a with offsets between 0 and 100. However, there's no way of saying "foo" in module.list where module.list is a list of strings, or in general anything that is iterable. The current alternative is...

for any s in module.list : ( s == "foo")

... which is very verbose.

An alternative is implementing in that returns true if the value of some expression <expr> is contained in an iterable <iter>. The syntax would be like:

<expr> in <iterable>

However this makes $a in (0..100) ambiguous, as this expression could be interpreted both with its current semantics but also like... the result of expression $a, which is a boolean, should be contained in the iterator returned by the range (0..100). This expression wouldn't be valid, as a boolean can't be contained in an iterator of integers, but things get worst with #a in (0..100), where #a is an integer, and therefore #a in (0..100) could be validly interpreted as... the integer returned by #a should be contained in the iterator returned by (0..100). This interpretation conflicts with the current semantics.

[metthal]

I guess only arrays of primitive types are going to be able to utilize this right?

[plusvic]

Yes, only arrays of primitive types.

[plusvic]

Allow modules that unpack/uncompress data

Having a mechanism for allowing file unpacking/decompression within YARA itself would be great. For example, a module could implement ZIP decompression, and it would uncompress any ZIP file scanned by YARA, allowing to scan the ZIP's content as well.

[hillu]

Do you see this as a library feature or something that would be implemented within the YARA CLI program?

[plusvic]

As library feature, something like a special kind of modules that receives an input buffer and returns potentially multiple output buffers that YARA scans in addition to the original data.

[plusvic]

Composable modifiers

Implement composable modifiers as described in: https://gist.github.com/wxsBSD/44aa8b8133e3ea96e738b66ec1c600f2

[wxsBSD]

Actions Clause

I've got a friend who has repeatedly tried to get me to implement an "actions" clause in rules. The idea is that we could tie together detections with actions to be taken directly in the rule. Currently this is implemented in various systems I've helped build (or know about) over the years outside of the language, and while it works well it would be great if we supported this in the language.

I'm thinking of something like this:

rule a {
  strings:
    $a = "FOOBAR"
  condition:
    true
  action:
    cmd = "python3"
    args = ["decoder.py", $a, uint32(filesize - 4)]
}

I've always thought being able to ship rules and a script to do something with them automatically is a great idea but have not implemented because I thought it was a stretch for the current design. Obviously there are considerations to be made around the safety of this too.

I think supporting this might help take YARA to a new level for the power users.

[plusvic]

I feature like these could be useful, but I'm not quite convinced about putting that into the rules themselves. Adding actions to your rules make them very dependant of your environment, the actions won't work properly unless you have the right programs at the right paths, so your rules start making sense only to you.

However we could implement a similar idea via more powerful tooling, instead of extending the language. For example, the command-line tool could accept a config file where you establish a relation between a rule name and some program to be executed, if rule foo matches, run program bar. The issue with this approach is how to convey information like the value of filesize - 4 to the outside world. Maybe we can use the local variables for that, when YARA finds a matches it could output the value of local variables to the outside world. So, you can use YARA not only for knowing which files match which rules, but also for exposing information extracted from the files.

[wxsBSD]

I feature like these could be useful, but I'm not quite convinced about putting that into the rules themselves. Adding actions to your rules make them very dependant of your environment, the actions won't work properly unless you have the right programs at the right paths, so your rules start making sense only to you.

I can certainly see that point. I was thinking it would be a run-time option that you have to enable, and/or we could have a #pragma that allows you to toggle it on or off at compile time. One downside of putting it in the language is that it does make the rule less concise. If you aren't running the actions you probably don't want to see it.

I don't think there's any way around the problem of the actions not working properly on your system, regardless of extending the language or building better tooling. If an action is a python script that requires a specific module to be installed it doesn't matter if the language is extended or better tooling, the action isn't going to work unless you have that module installed.

However, I do see an argument to be made for keeping them separate as it helps keep the language cleaner and lets us evolve the two systems separately, which is nice.

However we could implement a similar idea via more powerful tooling, instead of extending the language. For example, the command-line tool could accept a config file where you establish a relation between a rule name and some program to be executed, if rule foo matches, run program bar. The issue with this approach is how to convey information like the value of filesize - 4 to the outside world. Maybe we can use the #1781 (comment) for that, when YARA finds a matches it could output the value of local variables to the outside world. So, you can use YARA not only for knowing which files match which rules, but also for exposing information extracted from the files.

The idea of putting it directly in the language is that we have the features of the language to do initial processing of these arguments. The compiler could output the necessary bytecode for uint32(filesize - 4) and pass the result as an argument to the script. For $a as an argument we could output the contents of the match as an argument. However, this won't work well for regex or hex strings with jumps or ranges where the content can change across matches.

I can see how it isn't a great syntax for expressing this, and when combined with things like local variables I am starting to come around to the idea of better tooling and exposing the appropriate things to the action outside of the language.

In any case, I think we are getting down in the details but I am glad you like the idea here. We can hash out the details when the time is right. :)

[metthal]

Just to throw in my 2 cents. I don't really like throwing random commands into the rules themselves but I see a good point in this

The compiler could output the necessary bytecode for uint32(filesize - 4) and pass the result as an argument to the script. For $a as an argument we could output the contents of the match as an argument.

I have heard some requests from people wanting YARA to output something else other than hit/not-hit. Now that we have console module, the option is kind of there but the issue is that it's heavily suited towards working with strings. Maybe there needs to be even more generalized console module which gets the callback like CALLBACK_MSG_CONSOLE_LOG but instead of always getting string, it's capable of obtaining arbitrary data of any type. What this wouldn't get rid of though it that it would still be present in the rule itself.

If we'd like to get rid of that, what I think could come useful here are Local variables in its original proposal with their own section. You could have a general callback for each rule and obtain list of all variables and their values with this callback. I can't think of any other way to solve this.

Maybe someone else can iterate on this idea even further :)

[wxsBSD]

Compiler Improvements

One of the issues with current YARA is that it is difficult to use the compiler as anything other than a way to output the necessary structures and bytecode for execution by libyara. Being able to have programatic access to the rules as an AST is helpful for managing complex rulesets and possibly even allowing for optimizations that are specific to certain user environments (if you know you're going to only be scanning PE files with this run then if you have a way to programatically recognize ELF specific rules then you can remove them just for this run).

This would also allow us to have one true compiler instead of splitting efforts between gyp and the main compiler. I don't have specific ideas here but am hoping others can share their ideas here.

[hillu]

Loadable modules

Modules that extend YARA's core pattern matching functionality should be decoupled from the library, so they can be developed separately.

1. Loading those separately developed modules as shared objects/DLLs should be possible
2. It should also be possible for programs embedding libyara to bring along their own modules.
3. Implementing them in scripting languages (Python, Lua?) should be an option for users who don't feel comfortable developing in the same language the next version of YARA is going to be written in.

For the YARA 4 codebase, I have begun working towards such a goal (see #1772).

[plusvic]

I would like to hear more about real-life use cases for dynamically loadable modules. Certainly it's a nice-to-have feature, but I would like to understand real cases in which statically-compiled modules are not enough.

I can imagine a situation in which someone provides a software that have YARA embedded and the users of this software can provide their own YARA rules. In such a situation the users may also want to extend the software with their own modules, but don't have the possibility of rebuilding the software. Is that kind of scenarios what you have in mind?

Dynamically loadable modules makes the module system a bit more complicated, as YARA and modules must define how they talk to each other at the ABI level, for example by using C calling and naming conventions.

[hillu]

All the use cases I have come up so far come down to easier integration / extension, I'll try to phrase them as more concrete examples:

Colleagues have recently expressed interest in trying out the a-ray-grass module. While providing patched YARA binaries are not of a problem for me personally, things would generally be easier if that could be skipped and we could just copy a .dll or .so file into place and start writing rules.

Somebody has pestered me for years (at this point it might well be a running joke) how he would really like a component document format (old MS Office etc.) parser to be added to YARA and if I'd like to write one. My main reason for not doing this is that I didn't feel like writing a parser for such a messy format in C. For prototyping, Python may have been ok. For use-cases with more strict performance requirements, I might have chosen Go in the past, nowadays I would go with Rust. All those options would have required considerable changes to the build system.

Reading about your idea to enable rules to unpack data and @methal's data extraction idea made me think that both would be perfect matches for loadable extension modules. In the messy malware world, unpacking/uncompressing does not only deal with extracting stuff from compressed zlib streams or similar. Often deobfuscation or configuration extraction is just as important and being able to add ad-hoc routines for the task at hand would be a useful capability for doing analysis.

[metthal]

Local variables

In its current form, YARA already supports variable on a global level, by providing a list of external variables. I think having local variables on a per-rule basis would bring some benefits, such as:

1. Readability - complicated conditions which reference several nested structures in pe and dotnet modules can become really messy if used extensively within the rule condition.
2. Performance - local variables are going to be evaluated only once (even could be evaluated lazily when the variable is referenced for the first time) and their value is going to be stored. If you need to use certain value in your condition multiple times, you can store it somewhere and then load it on each reference to a variable. This assumes the current VM implementation, so some of this might be addressed by new VM.

The local variable is valid only within the scope of the rule. One local variable can reference other local variable.

Example:

variables:
    last_section = pe.sections[pe.number_of_sections - 1]
condition:
    last_section.name == ".tls" and
    last_section.characteristics == 0xC0000000 and
    uint32(last_section.raw_data_offset) == pe.image_base + last_section.virtual_address + 0x20 ...

The implementation was already presented in this PR but has been abandoned for a long time.

[metthal]

That sounds great and I love that. I guess if we put it together with thread about external variables then it all fits together.

[plusvic]

This sounds very promising, global and local variables, where values for global variables can be set in the source code itself, or overridden from the outside world as external variables work now.

[plusvic]

Idea: Instead of declaring the local variables at the rule level, introduce a statement that creates a new scope where certain identifiers have predefined values. Example:

condition: 
    with foo = <expr>, 
         bar = <expr> : (
       <a boolean expression where you can use identifiers foo and bar>
    )

The advantage is that you can use this inside a loop:

condition: 
    for all x in <iterator> : (
      with foo = <expression where you can use x>, 
           bar = <expression where you can use x> : (
         <a boolean expression where you can use identifiers foo and bar>
      )
    )

[metthal]

Hm, I kinda like this because it solves the problem with loops which can't really be expressed within its own section.

[plusvic]

RFC for the with statement: #1783

[hillu]

External variables should be defined within the rule language

Working with external variables (-d var=value on the command line, yr_compiler_define_{float,integer,boolean,string}_variable in the API) is cumbersome. There should be a simple define variable statement in the rule language so rulesets can be self-contained.

[metthal]

Namespaces on the language level

Similar to the external variable should be defined within the rule language, I also think the same about namespaces. Namespaces defined on the language level would make it much more easier to share rules and avoid name clashes without having to rely on the user compiling everything in a separate namespace.

Example:

namespace foo {
// ...
}

[plusvic]

Avoid data scanning if not necessary

The current implementation scans the data first and then evaluate all the rule conditions. This has been a source of frustration for many users, who expect that a condition like filesize < 1MB and $b should return immediately with files larger than 1MB, without scanning the whole file looking for $b.

[regeciovad]

Is it possible to broaden this to how much memory is being scanned? For example, I recently was helping one of our analysts to write a rule that would check the first XYZ bytes in the samples. His first version looked like this:

rule check_sample
{
  strings:
    $re = /[A-Za-z]{XYZ}/
  condition:
    $re at 0
}

This was, of course, generating warnings. However, if YARA could first analyze that the match has to start in a fixed position, this could make the scanning process much faster, even with regular expressions.

[plusvic]

Sure, that's something that it's worth doing for sure. One solution that I have in mind is having a list of patterns that are used exclusively in expressions like $a at <offset>, where <offset> is known at compile time.

In such cases the pattern $a wouldn't be added to the Aho-Corasick automata, instead it would be added to a list of patterns sorted by offset, every time we advance one byte while scanning the file, we compare the current offset with the offset at the top of the list. When they match, we proceed to check for the patterns that should be found at that offset.

[plusvic]

Already done in YARA-X.

[metthal]

User-defined arrays and their unification with iterators

Very hard thing to express (but not impossible) in YARA is "N of M expressions are evaluated to true". What I propose is to have a way for users to define user-defined arrays which would cooperate with operator of, giving a possibility to express N of [<expression1>, <expression2>, ...].

At the same time, I see an opportunity to unify it with existing concepts like integer enumerations which are now denoted the same way as string sets and rule sets. When you even combine it with operator in, everything fits together perfectly. Rule enumerations can be removed as they are and replaced with user-defined arrays full of integer literals.

[plusvic]

Note about possible implementation: The of operator could be implemented as x of <iterator of booleans>, where <iterator of booleans> is any kind of iterator that returns boolean values. It should work with a tuple of boolean expressions like (<boolean expr>, <boolean expr>, ...). This automatically allows x of (rule1, rule2, rule3), which is already supported by YARA.

The case x of <string set> could be reduced to x of <iterator of booleans>, because $a, $b, etc are already boolean expressions, and ($foo*) could be viewed as a self-expanding tuple where $foo* is automatically replaced by all the matching identifiers. The variant x of <string set> in <range>, is a special case where<string set> is a tuple that can contain only string identifiers, not boolean expressions in general.

However the case x of (foo*) becomes a problem with this approach. Self-expanding tuples are ok for string identifiers that are local to the rule, but expanding foo* into a sequence of all identifiers that match foo* looks like too much complexity for relatively little gain. It would be implemented this way if backward-compatibility is a must.

This approach still requires some exploration, for example for verifying that we can express all of this in the grammar without conflicts.

[plusvic]

Related to #1338

[metthal]

YARA as a data extraction tool

YARA is doing complex data processing through modules like .NET, PE, ELF, Mach-O etc. Some of the data these modules extract is currently available through -D command line options but you can't get the output of function calls so for example you can't figure out pe.imphash() just by looking the the output of -D. Also the output of -D is not really machine-readable. What I propose is to have:

1. This output machine-readable
2. Run YARA without actual scanning while these data are extracted for you

I can see this being achieved by modules being its own standalone modules which you can use to obtain all the information. In the same way yara and yarac now exists, we could have more tools like this. One of them could be yaradump (or whatever name someone can come up with) that would work as an data extractor. The reason why anyone would need such a thing would be to process what YARA sees on the particular sample without relying on 3rd party tools (we know how PE parsers differ based on implementation, especially on exotic malware samples)..

This kind of resembles what loadable modules is talking about, but because the use-case is a little different, I wanted to list it out as a separate thing.

[plusvic]

Already done in YARA-X

[metthal]

Extensible rule sections

What if modules could extend the rule language by providing new sections into the rules? Sections meta, strings and condition would be provided by default (maybe some other that are going to exist in a new YARA). Each module would be able to define their own sections. The way I imagine it is for example cuckoo module providing cuckoo.mutexes section. Each section would also put a reserved word into the rule with the same name, that could be used for example to express N of <reserved_word> so in case of cuckoo it would be all of cuckoo.mutexes, any of cuckoo.mutexes, 3 of cuckoo.mutexes etc. The sections would be always in the format key = value so no extensions to the grammar itself. The logic would be provided by the module, what exactly it means to perform of operator on the reserved word of that module.

The reason I would like to see something like this is that I never really liked the fact that all the Cuckoo module features have to be functions and they need to be in the condition. I would welcome more declarative approach with magic happening in the backend. I focus mainly on the cuckoo module but maybe there are also other uses for other modules. Some example that I thought of just now is pe module providing pe.sections and then you could write

pe.sections:
   section1.name = ".text"
   section1.virtual_address = 0x400000
   section2.name = ".tls"
condition:
   all of pe.sections

I realize that this is maybe too much, but I am throwing out there to see the reactions :)

[plusvic]

IMHO, this adds a lot of complexity with very little gain. I agree regarding to the cuckoo module, I don't like the fact that everything is a function. But I see it as a problem with the design of the module itself.

[wxsBSD]

Better Callback API

The current callback mechanism could be easier to use. Currently you provide a single callback and have to handle a lot of logic in that single callback. It would be nicer if libyara took a structure of function pointers for the callbacks and called them automatically. I'm thinking of something like this:

YARA_CALLBACKS callbacks = {
    callback_matches,
    NULL, // We don't want to handle non-matches
    callback_module_info,
    ...
}
...
yr_register_callbacks(&callbacks);

This would make the interface more explicit for users, allowing them to handle only the events they care about, at the cost of a bit more logic inside libyara to figure out when to call each callback (if non-null).

[metthal]

I'd even say that having callback stack for a every single message would be the best, similar to how Linux Security Modules work in Linux kernel if anyone is familiar with the kernel. Do you want your hits to be logged to stdout? First callback in the stack. Do you want your hits to be sent over the network as some sort of event? Next callback in the stack. It's much more composable and even the new libyara could provide some basic ones which can provide the most basic functions like log human-readable to stdout, log JSON to stdout, etc.

This is how I imagine it, you define CALLBACK_TABLE structure with the callbacks you want and push it onto the callback stack.

CALLBACK_TABLE json_print_table = {
    .hit_callback = print_json_hit,
    .warning_callback = print_json_warning
};
yr_push_callback_table(&json_print_table);

CALLBACK_TABLE send_hit_over_network_table = {
   .hit_callback = send_hit,
   .warning_callback = send_warning
};
yr_push_callback_table(&send_hit_over_network_table);

[wxsBSD]

Pluggable Output

Having different output options built into yara (the CLI) would be awesome. We can separate out the ideas of formatting from destination. By making a separation of the formatting from the destination we can start to support things like "log in JSON to stdout" or "log in text to windows event log" or something like that. Ideally we would provide an API to let users of libyara register their own formatting and destination plugins so they can better route output.

It is a little unclear to me how this would fit in with the callback API discussed in a different comment but having different options for formatting and destination would be very powerful for places that are embedding YARA into their analysis pipelines/products.

[metthal]

Debugging of YARA

Finding out why your YARA rule doesn't hit can be kinda hard. Before console module being introduced, the only real way to debug YARA was through commenting out parts of rules and looking at the output of -s and -D. console module certainly helps, but I still think it has a lot of drawbacks:

1. Supports only printing of primitive types, so printing complex structures can get pretty verbose
2. Requires modification of rules. Even though I practice printf-like debugging myself sometimes, I don't do it all the time. I also wouldn't expect YARA rule authors to do it all the time.

What I think YARA would benefit from is having the full debugging support which would allow you to:

1. Debug YARA without modification of the rule
2. Step through the code as with any other debugger
3. View all the module data
4. View all the string matches
5. Evaluate immediate expressions (short snippets that could be put into condition part of YARA rules)

The debugger itself would provide Debug Adapter Protocol server so that it can be easily integrated into Visual Studio Code, and other IDEs/editors supporting DAP.

I already have working prototype for everything mentioned above (besides the point 5) including DAP support, but all of it is being shoehorned into YARA. If YARA were to be basically reimplemented, I would really welcome it to be something that's being thought of when designing the new core and I can certainly provide guidance on what is required for DAP to work properly.

[hillu]

For the current implementation, the core of such functionality could be easily implemented by adding a different callback type that is called from the bytecode interpreter before executing an instruction.

[metthal]

Partly. Debugger needs to be able to control the execution, so stop it for a moment. It also needs to be able to tell how many instructions to execute (stop over usually executes just the current line) and then stops. The way I implemented in my prototype is through extracting the execution of a single instruction into a separate function, so that the execution loop and execution of an instruction are in separate functions. Execution loop then just calls this function to execute an instruction. Debugger also calls it too, but it has its own logic for the execution loops. This way you can do it in a single thread without too much hassle. I was thinking about this when I was implementing the prototype and this was my conclusion.

[metthal]

Btw. I am going to share the prototype once I'm able to clear the code somehow and also integrate it into the current version of upstream YARA.

[plusvic]

A way for obtaining the length of an array/iterator

Some modules expose arrays like pe.sections, and many of these arrays are accompanied by another field that indicates the size of the array, like pe.number_of_sections. However that's not true in all cases, like vt.behaviour.files_dropped. A built-in function length(<iterator>), or a method <iterator>.length() would be very useful.

[wxsBSD]

This should be implemented for dictionaries too.

[metthal]

Standardization of the language/changes to the language

I know that this isn't really a feature request for YARA per-se but I think it fits the fact that big changes are coming into YARA and therefore I think it should be worth it to discuss this. As development of YARA is gaining more and more traction, and more and more people contribute, I think it's becoming really hard to keep up with all the changes and plan any kind of updates to YARA in production environments. A lot of bug fixes are interleaved with big breaking changes, sometimes on a day-by-day basis. Since the language itself is dictated by the implementation, when the changes reach the code it also changes the language and it's sometimes too much to track.

I really think a lightweight form of Rust RFCs might be a good example of how to deal with changes to the language. Things that it could see it would bring:

1. Direction - presence of RFC does not say anything about the implementation, so there could be an RFC for a specific feature while no-one has really implemented it yet. This paints clearer picture of where is YARA heading as new features are pre-approved before they are implemented.
2. Easier to jump-in - if anyone wants to start with development of YARA but they are not sure about implementing something, if you have RFCs that are not implemented yet but are approved, it's much more easier to jump-in and get involved.
3. Record keeping - Lately, Keybase has been a great place where discussions about potential features has been happening. But since it's more or less IRC with history, things will get lost, forgotten, because they just simply scroll-up. Moving things to GitHub issues is much better for record keeping, but GitHub issues doesn't really tell you anything about the feature being approved/in progress/not approved.
4. Distributing workload - Lately, me and my team started to get involved with upstream to the full extent. If we had a clearer picture of where is YARA heading, what are the plans, what kind of features are being planned etc., I feel like we could be much more helpful with the upkeep of the YARA repository. Sometimes we introduce changes to the language which are not really accepted here. Maybe we could have brought those features to the table first and that's on us, but that kinda brings us to the points I mentioned earlier, where if the whole process was defined like this, it would be much easier to maintain course.
5. Documentation - currently the documentation of YARA is more or less based on examples and full-text descriptions. Many times it suffices but other times, nuances can be easily lost. For example if I want to know the details of the operator of, my best shot is directly looking at the code.

I don't ultimately call for having a separate RFC repository and to have complicated processes for every single change to the language. Even having approved label in GitHub issues would be kinda step in the right direction to know which feature requests have been approved and are about to be implemented. Another would be markdown/RTF files describing these new features and how exactly they should interact with everything else in the language. I think this could also heavily help with this whole "New YARA" initiative, since there are a lot of topics that are touching the same subject from different directions and one might not see all the details in their head right away. Having it written as a "(almost) formal" proposal would allow us to define these fine details. Thoughts?

[plusvic]

I think that a lightweight RFC process can be very helpful, specially with changes in the language itself. It's an opportunity to expose the idea to more eyeballs, and identify potential issues or better alternatives. I plan to draft a couple of such RFCs as a starting point for this initiative.

[plusvic]

The very first RFC: #1783.

[tlansec]

Add possibility to use full set of data types for Externals

At $dayjob we use YARA's Externals feature to scan objects with YARA passing in a lot of metadata about those objects. One of the limitations of this is that the Externals feature only supports str/int fields, but sometimes this means fudging the data to fit these limited data types, e.g. when really the object is a list, we have to change it to a pipe delimited string for matching purposes.

It would be nice if lists/dictionaries were available to use in conjunction with externals.

[malvidin]

Would better integration of protoc-gen-yara modules cover this use case?

[tlansec]

Adding to this, another thing that isn't possible for Externals is to pass a None value (i.e. check if variable is defined).

I misesd the comment above at the time - sorry for the late reply. I had not seen the protoc-gen-yara module before, and it does look pretty useful! We have generally used Externals over writing our own YARA modules in the past for a few reasons:

1. Concerns that we'd write a module of our own, and then find that it breaks when changes are made to YARA itself, causing extra engineering lift.
2. Difficulties associated with deployment of a custom module vs usage of externals ++ python utilities (e.g. deploying to lots of different OS's including normal end users).

I think that for some of the cases we're using custom modules for, the protoc-gen-yara definitely makes sense though - and thankyou for pointing me towards the project.

[plusvic]

Allow disabling/enabling specific types of warnings

Currently YARA has the --no-warnings command-line argument for disabling all warnings. However, sometimes you may be interested in disabling only certain types of warning while leaving the rest enabled, just like you can do with most C/C++compilers.

One solution is following the model used by gcc, where each type of warning have an identifier xxxxx and can be enabled with -Wxxxxx and disabled with -Wno-xxxxx. This allows certain warning enabled by default, and can be selectively disabled, while others can be disabled by default and can be selectively enabled.

[regeciovad]

The future of the Aho-Corasick algorithm and regex engine

This is more an open question than a proposal for some future version of the Yara. I read through the suggestions in this thread, which are amazing and exciting. Most of them are, however, additional features, understandably. I would also like to discuss the future of pattern-matching implementation, used algorithms, and heuristics even. @plusvic, are they any plans for this?

[plusvic]

I don't have any specific plans, but I'm open to fresh ideas in this regard. So far, my impression is that the current model of splitting the pattern in "atoms" and then using Aho-Corasick for finding those "atoms" is simple and effective enough. There are improvements to be made, and your work with byte-classes is a good example of that. In fact, the only thing that prevents me from merging that work upstream is the increase in memory usage that we noticed at VirusTotal.

A more drastic idea is switching to HyperScan altogether. This is feasible only in the case of a full rewrite in Rust, but this requires a more in-depth analysis, as the last time I took a look at HyperScan I found a few things that would make it hard to fit well for the YARA use case.

The bottom line is that I don't have any revolutionary idea in mind. I think that even in the case of rewrite I would try to follow a similar approach again, perhaps introducing some changes here and there.

[regeciovad]

I understand. I updated the PR with some changes that positively influenced the memory usage at Avast, but I can recheck it if needed.
I have a similar experience with HyperScan. It is a promising tool, we did some testing and prototyping, but we also noticed the challenges and some cases where it shows that HyperScan was created for a little bit different use cases.
I would also point out this interesting project, Regex Performance. I added Yara to the testing set recently. There are different solutions and even Rust implementations.
Once you know the general direction and the complete rewrite to Rust is, for example, in play, it could be interesting to check these and even more solutions that are currently available.

[tlansec]

Changes to support printing of matched data irrespective of source.

Currently it is possible to see the data that the rule matched if strings are the cause of the match using the -s flag. If the matches were the result of module data matching or from uint() type condition logic, then it can be hard to triage which entries are matching as expected.

I don't have a proposed solution, but ideally it would be possible to specify a single flag when running yara via the CLI which prints out any matched data regardless of the source of the match.

[stvemillertime]

Accessing an instance of a string match via $s[i]

Right now, while we can access the length and offset of a string $s via #s and @s I would like to be able to iterate through and do different things with $s[i]. This might open up some new doors and possibilities for iteration and processing and checking multiple instances of a string, especially when dealing with regular expression matches.

[imp0rtp3]

Implement chained comparisons

In many instances, it can be useful to use chained comparisons, like:
@a < @b <@c
#w == #x == 1
While it can be done today by splitting the chain into two comparisons, chaining it is more readable and easier to write

[imp0rtp3]

Enable wildcard with # operator in the condition

I would like to be able to write a condition like (#b*) > 10.
Right now, I'm able to write for 10 of ($b*) : (#) but it's much less readable, and it's harder to check for an exact number this way ((#b*) == 10).
Another alternative available now is to write #b1 + #b2 + #b.. > 10 but it's very inconvenient for a lot of variables, and requires changing the condition every time a new $b string is added.

[plusvic]

I think the semantics for (#b*) > 10 may be confusing. My first interpretation was: "make sure that the number of occurrences for every pattern starting with $b is larger than 10". In other words, equivalent to for all of ($b*) : (# > 10). However, your intention was #b1 + #b2 + #b.. > 10, the sum of occurrences from all strings starting with $b is larger than 10. Both interpretations could be valid, but is not clear what to expect.

Also notice that in for 10 of ($b*) : (#) you are relying on an implementation detail. A for loop expects a boolean expression in its condition (i.e: for <x> of <string set> : (<boolean expr>)). Boolean expressions are usually 0 or 1, so the result from <boolean expr> is added to an accumulator that is compared to 10. The implementation details is that when converting # into a boolean, the value of # is used as is. So, if # is 5, the accumulator gets incremented by 5, not by 1, as it would if boolean values would be normalized to 0 or 1. In resume, for 10 of ($b*) : (#) is semantically equivalent to for 10 of ($b*) : ($), if you rely on the fact that # is not currently 0 or 1, like $, your rule may brake in the future.

[imp0rtp3]

@tlansec's proposal seems good.
Thanks for the heads up regarding the implementation details.

[tlansec]

Perhaps something in the math module might be applicable here?

e.g.

math.sum($b*) or something along those lines, whose definition can be more explicit and not break any pre-existing rule semantics.

^ realised this comment I posted a couple of months ago should be in this thread.

[tlansec]

Allow x of () to accept expressions in the brackets.

I think a lot of the things people currently find hard to express in conditions could be resolved x of () accepted more complex expressions within the brackets.

To illustrate this, in the following rule, if you want to match 2 of, 2 of ($a*), 2 of ($b*) and 2 of ($c*) but would accept any combination of 2 of these, its hard to write the condition (unless i'm missing something). Whereas in the condition (currently not working) expressed it is simple

rule myrule
{
strings:
   $a1 = "foo"
   $a2 = "bar"
   $a3 = "rae"
   
   $b1 = "123"
   $b2 = "456"
   $b3 = "789"
   
   $c1 = {00 01 02}
   $c2 = {03 04 05}
   $c3 = {06 07 08}
condition:
    2 of (2 of ($a*), 2 of ($b*), 2 of ($c*))
}