Goldrush is a small Erlang app that provides fast event stream processing

Event processing compiled to a query module

• per module protected event processing statistics
• query module logic can be combined for any/all filters
• query module logic can be reduced to efficiently match event processing

Complex event processing logic

• match input events with greater than (gt) logic
• match input events with less than (lt) logic
• match input events with equal to (eq) logic
• match input events with wildcard (wc) logic
• match input events with notfound (nf) logic
• match no input events (null blackhole) logic
• match all input events (null passthrough) logic

Handle output events

• Once a query has been composed the output action can be overriden with an erlang function. The function will be applied to each output event from the query.

Usage

To use goldrush in your application, you need to define it as a rebar dep or include it in erlang’s path.

Before composing modules, you’ll need to define a query. The query syntax matches any number of `{erlang, terms}’ and is composed as follows:

Simple Logic

• Simple logic is defined as any logic matching a single event filter

Select all events where ‘a’ exists and is greater than 0.

glc:gt(a, 0).

Select all events where ‘a’ exists and is equal to 0.

glc:eq(a, 0).

Select all events where ‘a’ exists and is less than 0.

glc:lt(a, 0).

Select all events where ‘a’ exists.

glc:wc(a).

Select all events where ‘a’ does not exist.

glc:nf(a).

Select no input events. User as a black hole query.

glc:null(false).

Select all input events. Used as a passthrough query.

glc:null(true).

Combined Logic

• Combined logic is defined as logic matching multiple event filters

Select all events where both ‘a’ AND ‘b’ exists and are greater than 0.

glc:all([glc:gt(a, 0), glc:gt(b, 0)]).

Select all events where ‘a’ OR ‘b’ exists and are greater than 0.

glc:any([glc:gt(a, 0), glc:gt(b, 0)]).

Select all events where ‘a’ AND ‘b’ exists where ‘a’ is greater than 1 and ‘b’ is less than 2.

glc:all([glc:gt(a, 1), glc:lt(b, 2)]).

Select all events where ‘a’ OR ‘b’ exists where ‘a’ is greater than 1 and ‘b’ is less than 2.

glc:any([glc:gt(a, 1), glc:lt(b, 2)]).

Reduced Logic

• Reduced logic is defined as logic which can be simplified to improve efficiency.

Select all events where ‘a’ is equal to 1, ‘b’ is equal to 2 and ‘c’ is equal to 3 and collapse any duplicate logic.

glc_lib:reduce(
    glc:all([
        glc:any([glc:eq(a, 1), glc:eq(b, 2)]),
        glc:any([glc:eq(a, 1), glc:eq(c, 3)])])).

The previous example will produce and is equivalent to:

glc:all([glc:eq(a, 1), glc:eq(b, 2), glc:eq(c, 3)]).

To compose a module you will take your Query defined above and compile it.

glc:compile(Module, Query).

• At this point you will be able to handle an event using a compiled query.

Begin by constructing an event list.

Event = gre:make([{'a', 2}], [list]).

Now pass it to your query module to be handled.

glc:handle(Module, Event).

Handling output events

• You can override the output action with an erlang function.

Write all input events as info reports to the error logger.

glc:with(glc:null(true), fun(E) ->
     error_logger:info_report(gre:pairs(E)) end).

Write all input events where `error_level’ exists and is less than 5 as info reports to the error logger.

glc:with(glc:lt(error_level, 5), fun(E) ->
     error_logger:info_report(gre:pairs(E)) end).

Return the number of input events for this query module.

glc:input(Module).

Return the number of output events for this query module.

glc:output(Module).

Return the number of filtered events for this query module.

glc:filter(Module).

Build

$ ./rebar compile

or

$ make

CHANGELOG

0.1.6

• Add notfound event matching

0.1.5

• Rewrite to make highly crash resilient
  • per module supervision
  • statistics data recovery
• Add wildcard event matching
• Add reset counters