-
Notifications
You must be signed in to change notification settings - Fork 45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Created new window pipe for sliding window aggregations #3
base: master
Are you sure you want to change the base?
Conversation
Pipes now reset their state after PIPE_EOF. Added new window pipe for time windowing streamed events. Fixed bug in walk__time_range reporting str instead of node.
… results. Added documentation for window pipe.
@rw-access I removed the tight coupling with host_key associated with pipes, as this seems very environment specific and breaks some common scenarios. I had to "reset" state after the pipe receives I don't have your test data, so I wasn't able to do a full test. My biggest concern is cases where I missed an object that should be immutable e.g. 37ee552. |
Other things to consider, the analytic will technically hold onto the buffer longer than necessary, e.g. imagine we get a burst of events in a timespan, then no events. The buffer will retain this spike until another event in the future triggers the buffer cleanup. I thought about adding a garbage cleanup routine, but I feel it's not a significant issue given it's unlikely to cause a relevant memory issue. |
Rolled back host_key removal.
# docs/query-guide/pipes.rst # eql/ast.py # eql/engine.py # eql/parser.py # setup.cfg # tests/test_data.json # tests/test_eql.py # tests/test_python_engine.py
@rw-access I've updated this code for EQL 0.7.0 and rolled back the host_key changes to make it easier for you to review. I made a change to the |
� eql/etc/eql.ebnf � eql/etc/test_queries.toml � eql/pipes.py � tests/test_python_engine.py
The concept of pipes doesn't work very well in context of streaming events. In order to allow pipes to function as expected in the streaming scenario, I've added a new "window" pipe that will buffer events into a sliding window, allowing all pipes to function as expected even when streaming.
A typical example of this feature would be the following:
In this example, the query is effective in both streaming and one off batches.