Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

processors: optimize via caching #69

Open
3 tasks
slint opened this issue Jul 10, 2018 · 0 comments
Open
3 tasks

processors: optimize via caching #69

slint opened this issue Jul 10, 2018 · 0 comments

Comments

@slint
Copy link
Member

slint commented Jul 10, 2018

Some of the processors are actually run multiple times in a row for very similar sets of values, since requests/events from the same (or similar) user are happening close to each other. Some examples:

  • GeoIP lookups
  • Robot/Machine detection
  • Any DB-dependent processors

To optimize these processors, simple functools.lru_cache decorators can be added to heavy-duty functions (this might require some function calls/signatures refactoring). On top of that, a Redis-based LRU cache can be used in order to share/persist the cache between different processes/workers/nodes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant