Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Async API, leverage rq+redis, use docker-compose for service setup #2

Merged
merged 31 commits into from
Sep 19, 2023
Merged

Async API, leverage rq+redis, use docker-compose for service setup #2

merged 31 commits into from
Sep 19, 2023

Conversation

awelzel
Copy link
Contributor

@awelzel awelzel commented Sep 13, 2023

No description provided.

@awelzel awelzel mentioned this pull request Sep 13, 2023
Closed
@awelzel
Snapshot
32d81ee 
* Make /zeek asynchronous using rq and redis
* Add tiny-benchmark.sh for running micro-benchmarks
* Allow specifying tests in config.yml
* Splits API, test execution and storage into various files
* Need to make the original pcap test case work again.
@awelzel awelzel force-pushed the topic/awelzel/add-rq branch from 5a9a84d to 32d81ee Compare September 13, 2023 17:44
@awelzel awelzel force-pushed the topic/awelzel/add-rq branch 4 times, most recently from df83fb0 to ae679aa Compare September 15, 2023 16:14
@awelzel awelzel marked this pull request as ready for review September 15, 2023 16:50
timwoj
timwoj reviewed Sep 15, 2023
View reviewed changes
ci-based/settings.py Outdated Show resolved Hide resolved
timwoj
timwoj approved these changes Sep 15, 2023
View reviewed changes
Copy link
Member

@timwoj timwoj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me overall. I had one minor question about configuration. If you want to rework that docker-compose file into the code that's fine with me.

@awelzel awelzel mentioned this pull request Sep 18, 2023
Open
@awelzel awelzel force-pushed the topic/awelzel/add-rq branch 2 times, most recently from ade0a54 to b35ba15 Compare September 18, 2023 16:02
@awelzel
Split benchmarker.py into zeek_benchmarker.app
c6180ed 
Use the create_app() pattern and current_app context. This makes
it easier to test the application via Flask's test_client(). See
the new tests/test_app.py file.
@awelzel
Tweak microbenchmark n values
4c6f2b0 
Make them run a bit longer.
@awelzel
client: Support /broker, add optional commit parameter
3d3aabf
@awelzel
Make /broker asynchronous, too.
b75527d 
This is untested and not re-worked due to broker builds not
containing the needed broker-cluster-benchmark file.

app.py is now quite shallow
@awelzel
Unpack build.tgz once
5b17193 
Instead of unpacking the build.tgz archive, unpack it once into
a volume (zeek_install_data) and mount the contents of that volume
at /zeek/install.

This reduces runtime by not redundantly extracting build.tgz and
also simplifies the setup a bit as we do not need to set ZEEKPATH
anymore (/zeek/install is where Zeek is configured to be installed).
@awelzel
microbenchmarks: Add complex-key-value-2, timing tweaks, WRITER_NONE
8d1e40c
@awelzel
Fixup API tests
052a763
@awelzel
tasks: Use docker library for running unpack container
b9b4104 
...instead of subprocess.run() of /usr/bin/docker
@awelzel
pre-commit-config: Add isort
f69c9d2
@awelzel
Add GitHub workflow for pre-commit
3c46b9c
@awelzel
client.py: Move into tools, fix formatting
fec6e36
@awelzel
Add GitHub action running tests
3757d88
@awelzel
storage: UNIXEPOCH is too new
a3e1166 
Not available on Python 3.10 or Python 3.11 in GitHub actions :-(
@awelzel
docker/docker-compose: Run api, rq and redis in docker-compose
d9127cb
@awelzel
config: Include rq dictConfig
05695dd
@awelzel
Run benchmarker containers without docker-compose
23c56ff 
We continue to use docker-compose, but it is now running
the infrastructure. Containers are launched from the rq
container via the docker Python library.

Broker benchmarking is now definitely not working.
@awelzel
pre-commit: Add yamllint
8563d62
@awelzel
pre-commit: Add hadolint for ci-builder
2adc554
@awelzel
api: Remove reliance on git subprocess
8e8ae83 
No need to install the git package for checking the validity of
a branch name.
@awelzel
pyupgrade: Use --py310-plus, nicer type annotations
a5e0058
@awelzel awelzel force-pushed the topic/awelzel/add-rq branch from b35ba15 to a5e0058 Compare September 18, 2023 16:14
@awelzel awelzel changed the title Topic/awelzel/add rq Async API, leverage rq+redis, use docker-compose for service setup Sep 18, 2023
@awelzel awelzel force-pushed the topic/awelzel/add-rq branch from 1907668 to 07f7a5d Compare September 18, 2023 17:36
@awelzel
Copy link
Contributor Author

awelzel commented Sep 18, 2023

This looks good to me overall. I had one minor question about configuration. If you want to rework that docker-compose file into the code that's fine with me.

@timwoj - feel free to take another looks. I mentioned the locations of the new database in DM if you want to look at some of the results.

@timwoj timwoj self-requested a review September 18, 2023 20:21
@awelzel awelzel merged commit 5635863 into master Sep 19, 2023
3 checks passed
@awelzel awelzel deleted the topic/awelzel/add-rq branch September 19, 2023 10:03
@ckreibich ckreibich mentioned this pull request Oct 10, 2023
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@timwoj timwoj timwoj approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@awelzel @timwoj