-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: reshuffle files, loadtest is now the main module #26
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -14,3 +14,6 @@ Cargo.lock | |
*.pdb | ||
|
||
/.idea | ||
|
||
report.* | ||
baseline.json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,25 @@ | ||
[package] | ||
name = "scale-testing" | ||
name = "loadtest" | ||
version = "0.1.0" | ||
edition = "2021" | ||
|
||
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html | ||
[workspace] | ||
|
||
[dependencies] | ||
[dependencies] | ||
anyhow = "1" | ||
chrono = "0.4" | ||
clap = { version = "4", features = ["derive", "env"] } | ||
goose = "=0.17.3-dev" | ||
goose-eggs = "0.5.3-dev" | ||
humantime = "2" | ||
log = "0.4" | ||
openid = "0.14" | ||
reqwest = "0.12" | ||
tokio = { version = "1.38.0", features = ["sync"] } | ||
|
||
[patch.crates-io] | ||
#goose = { path = "../../goose" } | ||
#goose-eggs = { path = "../../goose-eggs" } | ||
|
||
goose = { git = "https://github.com/ctron/goose", branch = "feature/baseline_1" } | ||
goose-eggs = { git = "https://github.com/ctron/goose-eggs", branch = "feature/uptick_deps_1" } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,88 +1,38 @@ | ||
# scale-testing | ||
# trustify loadtest | ||
|
||
[![ci](https://github.com/trustification/scale-testing/actions/workflows/ci.yaml/badge.svg)](https://github.com/trustification/scale-testing/actions/workflows/ci.yaml) | ||
A set of simple [goose](https://book.goose.rs/) load tests against the web and rest endpoints. | ||
|
||
Utility for testing trustification at scale. | ||
## quickstart | ||
|
||
This tool is to help replicating existing SBOMs (SPDX or CycloneDX) file in order to augment an existing data set by multiplying the number SBOMs files. | ||
1. Ensure trustify is running. | ||
|
||
For instance let's say we have a total of 1000 SBMS (500 SPDX and 500) and we'd like to obtain a total of 10K SBOMs files for our scale test, so we can run the tool using a replication size of 10. | ||
2. Set environment variables for OIDC authentication: | ||
```bash | ||
export ISSUER_URL = "http://localhost:8090/realms/trustify" | ||
export CLIENT_ID = "testing-user" | ||
export CLIENT_SECRET = "****************" | ||
``` | ||
|
||
The tool replicates existing SBOMs, by copying each file content and change its file name and its key records. | ||
To change wait times between http invokes set the following env vars: | ||
|
||
## Usage | ||
After installing trustification/scale-testing repo, | ||
```bash | ||
export WAIT_TIME_FROM = 1 | ||
export WAIT_TIME_TO = 2 | ||
``` | ||
|
||
We can run the tool, by providing the size of the replication, the source directory and the destination directory : | ||
Alternately, for no wait times between http invokes set these env vars to 0. | ||
|
||
We need to provide a source of SBOMs, here located in /SBOMs which contains SBOM files in json format. | ||
3. To load trustify endpoints with 3 concurrent users. | ||
```bash | ||
cargo run --release --bin loadtest -- --host http://localhost:8080 -u 3 | ||
``` | ||
|
||
Also the target directory must not exists, this is to ensure we're not erasing an existing test set. | ||
To stop load test hit [ctl-C], which should generate aggregate statistics. | ||
|
||
```sh | ||
$ rm -rf /data-set | ||
$ cargo run -- 10 /SBOMs/ /data-set/ | ||
To load trustify endpoints against 10 concurrent users, generating an html report. | ||
|
||
The latter will replicate 10 times each SBOM file available in /SBOMs/. | ||
```bash | ||
cargo run --release -- --host http://localhost:8080 --report-file=report.html --no-reset-metrics -u 10 | ||
``` | ||
|
||
Each replicated SBOM file will be created under its corresponding batch directory under `/data-set`. | ||
|
||
> Example | ||
|
||
```sh | ||
$ cargo run -- 2 ./SBOMs ./data-set | ||
Compiling scale-testing v0.1.0 (/home/gildub/github.com/gildub/scale-testing) | ||
Finished dev [unoptimized + debuginfo] target(s) in 0.24s | ||
Running `target/debug/scale-testing 2 ./SBOMs ./data-set` | ||
Replication multiplier 2 | ||
Source directory ./SBOMs | ||
Destination directory ./data-set | ||
successfully wrote to metadata file | ||
successfully wrote to metadata file | ||
Amending version: "version": 1, | ||
successfully wrote to ./data-set/batch1/A7ED160707AB4BC.replicate1.cdx.json | ||
Amending version: "version": 1, | ||
successfully wrote to ./data-set/batch2/A7ED160707AB4BC.replicate2.cdx.json | ||
Amending name: "name": "quarkus-2.13", | ||
Amending documentNameSpacekey: "documentNamespace": "https://access.redhat.com/security/data/sbom/beta/spdx/quarkus-2.13-1a6ac4c55918a44fb3bada1b7e7d12f887d67be4", | ||
successfully wrote to ./data-set/batch1/quarkus-2.replicate1.13.json | ||
Amending name: "name": "quarkus-2.13", | ||
Amending documentNameSpacekey: "documentNamespace": "https://access.redhat.com/security/data/sbom/beta/spdx/quarkus-2.13-1a6ac4c55918a44fb3bada1b7e7d12f887d67be4", | ||
successfully wrote to ./data-set/batch2/quarkus-2.replicate2.13.json | ||
``` | ||
|
||
```sh | ||
$ tree data-set/ | ||
data-set/ | ||
├── batch1 | ||
│ ├── metadata | ||
│ │ └── metadata.json | ||
│ ├── A7ED160707AB4BC.replicate1.cdx.json | ||
│ └── quarkus-2.replicate1.13.json | ||
└── batch2 | ||
├── metadata | ||
│ └── metadata.json | ||
├── A7ED160707AB4BC.replicate2.cdx.json | ||
└── quarkus-2.replicate2.13.json | ||
|
||
5 directories, 6 files | ||
``` | ||
|
||
## Using bombastic_walker | ||
|
||
### Prepare initial SBOMs data set | ||
|
||
Provide initial set of SBOMs files to be replicated, i.e `/SBOMs/`. | ||
Use only SPDX files, for now, because the CycloneDX files tested were rejected with following error: `JSON: Unsupported CycloneDX version: 1.4` | ||
|
||
### Replicate SBOMs | ||
|
||
Run the replication tool to multiply your existing SBOM files set | ||
|
||
`cargo run -- 10 /SBOMs /data-set` | ||
|
||
### Use the replicated SBOMs | ||
|
||
The bombastic_walker could exploit each replicated SBOMs batch, for example in devmode : | ||
|
||
`RUST_LOG=info cargo run -p trust bombastic walker --sink http://localhost:8082 --source /data-set/batch1/ --devmode -3` | ||
4. More goose run-time options [here](https://book.goose.rs/getting-started/runtime-options.html) |
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
allow-unwrap-in-tests = true | ||
allow-expect-in-tests = true |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
# Generated by Cargo | ||
# will have compiled files and executables | ||
debug/ | ||
target/ | ||
|
||
# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries | ||
# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html | ||
Cargo.lock | ||
|
||
# These are backup files generated by rustfmt | ||
**/*.rs.bk | ||
|
||
# MSVC Windows builds of rustc generate these, which store debugging information | ||
*.pdb | ||
|
||
/.idea | ||
|
||
report.* | ||
baseline.json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
[package] | ||
name = "scale-testing" | ||
version = "0.1.0" | ||
edition = "2021" | ||
|
||
[workspace] |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thx for editing README (and removing oidc-cli instructions!)