DEV-376 new stressng format #30
Conversation
WalkthroughThe changes involve modifications to the Changes
Assessment against linked issues
Possibly related PRs
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Outside diff range and nitpick comments (5)
src/sc_crawler/lookup.py (4)
Line range hint
270-278: Approve deprecation notice and suggest CHANGELOG updateThe addition of the deprecation notice for the "stress-ng CPU all" benchmark in favor of the new "stress_ng:div16" benchmark is appropriate and aligns with the PR objectives.
Please ensure that this deprecation is mentioned in the CHANGELOG.md file to keep users informed about this upcoming change.
279-290: Approve new div16 benchmark and suggest minor improvementThe addition of the "stress_ng:div16" benchmark is well-structured and provides a more specific measurement compared to the deprecated "CPU all" benchmark. This aligns with the PR objectives to implement a new format for stress-ng output handling.
Consider adding a brief explanation of what the "div16" method does in the description to provide more context for users unfamiliar with stress-ng internals.
291-301: Approve single-core benchmark and suggest clarificationThe addition of the "stress_ng:best1" benchmark for single-core stress testing is a valuable addition, providing more granular performance data.
Consider clarifying in the description why this benchmark is named "best1". Is it because it measures the best performance of a single core? This would help users better understand the benchmark's purpose.
Missing Deprecation Notice for
stress_ng:cpu_allThe deprecation notice for
stress_ng:cpu_allis missing insrc/sc_crawler/lookup.py. Additionally, there is still a reference tostress_ng:cpu_allinsrc/sc_crawler/inspector.py, which needs to be addressed to fully deprecate this benchmark.
- src/sc_crawler/lookup.py: Add a deprecation message for
stress_ng:cpu_allin the relevant section.- src/sc_crawler/inspector.py: Remove or update the comment referencing the deprecated
stress_ng:cpu_allbenchmark.🔗 Analysis chain
Line range hint
270-312: Summary: New stress-ng benchmarks align with PR objectivesThe changes to the stress-ng benchmarks in this file successfully implement a new format for handling the output of the stress-ng tool, as outlined in the PR objectives. The new benchmarks (div16, best1, and bestn) provide more specific and granular measurements compared to the deprecated "CPU all" benchmark, enhancing the overall benchmarking capabilities.
These changes align well with the goal of eliminating the need for a fallback mechanism and waiting until the new task has been successfully executed on all servers, as mentioned in the linked issue DEV-376.
To ensure these changes are properly reflected in the codebase, please run the following script:
This script will help verify the implementation of the new benchmarks and identify any other files that might need updates due to the deprecation of the "stress_ng:cpu_all" benchmark.
To resolve the issue with the
rgcommand not recognizing thepythonfile type, please run the following updated script:This updated script removes the
--type pythonflag and instead targets Python files by their.pyextension. This should prevent the "unrecognized file type" error and allow for accurate searching within the codebase.🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the implementation of new stress-ng benchmarks and the deprecation notice. # Test 1: Check for the deprecation notice in the stress_ng:cpu_all benchmark echo "Checking for deprecation notice:" rg --type python 'stress_ng:cpu_all.*to be deprecated in favor of stress_ng:div16' src/sc_crawler/lookup.py # Test 2: Verify the presence of new stress-ng benchmarks echo "Checking for new stress-ng benchmarks:" rg --type python 'benchmark_id="stress_ng:(div16|best1|bestn)"' src/sc_crawler/lookup.py # Test 3: Check for updates in other files that might be affected by these changes echo "Checking for potential updates needed in other files:" rg --type python 'stress_ng:cpu_all' -g '!src/sc_crawler/lookup.py'Length of output: 640
Script:
# #!/bin/bash # Description: Verify the implementation of new stress-ng benchmarks and the deprecation notice. # Test 1: Check for the deprecation notice in the stress_ng:cpu_all benchmark echo "Checking for deprecation notice:" rg 'stress_ng:cpu_all.*to be deprecated in favor of stress_ng:div16' src/sc_crawler/lookup.py # Test 2: Verify the presence of new stress-ng benchmarks echo "Checking for new stress-ng benchmarks:" rg 'benchmark_id="stress_ng:(div16|best1|bestn)"' src/sc_crawler/lookup.py # Test 3: Check for updates in other files that might be affected by these changes echo "Checking for potential updates needed in other files:" rg 'stress_ng:cpu_all' --glob '!src/sc_crawler/lookup.py' --type pyLength of output: 757
src/sc_crawler/inspector.py (1)
290-290: Reminder to address the deprecation TODOThere is a
# TODO deprecatecomment at line 290 indicating the intention to deprecate this section of the code. Please ensure that a deprecation plan is in place and that any necessary steps are documented or implemented.Would you like assistance in drafting a deprecation plan or updating the code to reflect the deprecation?
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (2)
- src/sc_crawler/inspector.py (2 hunks)
- src/sc_crawler/lookup.py (2 hunks)
🧰 Additional context used
🪛 Ruff
src/sc_crawler/inspector.py
313-313: Local variable
eis assigned to but never usedRemove assignment to unused variable
e(F841)
🪛 GitHub Check: build
src/sc_crawler/inspector.py
[failure] 313-313: Ruff (F841)
src/sc_crawler/inspector.py:313:25: F841 Local variableeis assigned to but never used
| Benchmark( | ||
| benchmark_id="stress_ng:bestn", | ||
| name="stress-ng div16 multi-core", | ||
| description="Stress the CPU with the div16 method of stress-ng using a varying number of vCPU cores, and count the measured maximum total bogo operations per second based on wall clock run time.", | ||
| framework="stress_ng", | ||
| measurement="bestn", | ||
| config_fields={ | ||
| "framework_version": "Version number of stress-ng.", | ||
| }, | ||
| unit="bogo ops/s (real time)", | ||
| ), |
There was a problem hiding this comment.
Approve multi-core benchmark and address missing parameter
The addition of the "stress_ng:bestn" benchmark for multi-core stress testing is valuable, complementing the single-core benchmark and providing a comprehensive view of CPU performance.
The description mentions "using a varying number of vCPU cores", but the config_fields don't include a "cores" parameter. This inconsistency should be addressed. Consider adding a "cores" parameter to the config_fields or clarify in the description how the number of cores is determined.
src/sc_crawler/inspector.py
Outdated
| except Exception as e: | ||
| # backfill with newer method - can be dropped once we deprecate stress_ng:cpu_all | ||
| try: | ||
| records = [] | ||
| with open( | ||
| _server_framework_stdout_path(server, "stressngfull"), newline="" | ||
| ) as f: | ||
| rows = csv.reader(f, quoting=csv.QUOTE_NONNUMERIC) | ||
| for row in rows: | ||
| records.append(row) | ||
| for i in [0, len(records) - 1]: | ||
| stressng_version = _server_framework_meta(server, "stressngfull")[ | ||
| "version" | ||
| ] | ||
| benchmarks.append( | ||
| { | ||
| **_benchmark_metafields( | ||
| server, | ||
| framework="stressngfull", | ||
| benchmark_id=":".join([framework, "cpu_all"]), | ||
| ), | ||
| "config": { | ||
| "cores": records[i][0], | ||
| "framework_version": stressng_version, | ||
| }, | ||
| "score": records[i][1], | ||
| } | ||
| ) | ||
| except Exception as e: | ||
| _log_cannot_load_benchmarks(server, framework, e, True) |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Refactor nested try-except blocks and narrow exception handling
The nested try-except blocks starting at line 313 can reduce code readability and make debugging more difficult. Additionally, catching the broad Exception class may mask unexpected errors.
Consider refactoring the exception handling to improve readability and robustness:
- Narrow down the exceptions to catch specific errors (e.g.,
FileNotFoundError,IOError). - Flatten the nested
try-exceptblocks to simplify the flow.
Here's a refactored version of the code:
# TODO deprecate
cores_per_path = {"stressng": server.vcpus, "stressngsinglecore": 1}
for cores_path in cores_per_path.keys():
try:
stressng_version = _server_framework_meta(server, cores_path)["version"]
line = _extract_line_from_file(
_server_framework_stderr_path(server, cores_path),
"bogo-ops-per-second-real-time",
)
benchmarks.append(
{
**_benchmark_metafields(
server,
framework=cores_path,
benchmark_id=":".join([framework, "cpu_all"]),
),
"config": {
"cores": cores_per_path[cores_path],
"framework_version": stressng_version,
},
"score": float(line.split(": ")[1]),
}
)
except (FileNotFoundError, KeyError, ValueError) as e:
_log_cannot_load_benchmarks(server, framework, e, True)
# Attempt to backfill with newer method
try:
records = []
with open(
_server_framework_stdout_path(server, "stressngfull"), newline=""
) as f:
rows = csv.reader(f, quoting=csv.QUOTE_NONNUMERIC)
for row in rows:
records.append(row)
for i in [0, len(records) - 1]:
stressng_version = _server_framework_meta(server, "stressngfull")[
"version"
]
benchmarks.append(
{
**_benchmark_metafields(
server,
framework="stressngfull",
benchmark_id=":".join([framework, "cpu_all"]),
),
"config": {
"cores": records[i][0],
"framework_version": stressng_version,
},
"score": records[i][1],
}
)
except Exception as e:
_log_cannot_load_benchmarks(server, framework, e, True)This refactoring:
- Moves the
try-exceptinside the loop for clearer logic. - Catches specific exceptions to prevent masking other errors.
- Utilizes logging to handle exceptions appropriately.
🧰 Tools
🪛 Ruff
313-313: Local variable
eis assigned to but never usedRemove assignment to unused variable
e(F841)
🪛 GitHub Check: build
[failure] 313-313: Ruff (F841)
src/sc_crawler/inspector.py:313:25: F841 Local variableeis assigned to but never used
src/sc_crawler/inspector.py
Outdated
| @@ -309,6 +310,78 @@ | |||
| "score": float(line.split(": ")[1]), | |||
| } | |||
| ) | |||
| except Exception as e: | |||
There was a problem hiding this comment.
Remove unused variable e in exception handler
The variable e is assigned but never used in the except block at line 313. This can be cleaned up by removing the variable if it's not needed.
Apply this diff to remove the unused variable:
- except Exception as e:
+ except Exception:📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| except Exception as e: | |
| except Exception: |
🧰 Tools
🪛 Ruff
313-313: Local variable
eis assigned to but never usedRemove assignment to unused variable
e(F841)
🪛 GitHub Check: build
[failure] 313-313: Ruff (F841)
src/sc_crawler/inspector.py:313:25: F841 Local variableeis assigned to but never used
Overview
Required checks before merge
mainCHANGELOG.mdpyproject.tomladd_vendor.mdis up-to-datesc-crawler pullwas tested on all vendors and recordsSummary by CodeRabbit
New Features
Deprecations
Bug Fixes