538 feature update and use ultralytics yolov11 for object detection #543
Conversation
WalkthroughThe changes in this pull request involve updates to the perception system's launch configuration and the Changes
Assessment against linked issues
Possibly related PRs
Suggested reviewers
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Outside diff range and nitpick comments (3)
code/perception/src/vision_node_helper.py (2)
1-16: Translate German comments to English and consider using a more type-safe approach for colors.
- Replace "Carla-Farben" with "Carla Colors" for consistency with English codebase.
- Consider using a more type-safe approach for RGB values.
-# Carla-Farben +# Carla Colors +from typing import List, Tuple + +carla_colors: List[Tuple[int, int, int]] = [ (0, 0, 0), # 0: None (70, 70, 70), # 1: Buildings # ... rest of the colors ... ]
18-32: Consider using an Enum for class names.Using an Enum would provide better type safety, IDE support, and prevent invalid class references.
+from enum import IntEnum + +class CarlaClass(IntEnum): + NONE = 0 + BUILDINGS = 1 + FENCES = 2 + OTHER = 3 + PEDESTRIANS = 4 + POLES = 5 + ROADLINES = 6 + ROADS = 7 + SIDEWALKS = 8 + VEGETATION = 9 + VEHICLES = 10 + WALLS = 11 + TRAFFIC_SIGNS = 12 + -carla_class_names = [ - "None", # 0 - "Buildings", # 1 - # ... rest of the names ... -] +carla_class_names = [cls.name.title().replace('_', '') for cls in CarlaClass]code/perception/src/vision_node.py (1)
369-371: Improve code readability by simplifying the conditionYou can invert the condition to make the code more readable and avoid unnecessary indentation.
Apply this diff to simplify the condition:
-if self.dist_arrays is None: - continue +if self.dist_arrays is not None: + # Proceed with processing
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (4)
code/perception/launch/perception.launch(1 hunks)code/perception/src/vision_node.py(7 hunks)code/perception/src/vision_node_helper.py(1 hunks)code/requirements.txt(1 hunks)
✅ Files skipped from review due to trivial changes (1)
- code/requirements.txt
🔇 Additional comments (8)
code/perception/launch/perception.launch (2)
61-65: LGTM! Well-documented model options
The new YOLOv11 segmentation models are properly documented with clear size variants, which helps developers choose the appropriate model for their use case. This aligns with the PR objective of implementing segmentation masks.
68-68: Verify performance impact of using yolo11n-seg
While switching to YOLOv11 aligns with the PR objectives, using the smallest variant (-n) might impact detection accuracy. The PR objectives state that "the new model does not perform worse than the previous version."
Let's verify the model choice:
code/perception/src/vision_node_helper.py (1)
34-116: 🛠️ Refactor suggestion
Improve type safety and verify mapping strategy.
- The comment should be in English for consistency.
- A large number of COCO classes are mapped to "Other". Please verify if this is intentional or if more specific mappings could be beneficial.
-# COCO-Klassen → Carla-Klassen Mapping
+# COCO Classes to Carla Classes Mapping
+from typing import Final, List
+
+COCO_CLASS_COUNT: Final = 80
+coco_to_carla: List[int] = [
4, # 0: Person -> Pedestrians
10, # 1: Bicycle -> Vehicles
# ... rest of mapping ...
]
+
+def validate_mapping() -> None:
+ """Validate the COCO to Carla class mapping at module load time."""
+ if len(coco_to_carla) != COCO_CLASS_COUNT:
+ raise ValueError(f"Expected {COCO_CLASS_COUNT} COCO classes, got {len(coco_to_carla)}")
+ if not all(0 <= x < len(carla_class_names) for x in coco_to_carla):
+ raise ValueError("Invalid Carla class index in mapping")
+
+validate_mapping()code/perception/src/vision_node.py (5)
248-248: Ensure consistent image encoding with OpenCV
The image is converted to a message with encoding "bgr8", which aligns with OpenCV's default BGR color format. This change ensures correct color representation when publishing the image.
18-18: Verify the import of helper functions from vision_node_helper
Ensure that the module vision_node_helper.py exists at the specified path and that the functions get_carla_class_name and get_carla_color are correctly implemented and accessible.
Run the following script to verify the existence of the module and functions:
✅ Verification successful
Import verification successful
The verification confirms that:
- The module
vision_node_helper.pyexists at the correct path:code/perception/src/vision_node_helper.py - Both required functions
get_carla_class_nameandget_carla_colorare properly defined in the module
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Verify that 'vision_node_helper.py' contains the required functions.
# Find 'vision_node_helper.py' in the codebase
fd 'vision_node_helper.py'
# Search for the function definitions in 'vision_node_helper.py'
fd 'vision_node_helper.py' | xargs grep -E 'def get_carla_class_name|def get_carla_color'
Length of output: 229
28-28: Confirm the availability of scale_masks in the ultralytics package
Ensure that the function scale_masks exists in ultralytics.utils.ops and that you have the correct version of the ultralytics library installed.
Run the following script to check for the scale_masks function:
✅ Verification successful
scale_masks is available and being used correctly
The verification shows that:
- The
scale_masksfunction is imported fromultralytics.utils.opsinvision_node.py - The function is actively used in the codebase with correct parameters
- The project uses
ultralytics==8.3.32, which is a stable version that includes this functionality
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Verify that 'scale_masks' is defined in 'ultralytics.utils.ops'
# Search for the definition of 'scale_masks' in the 'ultralytics' package
rg --files-with-matches 'def scale_masks' | grep 'ultralytics/utils/ops.py'
Length of output: 75
Script:
#!/bin/bash
# Let's check for any imports of scale_masks in the codebase
rg "from ultralytics.*scale_masks" -A 2
# Also check for any direct usage of scale_masks
rg "scale_masks\(" -A 2
# Let's also check requirements.txt or similar files for ultralytics version
fd "requirements" --type f --exec cat {}
Length of output: 7686
428-436: 🛠️ Refactor suggestion
Use consistent colors for bounding boxes and segmentation masks
Currently, bounding boxes are drawn with the color "blue", while segmentation masks use class-specific colors from c_colors. For better visual consistency and to differentiate between classes, consider using c_colors for bounding boxes as well.
Apply this diff to use c_colors for bounding boxes:
bounding_box_images = draw_bounding_boxes(
image_np_with_detections,
c_boxes,
c_labels,
- colors="blue",
+ colors=c_colors,
width=3,
font_size=12,
)Likely invalid or redundant comment.
350-350: Confirm the use of the track method for YOLO models
Verify that the track method is appropriate for the YOLO models being used. The track method may not be supported for all model types or versions in the ultralytics library.
Run the following script to check if the track method exists:
✅ Verification successful
The track method is appropriate for the YOLO models being used
The codebase shows that:
- The model is properly initialized from the
ultralyticspackage using supported YOLO variants (yolov8n through yolov8x) - The model dictionary correctly configures YOLO models with their respective weights and framework
- The
trackmethod is a standard feature in YOLOv8 models from the ultralytics package, used for object detection with tracking capabilities
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Verify that the YOLO model class has a 'track' method
# Search for the 'track' method in the YOLO model class definition
rg -A 5 'class YOLO' | grep 'def track'
Length of output: 41
Script:
#!/bin/bash
# Let's try a broader search to understand the model initialization and usage
# First, let's find where the model is initialized
rg -l "from ultralytics"
# Then search for YOLO-related imports and model initialization
rg "from ultralytics|import YOLO|model = YOLO" -A 3
# Also search for track method usage
rg "\.track\(" -A 2
Length of output: 1212
Script:
#!/bin/bash
# Let's check how the model is initialized and configured in vision_node.py
rg "self\.model = " -B 5 -A 2 code/perception/src/vision_node.py
# Also check for any model configuration or type specification
rg "model_type|weights|cfg" -B 2 -A 2 code/perception/src/vision_node.py
Length of output: 2067
Script:
#!/bin/bash
# Let's check the model dictionary definitions to see YOLO configurations
rg "YOLO" -B 2 -A 5 code/perception/src/vision_node.py
# Also check for any track-specific configurations or settings
rg "track.*=|tracking" code/perception/src/vision_node.py
Length of output: 1916
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Outside diff range and nitpick comments (7)
code/perception/src/vision_node_helper.py (5)
1-1: Remove unused importTupleThe
Tupletype is imported but never used in the code.-from typing import List, Tuple, Union +from typing import List, Union🧰 Tools
🪛 Ruff (0.8.0)
1-1:
typing.Tupleimported but unusedRemove unused import:
typing.Tuple(F401)
4-4: Translate German comments to English for consistencyReplace German comments with English translations:
-# Carla-Farben +# Carla Colors -# COCO-Klassen → Carla-Klassen Mapping +# COCO Classes to Carla Classes MappingAlso applies to: 37-37
5-119: Add validation for data structure consistencyThe mapping arrays should be validated to ensure consistency between COCO and Carla classes. Consider adding runtime validation:
def validate_mappings(): """Validate the consistency of COCO to Carla mappings.""" # Validate array lengths assert len(coco_to_carla) == COCO_CLASS_COUNT, "Incorrect number of COCO classes" # Validate Carla class indices max_carla_class = len(carla_colors) - 1 invalid_mappings = [ (i, class_idx) for i, class_idx in enumerate(coco_to_carla) if not 0 <= class_idx <= max_carla_class ] assert not invalid_mappings, f"Invalid Carla class indices in mapping: {invalid_mappings}" # Validate color and name arrays match assert len(carla_colors) == len(carla_class_names), ( "Mismatch between Carla colors and class names arrays" ) # Call validation on module import validate_mappings()
124-158: Refactor helper functions to reduce code duplicationBoth functions share similar logic for input validation and mapping. Consider extracting the common functionality:
def _get_carla_class(coco_class: Union[int, float]) -> int: """Convert COCO class index to Carla class index with validation. Args: coco_class: COCO class index (0-79) Returns: Corresponding Carla class index Raises: ValueError: If coco_class is out of valid range """ coco_idx = int(coco_class) if not 0 <= coco_idx < COCO_CLASS_COUNT: raise ValueError(f"Invalid COCO class index: {coco_idx}") return coco_to_carla[coco_idx] def get_carla_color(coco_class: Union[int, float]) -> List[int]: """Get the Carla color for a given COCO class.""" return carla_colors[_get_carla_class(coco_class)] def get_carla_class_name(coco_class: Union[int, float]) -> str: """Get the Carla class name for a given COCO class.""" return carla_class_names[_get_carla_class(coco_class)]
125-134: Improve docstring formattingThe docstring formatting could be improved to better follow Google style:
def get_carla_color(coco_class: Union[int, float]) -> List[int]: """Get the Carla color for a given COCO class. + Args: - coco_class: COCO class index (0-79) + coco_class: COCO class index (0-79) Returns: - RGB color values for the corresponding Carla class + RGB color values for the corresponding Carla class Raises: - ValueError: If coco_class is out of valid range + ValueError: If coco_class is out of valid range """Also applies to: 143-152
code/perception/src/vision_node.py (2)
372-439: Consider refactoring distance calculation logicThe distance calculation logic is complex and contains multiple responsibilities:
- Distance array handling
- Coordinate system transformations
- Distance calculations
- Output formatting
Consider extracting this into a separate method for better maintainability.
- if self.dist_arrays is None: - continue - # ... (current distance calculation code) ... + distance_info = self._calculate_object_distance(pixels, cls) + if distance_info: + distance_output.extend(distance_info) + obj_dist_min_x, obj_dist_min_abs_y, abs_distance = distance_info[1:] + else: + obj_dist_min_x = obj_dist_min_abs_y = (np.inf, np.inf, np.inf) + abs_distance = np.inf
406-414: Add documentation for coordinate system assumptionsThe comment mentions coordinate system differences for different camera angles, but the implementation assumes center view. Consider:
- Adding clear documentation about the coordinate system conventions
- Making the coordinate system handling more explicit in the code
+ def _get_coordinate_system_transform(self, camera_angle: str) -> np.ndarray: + """Returns the transformation matrix for the given camera angle. + + The coordinate system conventions are: + - Center: +x forward, +y left, +z up + - Back: -x forward, -y right, +z up + - Left: +y forward, -x left, +z up + - Right: -y forward, +x right, +z up + """
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (2)
code/perception/src/vision_node.py(7 hunks)code/perception/src/vision_node_helper.py(1 hunks)
🧰 Additional context used
📓 Learnings (1)
code/perception/src/vision_node.py (2)
Learnt from: SirMDA
PR: una-auxme/paf#543
File: code/perception/src/vision_node.py:82-85
Timestamp: 2024-11-29T15:37:29.790Z
Learning: YOLOv11 segmentation models (`yolo11n-seg.pt`, `yolo11s-seg.pt`, `yolo11m-seg.pt`, `yolo11l-seg.pt`) are available and can be used in the codebase.
Learnt from: Toni2go
PR: una-auxme/paf#422
File: doc/research/paf24/perception/VisionNode_CodeSummary.md:114-114
Timestamp: 2024-11-12T07:33:22.092Z
Learning: In `vision_node.md`, the `yolov8x-seg` model performs segmentation (not detection) and can also calculate distances.
🪛 Ruff (0.8.0)
code/perception/src/vision_node_helper.py
1-1: typing.Tuple imported but unused
Remove unused import: typing.Tuple
(F401)
🔇 Additional comments (6)
code/perception/src/vision_node.py (6)
18-18: LGTM: Helper function imports and scale_masks import
The addition of helper functions for Carla class names and colors, along with the scale_masks utility, aligns well with the PR objectives for enhanced visualization.
Also applies to: 28-28
82-85: LGTM: YOLOv11 segmentation models integration
The addition of YOLOv11 segmentation models aligns with the PR objectives. Based on the learnings from SirMDA, these models are confirmed to be available.
248-248: Fix confirmed: RGB to BGR color encoding
The change from "rgb8" to "bgr8" encoding addresses the color channel bug mentioned in the PR objectives.
359-362: LGTM: Robust mask handling
Good implementation of mask availability checking, addressing the previous review comment about handling cases where masks are unavailable.
453-475: LGTM: Enhanced visualization with segmentation masks
The implementation correctly handles:
- Bounding box drawing with tracking IDs
- Segmentation mask scaling and overlay
- Color mapping based on class
- Proper color space conversion
This aligns well with the PR objectives for enhanced visual output.
350-350: Consider performance implications of tracking
The switch from predict to track method adds object tracking capabilities but might have performance implications. Ensure that the tracking overhead is acceptable for your real-time requirements.
✅ Verification successful
Performance implications of tracking are acceptable for real-time operation
Based on the code analysis, the tracking implementation is well-optimized:
-
The model.track() call uses performance-optimized settings:
half=True- enables half-precision (FP16) inference for better performanceverbose=False- disables verbose outputimgsz=640- uses a reasonable input resolution
-
The codebase shows careful consideration of performance:
- CUDA memory management with
torch.cuda.empty_cache() - Memory fraction limiting with
torch.cuda.memory.set_per_process_memory_fraction(0.1) - Efficient processing of lidar data only when available
- Optimized visualization with minimal overhead
- CUDA memory management with
-
The documentation indicates this is used in production with real-time requirements, and tracking is a core feature of the perception system.
The tracking overhead appears to be well-managed and suitable for real-time autonomous vehicle perception requirements.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Check if there are any performance-related issues or discussions
rg -g "!{*.pyc,*.min.js}" -A 3 "performance|latency|fps" "*.md" "*.rst" "*.txt"
Length of output: 218
Script:
#!/bin/bash
# Search for performance-related discussions in all text files
rg -t md -t rst -t txt "performance|latency|fps"
# Look for track vs predict method usage and any performance configurations
ast-grep --pattern 'track($$$)'
ast-grep --pattern 'predict($$$)'
# Search for any performance-related configurations or comments
rg -g "!{*.pyc,*.min.js}" "(#.*performance|#.*latency|#.*fps)"
Length of output: 7008
Script:
#!/bin/bash
# Look for the model configuration and any performance-related settings in vision_node.py
rg -A 5 "model\." code/perception/src/vision_node.py
# Search for any documentation about tracking vs detection modes
fd -t f "vision_node" -x cat {}
Length of output: 32186
…11-for-object-detection
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (5)
code/perception/src/vision_node_helper.py (5)
4-4: Translate German comments to English for consistencyThe codebase should maintain consistent language usage. Consider translating:
- "Carla-Farben" → "Carla Colors"
- "COCO-Klassen → Carla-Klassen Mapping" → "COCO Classes to Carla Classes Mapping"
Also applies to: 37-37
5-119: Add type hints and validation for data structuresConsider adding type hints and runtime validation:
+from typing import Final, List -carla_colors = [ +carla_colors: Final[List[List[int]]] = [ [0, 0, 0], # 0: None # ... rest of the colors ] -carla_class_names = [ +carla_class_names: Final[List[str]] = [ "None", # 0 # ... rest of the names ] -coco_to_carla = [ +coco_to_carla: Final[List[int]] = [ 4, # 0: Person -> Pedestrians # ... rest of the mapping ] +# Validate data structure consistency +assert len(carla_colors) == len(carla_class_names), "Mismatched Carla colors and class names" +assert all(0 <= idx < len(carla_colors) for idx in coco_to_carla), "Invalid Carla class index in mapping"
38-119: Consider refining COCO to Carla class mappingMany COCO classes (e.g., traffic-related objects like parking meters) are mapped to the generic "Other" category. This might reduce the system's ability to handle these objects appropriately in the Carla simulation environment.
Consider:
- Creating additional Carla classes for important traffic-related objects
- Grouping similar COCO classes into more specific Carla categories
- Documenting the rationale for mapping choices
124-158: Refactor duplicate validation logic and consider cachingBoth functions share similar validation logic. Consider these improvements:
+from functools import lru_cache + +def _validate_coco_index(coco_idx: int) -> None: + """Validate COCO class index. + + Args: + coco_idx: COCO class index to validate + + Raises: + ValueError: If index is out of valid range + """ + if not 0 <= coco_idx < COCO_CLASS_COUNT: + raise ValueError(f"Invalid COCO class index: {coco_idx}") + +@lru_cache(maxsize=COCO_CLASS_COUNT) def get_carla_color(coco_class: Union[int, float]) -> List[int]: """Get the Carla color for a given COCO class. Args: coco_class: COCO class index (0-79) Returns: RGB color values for the corresponding Carla class Raises: ValueError: If coco_class is out of valid range """ coco_idx = int(coco_class) - if not 0 <= coco_idx < COCO_CLASS_COUNT: - raise ValueError(f"Invalid COCO class index: {coco_idx}") + _validate_coco_index(coco_idx) carla_class = coco_to_carla[coco_idx] return carla_colors[carla_class] +@lru_cache(maxsize=COCO_CLASS_COUNT) def get_carla_class_name(coco_class: Union[int, float]) -> str: """Get the Carla class name for a given COCO class. Args: coco_class: COCO class index (0-79) Returns: Name of the corresponding Carla class Raises: ValueError: If coco_class is out of valid range """ coco_idx = int(coco_class) - if not 0 <= coco_idx < COCO_CLASS_COUNT: - raise ValueError(f"Invalid COCO class index: {coco_idx}") + _validate_coco_index(coco_idx) carla_class = coco_to_carla[coco_idx] return carla_class_names[carla_class]Benefits:
- Reduces code duplication
- Improves performance through caching
- Centralizes validation logic
124-158: Add test cases for edge casesThe functions handle input validation, but test cases should be added to verify behavior with:
- Integer and float inputs
- Boundary values (0, 79)
- Invalid inputs (negative, out of range)
- Non-numeric inputs
Would you like me to generate comprehensive unit tests for these functions?
There was a problem hiding this comment.
I really don't know why, but the Vison Node currently crashes for me with the following error:
Traceback (most recent call last):
File "/catkin_ws/src/code/perception/src/vision_node.py", line 18, in <module>
from perception.src.vision_node_helper import get_carla_class_name, get_carla_color
ModuleNotFoundError: No module named 'perception.src'
But manually importing this module in python inside the container works and I don't see anything wrong with your code...
…11-for-object-detection
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Outside diff range and nitpick comments (1)
code/perception/src/vision_node.py (1)
389-401: Document coordinate system assumptions.The inline comments about coordinate system differences for different camera angles should be moved to method documentation.
+ """ + Calculate distances for objects in the camera view. + + Note on coordinate systems: + - Center view: Uses direct x, y coordinates + - Back view: Inverted x-axis (-x) + - Left/Right views: Adjusted coordinate systems + + TODO: Implement proper coordinate transformations for all camera angles + """
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (1)
code/perception/src/vision_node.py(7 hunks)
🧰 Additional context used
📓 Learnings (1)
code/perception/src/vision_node.py (1)
Learnt from: SirMDA
PR: una-auxme/paf#543
File: code/perception/src/vision_node.py:82-85
Timestamp: 2024-11-29T15:37:29.790Z
Learning: YOLOv11 segmentation models (`yolo11n-seg.pt`, `yolo11s-seg.pt`, `yolo11m-seg.pt`, `yolo11l-seg.pt`) are available and can be used in the codebase.
🔇 Additional comments (5)
code/perception/src/vision_node.py (5)
18-18: LGTM! Good separation of concerns.
The addition of helper functions get_carla_class_name and get_carla_color improves code organization by moving class-specific logic to a separate module.
82-85: LGTM! YOLOv11 segmentation models integration.
The addition of YOLOv11 segmentation models aligns with the PR objectives to enhance object detection capabilities.
248-248: Bug fix: Correct color channel encoding.
Changed image encoding from 'rgb8' to 'bgr8' to fix the color channel bug mentioned in the PR objectives.
361-361:
Add null check for masks attribute.
Direct access to masks.data could raise AttributeError if masks are not available.
458-471:
Add error handling for segmentation mask operations.
The segmentation mask scaling and drawing operations could fail if the masks are invalid or incompatible.
- scaled_masks = np.squeeze(
- scale_masks(masks.unsqueeze(1), cv_image.shape[:2], True).cpu().numpy(),
- 1,
- )
-
- drawn_images = draw_segmentation_masks(
- drawn_images,
- torch.from_numpy(scaled_masks > 0),
- alpha=0.6,
- colors=c_colors,
- )
+ try:
+ scaled_masks = np.squeeze(
+ scale_masks(masks.unsqueeze(1), cv_image.shape[:2], True).cpu().numpy(),
+ 1,
+ )
+
+ drawn_images = draw_segmentation_masks(
+ drawn_images,
+ torch.from_numpy(scaled_masks > 0),
+ alpha=0.6,
+ colors=c_colors,
+ )
+ except Exception as e:
+ self.get_logger().error(f"Failed to process segmentation masks: {str(e)}")
+ return cv2.cvtColor(np.transpose(drawn_images.detach().numpy(), (1, 2, 0)),
+ cv2.COLOR_BGR2RGB)Likely invalid or redundant comment.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Outside diff range and nitpick comments (1)
code/perception/src/vision_node.py (1)
Line range hint
1-624: Consider splitting the VisionNode class for better maintainability.The
VisionNodeclass is handling multiple responsibilities:
- ROS node management
- Model prediction
- Distance calculation
- Visualization
Consider splitting these into separate classes:
VisionNode: ROS node managementModelPredictor: Model prediction and trackingDistanceCalculator: Distance calculation logicVisualizer: Visualization operationsThis would improve:
- Code organization
- Testability
- Maintainability
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (1)
code/perception/src/vision_node.py(7 hunks)
🧰 Additional context used
📓 Learnings (1)
code/perception/src/vision_node.py (1)
Learnt from: SirMDA
PR: una-auxme/paf#543
File: code/perception/src/vision_node.py:82-85
Timestamp: 2024-11-29T15:37:29.790Z
Learning: YOLOv11 segmentation models (`yolo11n-seg.pt`, `yolo11s-seg.pt`, `yolo11m-seg.pt`, `yolo11l-seg.pt`) are available and can be used in the codebase.
🔇 Additional comments (3)
code/perception/src/vision_node.py (3)
18-18: LGTM! Well-structured model configuration and imports.
The addition of YOLOv11 segmentation models and helper functions is clean and follows the existing pattern. The separation of helper functions into a separate module improves code organization.
Also applies to: 28-28, 82-85
248-248: Verify color channel consistency across the pipeline.
The change from "rgb8" to "bgr8" encoding aligns with OpenCV's BGR format. However, we should verify that this change is consistent with:
- Input image processing
- Model expectations
- Visualization output
✅ Verification successful
Color channel conversions are correctly aligned throughout the pipeline
The verification shows a consistent color space handling pattern:
- Input images are converted from RGB to BGR (
COLOR_RGB2BGR) for processing - Model outputs are converted back to RGB (
COLOR_BGR2RGB) before visualization - Final output uses "bgr8" encoding which matches OpenCV's native format
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Check for potential color space conversion inconsistencies
# Search for color space conversions
rg "cvtColor.*COLOR_(BGR|RGB)" -A 2 -B 2
# Search for image encoding specifications
rg "encoding.*[\"'](rgb|bgr)" -A 2 -B 2
Length of output: 2972
453-475:
Add error handling for visualization operations.
The visualization code needs better error handling:
- The tracking operation could fail
- The mask visualization needs null checks
Apply these changes:
- drawn_images = draw_bounding_boxes(
- image_np_with_detections,
- c_boxes,
- c_labels,
- colors="blue",
- width=3,
- font_size=12,
- )
+ try:
+ drawn_images = draw_bounding_boxes(
+ image_np_with_detections,
+ c_boxes,
+ c_labels,
+ colors="blue",
+ width=3,
+ font_size=12,
+ )
+ except Exception as e:
+ self.get_logger().error(f"Failed to draw bounding boxes: {str(e)}")
+ return cv_image
if masks is not None:
- scaled_masks = np.squeeze(
- scale_masks(masks.unsqueeze(1), cv_image.shape[:2], True).cpu().numpy(),
- 1,
- )
+ try:
+ scaled_masks = np.squeeze(
+ scale_masks(masks.unsqueeze(1), cv_image.shape[:2], True).cpu().numpy(),
+ 1,
+ )
+ drawn_images = draw_segmentation_masks(
+ drawn_images,
+ torch.from_numpy(scaled_masks > 0),
+ alpha=0.6,
+ colors=c_colors,
+ )
+ except Exception as e:
+ self.get_logger().error(f"Failed to draw segmentation masks: {str(e)}")Likely invalid or redundant comment.
Description
Fixing issue #538 . This PR adds the latest YOLO nets (YoloV11) from ultralytics. The Yolo 11n is then activated in the vision_node. Object tracking is then activated. Also a bug with the color channels is fixed. The segmentation masks are used and displayed on the camera image.
Fixes #538
Type of change
Does this PR introduce a breaking change?
No
Most important changes
The model does not predict worse objects than the previous model. This is enough as we additionally add segmentation masks and tracking.
Checklist:
Summary by CodeRabbit
Release Notes
New Features
yolo11n-seg,yolo11s-seg,yolo11m-seg,yolo11l-seg).Bug Fixes
Documentation
Chores
ultralyticspackage version for improved performance.