diff --git a/depth_image_proc/doc/index.rst b/depth_image_proc/doc/index.rst index dd83269cc..d581149b2 100644 --- a/depth_image_proc/doc/index.rst +++ b/depth_image_proc/doc/index.rst @@ -1,5 +1,5 @@ -depth_image_proc -================ +Overview +======== ``depth_image_proc`` provides basic processing for depth images, much as ``image_proc`` does for traditional 2D images. The two packages are @@ -28,6 +28,7 @@ For an example of ``depth_image_proc`` in practice, examine the contents of .. toctree:: :maxdepth: 2 + self components tutorials changes diff --git a/image_pipeline/doc/camera_info.rst b/image_pipeline/doc/camera_info.rst new file mode 100644 index 000000000..84a1fee1f --- /dev/null +++ b/image_pipeline/doc/camera_info.rst @@ -0,0 +1,111 @@ +.. _Camera Info: + +Camera Info +=========== + +Camera Coordinate System +------------------------ +Below is a diagram of the camera coordinate system assumed by the +CameraInfo message. It is a right-handed system, with the world X +and Y aligned with the image x and y. This is the same coordinate +system used in OpenCV. It differs from the coordinate system of +Harley and Zisserman, which has Z forward, Y up, and X to the +left (looking towards +Z). + +|coords| + +.. |coords| image:: images/CameraCoords.png + +Derivation Diagram +------------------ + +Below is a diagram of the various images that are conceptually +derivable from the CameraInfo message. The boxed variables are +present in this message. + +|camera_info| + +.. |camera_info| image:: images/CameraInfo.png + +Projection Onto the Original Image +---------------------------------- +Starting with an initial 3D point X, the position of the corresponding +image point is found by going right-to-left in the upper half of the +diagram. The process is summarized by the following equations. + +.. code-block:: text + + X' = [R,t]X transform + sx = X' projection + x* = d(x) distortion + q = Kx* pixel coordinates + +This process uses the K camera matrix and D distortion vector from CameraInfo. +Ignoring the transform T, you can find the projection of a point X' into the +original camera frame using these equations. First, the 3D point X' is +projected onto the normalized, undistorted image via a projection operation +(division by Z). Then the distortion coefficients are used in the function +d() to move the point to its distorted position, still in a normalized image. +Finally, the normalized image is converted to a pixel-coordinate image by +applying the camera matrix to each image point. + +Rectification +------------- +Rectification is the process of transforming the input image into an output +image with the distortion corrected, and optionally the normalized image plane +transformed by rotation, optinain-image-plane translation, and optionally +in-image-plane scale. + +Simple Monocular Rectification +------------------------------ +In this case, there is a monocular device with distortion correction only. +To transform a pixel in the input image into one in the output image, it is +sent through the K - D - R - K' series of transformations. K - D gets to the +normalized, undistorted image; the rotation R is the identity because we +don't want to rotate the normalized undistorted image; and then K' converts +back to pixel coordinates in the output image. In this case, since there is +no need of in-image-plane translation, or scaling from the original image, +so K = K' , and only the K and D elements of CameraInfo are needed. +The ``camera_calibration`` package does monocular calibration to get K' +using OpenCV's getOptimalNewCameraMatrix() function with argument +'alpha'=0.0 that cause K' ≠ K. + +Projection onto the output image is by the P matrix, which is formed from K' +and the optional rotation and translation, in this case the identity and 0, +respectively. So if K' = K, P is redundant information, and could be formed +directly from K. + +Stereo and Complex Monocular Rectification +------------------------------------------ +After getting normalized undistorted image using K and D, here we apply a +transformation to the normalized, undistorted image: each points on this +image is rotated by R and then is passed to K' get the correspondent pixel +coordinate on the output image which is translated and scaled by K' with +respect to the original image. This transformation is important, for example, +in aligning two images of a stereo pair to lie in the same plane and have +coincident epipolar lines. + +The transform of a pixel from the input to output image is the same as in +the simple monocular case, except that R need not be the identity, and K' +need not be equal to K. As a consequence, the normalized undistorted image +can be rotated in 3D space, and the output image can be translated, scaled +relative to the original image. + +The projection matrix has an additional parameter, a translational offset t. +This parameter is used in the right stereo camera to reflect its external +position relative to the left camera. Since the point X in left camera +frame is transformed to the right camera by [I|t], that means the position +of the left camera frame origin with respect to the right camera frame is +given by the vector t in the right camera frame. + +Putting this all together, the projection matrix for the rectified output +image is given by: + +.. code-block:: text + + P = K' [I|t]. + +Note that in this case the internal camera parameters of the output image, +K', can be different from the input image, so P cannot be formed just +from K and D. And the I is the 3x3 rotation matrix representing left +camera frame in the right, not the identity. diff --git a/image_pipeline/doc/conf.py b/image_pipeline/doc/conf.py new file mode 100644 index 000000000..71dfb1c5e --- /dev/null +++ b/image_pipeline/doc/conf.py @@ -0,0 +1,193 @@ +# Copyright 2024 Open Source Robotics Foundation, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# -*- coding: utf-8 -*- +# +# Configuration file for the Sphinx documentation builder. +# +# This file does only contain a selection of the most common options. For a +# full list see the documentation: +# http://www.sphinx-doc.org/en/master/config + +# -- Path setup -------------------------------------------------------------- + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +import os +import sys +sys.path.insert(0, os.path.abspath('.')) + + +# -- Project information ----------------------------------------------------- + +project = 'image_pipeline' +copyright = '2008-2024, Open Source Robotics Foundation, Inc.' # noqa +author = 'Open Source Robotics Foundation, Inc.' + +# The short X.Y version +version = '' +# The full version, including alpha/beta/rc tags +release = '3.2.1' + + +# -- General configuration --------------------------------------------------- + +# If your documentation needs a minimal Sphinx version, state it here. +# +# needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [ + 'sphinx.ext.autodoc', + 'sphinx.ext.autosummary', + 'sphinx.ext.doctest', + 'sphinx.ext.coverage', + 'sphinx_rtd_theme', +] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The master toctree document. +master_doc = 'index' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This pattern also affects html_static_path and html_extra_path. +exclude_patterns = [] + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = None + + +# -- Options for HTML output ------------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'sphinx_rtd_theme' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# +# html_theme_options = {} + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = [] + +# Custom sidebar templates, must be a dictionary that maps document names +# to template names. +# +# The default sidebars (for documents that don't match any pattern) are +# defined by theme itself. Builtin themes are using these templates by +# default: ``['localtoc.html', 'relations.html', 'sourcelink.html', +# 'searchbox.html']``. +# +# html_sidebars = {} + + +# -- Options for HTMLHelp output --------------------------------------------- + +# Output file base name for HTML help builder. +htmlhelp_basename = 'image_pipeline_doc' + + +# -- Options for LaTeX output ------------------------------------------------ + +latex_elements = { + # The paper size ('letterpaper' or 'a4paper'). + # + # 'papersize': 'letterpaper', + + # The font size ('10pt', '11pt' or '12pt'). + # + # 'pointsize': '10pt', + + # Additional stuff for the LaTeX preamble. + # + # 'preamble': '', + + # Latex figure (float) alignment + # + # 'figure_align': 'htbp', +} + + +# -- Options for manual page output ------------------------------------------ + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + (master_doc, 'image_pipeline', 'image_pipeline Documentation', + [author], 1) +] + + +# -- Options for Texinfo output ---------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + (master_doc, 'image_pipeline', 'image_pipeline Documentation', + author, 'image_pipeline', 'ROS 2 components for image processing.', + 'Miscellaneous'), +] + + +# -- Options for Epub output ------------------------------------------------- + +# Bibliographic Dublin Core info. +epub_title = project + +# The unique identifier of the text. This can be a ISBN number +# or the project homepage. +# +# epub_identifier = '' + +# A unique identification for the text. +# +# epub_uid = '' + +# A list of files that should not be packed into the epub file. +epub_exclude_files = ['search.html'] + + +# -- Extension configuration ------------------------------------------------- + +autoclass_content = 'both' + +autodoc_default_options = { + 'members': True, # document members + 'undoc-members': True, # also document members without documentation +} diff --git a/image_pipeline/doc/images/CameraCoords.png b/image_pipeline/doc/images/CameraCoords.png new file mode 100644 index 000000000..82d30ce5b Binary files /dev/null and b/image_pipeline/doc/images/CameraCoords.png differ diff --git a/image_pipeline/doc/images/CameraInfo.png b/image_pipeline/doc/images/CameraInfo.png new file mode 100644 index 000000000..655622da9 Binary files /dev/null and b/image_pipeline/doc/images/CameraInfo.png differ diff --git a/image_pipeline/doc/index.rst b/image_pipeline/doc/index.rst new file mode 100644 index 000000000..a74210645 --- /dev/null +++ b/image_pipeline/doc/index.rst @@ -0,0 +1,43 @@ +Overview +======== + +The ``image_pipeline`` stack is designed to process raw camera images +into useful inputs to vision algorithms: rectified mono/color images, +stereo disparity images, and stereo point clouds. Components include: + + * **Calibration**: Cameras must be calibrated in order to relate the + images they produce to the three-dimensional world. The + ``camera_calibration`` package provides tools to calibrate monocular + and stereo cameras in your ROS system. The :ref:`Camera Info` page + provides a detailed description of the parameters used by the + pipeline. + * **Monocular processing**: The raw image stream can be piped through + the ``image_proc`` node to remove camera distortion. The node also + performs color interpolation for Bayer pattern color cameras. + * **Stereo processing**: The ``stereo_image_proc`` package performs + the duties of ``image_proc`` for a pair of cameras co-calibrated + for stereo vision. It also uses stereo processing to produce + disparity images and point clouds. + * **Depth processing**: ``depth_image_proc`` provides components + for processing depth images (as produced by the Kinect, + time-of-flight cameras, etc.), such as producing point clouds. + * **Visualization**: The ``image_view`` package provides a lightweight + alternative to ``rviz2`` for viewing an image topic. It also includes + a ``stereo_view`` tool for viewing stereo pairs and disparity images. + +.. toctree:: + :maxdepth: 2 + + self + camera_info + camera_calibration <../camera_calibration/index.html#http://> + depth_image_proc <../depth_image_proc/index.html#http://> + image_proc <../image_proc/index.html#http://> + image_view <../depth/image_view/index.html#http://> + stereo_image_proc <../depth/stereo_image_proc/index.html#http://> + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`search` diff --git a/image_pipeline/package.xml b/image_pipeline/package.xml index 06f421926..46eae3409 100644 --- a/image_pipeline/package.xml +++ b/image_pipeline/package.xml @@ -34,5 +34,6 @@ ament_cmake + rosdoc2.yaml diff --git a/image_pipeline/rosdoc2.yaml b/image_pipeline/rosdoc2.yaml new file mode 100644 index 000000000..85ac4c03c --- /dev/null +++ b/image_pipeline/rosdoc2.yaml @@ -0,0 +1,20 @@ +type: 'rosdoc2 config' +version: 1 + +--- + +settings: + generate_package_index: false + always_run_doxygen: false + enable_breathe: false + enable_exhale: false + always_run_sphinx_apidoc: false + override_build_type: 'ament_python' + python_source: 'doc' + +builders: + - sphinx: { + name: 'image_pipeline', + sphinx_sourcedir: 'doc', + output_dir: '' + } \ No newline at end of file diff --git a/image_proc/doc/index.rst b/image_proc/doc/index.rst index 139100e77..4e764ca48 100644 --- a/image_proc/doc/index.rst +++ b/image_proc/doc/index.rst @@ -1,5 +1,5 @@ -image_proc -========== +Overview +======== This package contains a number of ROS 2 components and nodes for image processing. @@ -21,6 +21,7 @@ The images above were produced using the tutorial :ref:`Using image_proc Launch .. toctree:: :maxdepth: 2 + self components tutorials changes