Modifications for the upcoming Render-Based Tracking submodule

doc/tutorial/rendering/tutorial-panda3d.dox

@@ -165,31 +165,82 @@ vpPanda3DBaseRenderer, which implements basic functions for a panda renderer.
 
 - Installer are available for Windows browsing the [download](https://www.panda3d.org/download/) page.
 
-\section tutorial-panda3d-usage Rendere based on Panda3D usage
+\section tutorial-panda3d-usage Using Panda3D for rendering
 
 An example that shows how to exploit Panda3D in ViSP to render a color image with support for textures and lighting, a
-depth image, normals in world space and in camera space is given in tutorial-panda3d-renderer.cpp.
+depth image, normals in object space and in camera space is given in tutorial-panda3d-renderer.cpp.
 
-Here you will find the code used to create the renderer:
+
+To start rendering, we first instanciate a vpPanda3DRendererSet. This object allows to render multiple modalities (color, depth, etc.) in a single pass.
+To add different rendering modalities, we will use subclasses that will be registered to the renderer set.
+Internally, each sub renderer has its own scene: the renderer set synchronizes everything when the state changes (i.e, an object is added, an object is moved or the camera parameters change.)
+
+A panda3D renderer should be instanciated with a vpPanda3DRenderParameters object. This object defines:
+- The camera intrinsics (see vpCameraParameters): As of now, Only parameters for a distortion free model are supported
+- The image resolution
+- The near and far clipping plane values. Object parts that are too close (less than the near clipping value) or too far (greater than the far clipping value) will not be rendered.
+
+The creation of the renderer set can be found below
 \snippet tutorial-panda3d-renderer.cpp Renderer set
 
-Here you will find the code used to create the sub renderers:
+
+To actually render color, normals, etc., we need to define subrenderers:
 \snippet tutorial-panda3d-renderer.cpp Subrenderers init
 
-Here you will find the code used to add the sub renderers to the main renderer:
+The different subrenderers are:
+
+- vpPanda3DGeometryRenderer instances allow to retrieve 3D information about the object: these are the surface normals in the object or camera frame, as well as the depth information.
+- vpPanda3DRGBRenderer objects perform the traditional color rendering. Lighting interaction can be disable, as is the case for the second renderer (diffuse only).
+- Post processing renderers, such as vpPanda3DLuminanceFilter, vpPanda3DCanny, operate on the output image of another renderer. They can be used to further process the output data and can be chained together.
+In this case, the chain vpPanda3DLuminanceFilter -> vpPanda3DGaussianBlur -> vpPanda3DCanny will perform a canny edge detection (without hysteresis) on a blurred, grayscale image.
+
+For these subrenderers to actually be useful, they should be added to the main renderer:
 \snippet tutorial-panda3d-renderer.cpp Adding subrenderers
 
+\warning Once they have been added, a call to vpPanda3DBaseRenderer::initFramework() should be performed. Otherwise, no rendering will be performed and objects will not be loaded.
+
 Here you will find the code used to configure the scene:
 \snippet tutorial-panda3d-renderer.cpp Scene configuration
 
+We start by loading the object to render with vpPanda3DBaseRenderer::loadObject, followed by vpPanda3DBaseRenderer::addNodeToScene.
+For the Color-based renderer, we add lights to shade our object. Different light types are supported, reusing the available Panda3D features.
+
+Once the scene is setup, we can start rendering. This will be performed in a loop.
+
+The first step shown is the following:
+\snippet tutorial-panda3d-renderer.cpp Updating render parameters
+
+Each frame, we compute the values of the clipping planes, and update the rendering properties. This will ensure that the target object is visible.
+Depending on your use case, this may not be necessary.
+
+Once this is done, we can call upon Panda3D to render the object with
+\snippet tutorial-panda3d-renderer.cpp Render frame
+
+\note Note that under the hood, all subrenderers rely on the same Panda3D "framework": calling renderFrame on one will call it for the others.
+
+To use the renders, we must convert them to ViSP images.
+To do so, each subrenderer defines its own *getRender* method, that will perform the conversion from a panda texture to the relevant ViSP data type.
+
+For each render type, we start by getting the correct renderer via the vpPanda3DRendererSet::getRenderer, then call its *getRender* method.
+\snippet tutorial-panda3d-renderer.cpp Fetch render
+
+Now that we have the retrieved the images, we can display them. To do so, we leverage utility functions defined beforehand (see the full code for more information). This may be required in cases where the data cannot be directly displayed.
+For instance, normals are encoded as 3 32-bit floats, but displays require colors to be represented as 8-bit unsigned characters.
+The same goes for the depth render, which is mapped back to the 0-255 range, although its value unbound.
+\snippet tutorial-panda3d-renderer.cpp Display
+
+Finally, we use the snippet below to move the object, updating the scene.
+To have a constant velocity, we multiply the displacement by the time that has elapsed between the frame's start and end.
+\snippet tutorial-panda3d-renderer.cpp Move object
+
 \section tutorial-panda3d-full-code Tutorial full code
 
 The full code of tutorial-panda3d-renderer.cpp is given below.
 \include tutorial-panda3d-renderer.cpp
 
-\section tutorial-panda3d-run Execute the tutorial
+\section tutorial-panda3d-run Running the tutorial
 
-- Once ViSP is build, you may run the tutorial by:
+- Once ViSP is built, you may run the tutorial by:
   \code{.sh}
   $ cd $VISP_WS/visp-build
   $ ./tutorial/ar/tutorial-panda3d-renderer

modules/ar/include/visp3/ar/vpPanda3DBaseRenderer.h

@@ -71,17 +71,18 @@ class VISP_EXPORT vpPanda3DBaseRenderer
    * Will also perform the renderer setup (scene, camera and render targets)
    */
   virtual void initFramework();
+  virtual void initFromParent(std::shared_ptr<PandaFramework> framework, PointerTo<WindowFramework> window);
+  virtual void initFromParent(const vpPanda3DBaseRenderer &renderer);
 
-  /**
-   * @brief
-   *
-   * @param framework
-   * @param window
-   */
-  void initFromParent(std::shared_ptr<PandaFramework> framework, std::shared_ptr<WindowFramework> window);
-
-
+  virtual void beforeFrameRendered() { }
   virtual void renderFrame();
+  virtual void afterFrameRendered()
+  {
+    GraphicsOutput *mainBuffer = getMainOutputBuffer();
+    if (mainBuffer != nullptr) {
+      m_framework->get_graphics_engine()->extract_texture_data(mainBuffer->get_texture(), mainBuffer->get_gsg());
+    }
+  }
 
   /**
    * @brief Get the name of the renderer
@@ -90,6 +91,8 @@ class VISP_EXPORT vpPanda3DBaseRenderer
    */
   const std::string &getName() const { return m_name; }
 
+  void setName(const std::string &name) { m_name = name; }
+
   /**
    * @brief Get the scene root
    *
@@ -101,31 +104,7 @@ class VISP_EXPORT vpPanda3DBaseRenderer
    *
    * @param params the new rendering parameters
    */
-  virtual void setRenderParameters(const vpPanda3DRenderParameters &params)
-  {
-    unsigned int previousH = m_renderParameters.getImageHeight(), previousW = m_renderParameters.getImageWidth();
-    bool resize = previousH != params.getImageHeight() || previousW != params.getImageWidth();
-
-    m_renderParameters = params;
-
-    if (resize) {
-      for (GraphicsOutput *buffer: m_buffers) {
-        //buffer->get_type().is_derived_from()
-        GraphicsBuffer *buf = dynamic_cast<GraphicsBuffer *>(buffer);
-        if (buf == nullptr) {
-          throw vpException(vpException::fatalError, "Panda3D: could not cast to GraphicsBuffer when rendering.");
-        }
-        else {
-          buf->set_size(m_renderParameters.getImageWidth(), m_renderParameters.getImageHeight());
-        }
-      }
-    }
-
-    // If renderer is already initialized, modify camera properties
-    if (m_camera != nullptr) {
-      m_renderParameters.setupPandaCamera(m_camera);
-    }
-  }
+  virtual void setRenderParameters(const vpPanda3DRenderParameters &params);
 
   /**
    * @brief Returns true if this renderer process 3D data and its scene root can be interacted with.
@@ -143,6 +122,15 @@ class VISP_EXPORT vpPanda3DBaseRenderer
    */
   int getRenderOrder() const { return m_renderOrder; }
 
+  void setRenderOrder(int order)
+  {
+    int previousOrder = m_renderOrder;
+    m_renderOrder = order;
+    for (GraphicsOutput *buffer: m_buffers) {
+      buffer->set_sort(buffer->get_sort() + (order - previousOrder));
+    }
+  }
+
   /**
    * @brief Set the camera's pose.
    * The pose is specified using the ViSP convention (Y-down right handed).
@@ -206,8 +194,10 @@ class VISP_EXPORT vpPanda3DBaseRenderer
    * @param name name of the node that should be used to compute near and far values.
    * @param near resulting near clipping plane distance
    * @param far resulting far clipping plane distance
+   * @param fast Whether to use the axis align bounding box to compute the clipping planes.
+   * This is faster than reprojecting the full geometry in the camera frame
    */
-  void computeNearAndFarPlanesFromNode(const std::string &name, float &near, float &far);
+  void computeNearAndFarPlanesFromNode(const std::string &name, float &near, float &far, bool fast);
 
   /**
    * @brief Load a 3D object. To load an .obj file, Panda3D must be compiled with assimp support.
@@ -250,6 +240,8 @@ class VISP_EXPORT vpPanda3DBaseRenderer
 
   virtual GraphicsOutput *getMainOutputBuffer() { return nullptr; }
 
+  virtual void enableSharedDepthBuffer(vpPanda3DBaseRenderer &sourceBuffer);
+
 protected:
 
   /**
@@ -272,16 +264,14 @@ class VISP_EXPORT vpPanda3DBaseRenderer
    */
   virtual void setupRenderTarget() { }
 
-
   const static vpHomogeneousMatrix VISP_T_PANDA; //! Homogeneous transformation matrix to convert from the Panda coordinate system (right-handed Z-up) to the ViSP coordinate system (right-handed Y-Down)
   const static vpHomogeneousMatrix PANDA_T_VISP; //! Inverse of VISP_T_PANDA
 
-
 protected:
-  const std::string m_name; //! name of the renderer
+  std::string m_name; //! name of the renderer
   int m_renderOrder; //! Rendering priority for this renderer and its buffers. A lower value will be rendered first. Should be used when calling make_output in setupRenderTarget()
   std::shared_ptr<PandaFramework> m_framework; //! Pointer to the active panda framework
-  std::shared_ptr<WindowFramework> m_window; //! Pointer to owning window, which can create buffers etc. It is not necessarily visible.
+  PointerTo<WindowFramework> m_window; //! Pointer to owning window, which can create buffers etc. It is not necessarily visible.
   vpPanda3DRenderParameters m_renderParameters; //! Rendering parameters
   NodePath m_renderRoot; //! Node containing all the objects and the camera for this renderer
   PointerTo<Camera> m_camera;

modules/ar/include/visp3/ar/vpPanda3DGeometryRenderer.h

@@ -36,6 +36,7 @@
 #if defined(VISP_HAVE_PANDA3D)
 #include <visp3/core/vpCameraParameters.h>
 #include <visp3/core/vpImage.h>
+#include <visp3/core/vpRect.h>
 #include <visp3/ar/vpPanda3DBaseRenderer.h>
 
 BEGIN_VISP_NAMESPACE
@@ -71,6 +72,8 @@ class VISP_EXPORT vpPanda3DGeometryRenderer : public vpPanda3DBaseRenderer
    */
   void getRender(vpImage<vpRGBf> &colorData, vpImage<float> &depth) const;
 
+  void getRender(vpImage<vpRGBf> &normals, vpImage<float> &depth, const vpRect &bb, unsigned int h, unsigned w) const;
+
   /**
    * @brief Get render results into ViSP readable structures. This version only retrieves the normal data
    * @param colorData Depending on the vpRenderType, normals in the world or camera frame may be stored in this image.
@@ -84,6 +87,7 @@ class VISP_EXPORT vpPanda3DGeometryRenderer : public vpPanda3DBaseRenderer
 
   GraphicsOutput *getMainOutputBuffer() VP_OVERRIDE { return m_normalDepthBuffer; }
 
+
 protected:
   void setupScene() VP_OVERRIDE;
   void setupRenderTarget() VP_OVERRIDE;
@@ -93,7 +97,7 @@ class VISP_EXPORT vpPanda3DGeometryRenderer : public vpPanda3DBaseRenderer
   Texture *m_normalDepthTexture;
   GraphicsOutput *m_normalDepthBuffer;
 
-  static const char *SHADER_VERT_NORMAL_AND_DEPTH_WORLD;
+  static const char *SHADER_VERT_NORMAL_AND_DEPTH_OBJECT;
   static const char *SHADER_VERT_NORMAL_AND_DEPTH_CAMERA;
   static const char *SHADER_FRAG_NORMAL_AND_DEPTH;
 

modules/ar/include/visp3/ar/vpPanda3DLight.h

@@ -209,7 +209,6 @@ class VISP_EXPORT vpPanda3DDirectionalLight : public vpPanda3DLight
     PT(DirectionalLight) light = new DirectionalLight(m_name);
     light->set_color(LColor(m_color.R, m_color.G, m_color.B, 1));
     vpColVector dir = vpPanda3DBaseRenderer::vispVectorToPanda(m_direction);
-    std::cout << m_direction << ", " << dir << std::endl;
     light->set_direction(LVector3f(m_direction[0], m_direction[1], m_direction[2]));
     NodePath np = scene.attach_new_node(light);
     scene.set_light(np);

modules/ar/include/visp3/ar/vpPanda3DPostProcessFilter.h

@@ -62,13 +62,26 @@ class VISP_EXPORT vpPanda3DPostProcessFilter : public vpPanda3DBaseRenderer
     m_renderOrder = m_inputRenderer->getRenderOrder() + 1;
   }
 
+  ~vpPanda3DPostProcessFilter()
+  {
+    // delete m_texture;
+    // delete m_buffer;
+  }
+
   bool isRendering3DScene() const VP_OVERRIDE
   {
     return false;
   }
 
   GraphicsOutput *getMainOutputBuffer() VP_OVERRIDE { return m_buffer; }
 
+  void afterFrameRendered() VP_OVERRIDE
+  {
+    if (m_isOutput) {
+      vpPanda3DBaseRenderer::afterFrameRendered();
+    }
+  }
+
 protected:
   virtual void setupScene() VP_OVERRIDE;
 
@@ -81,7 +94,6 @@ class VISP_EXPORT vpPanda3DPostProcessFilter : public vpPanda3DBaseRenderer
   void getRenderBasic(vpImage<unsigned char> &I) const;
   void getRenderBasic(vpImage<vpRGBf> &I) const;
 
-
   virtual FrameBufferProperties getBufferProperties() const = 0;
 
   std::shared_ptr<vpPanda3DBaseRenderer> m_inputRenderer;

modules/ar/include/visp3/ar/vpPanda3DRGBRenderer.h

@@ -78,7 +78,6 @@ class VISP_EXPORT vpPanda3DRGBRenderer : public vpPanda3DBaseRenderer, public vp
    */
   vpPanda3DRGBRenderer(bool showSpeculars) : vpPanda3DBaseRenderer(showSpeculars ? "RGB" : "RGB-diffuse"), m_showSpeculars(showSpeculars) { }
 
-
   /**
    * @brief Store the render resulting from calling renderFrame() into a vpImage.
    *
@@ -96,10 +95,10 @@ class VISP_EXPORT vpPanda3DRGBRenderer : public vpPanda3DBaseRenderer, public vp
 
   bool isShowingSpeculars() const { return m_showSpeculars; }
 
-
 protected:
   void setupScene() VP_OVERRIDE;
   void setupRenderTarget() VP_OVERRIDE;
+
   virtual std::string makeFragmentShader(bool hasTexture, bool specular);
 
 private:
@@ -112,7 +111,6 @@ class VISP_EXPORT vpPanda3DRGBRenderer : public vpPanda3DBaseRenderer, public vp
   NodePath m_backgroundImage;
   DisplayRegion *m_display2d;
   Texture *m_backgroundTexture;
-
 };
 
 END_VISP_NAMESPACE

modules/ar/include/visp3/ar/vpPanda3DRendererSet.h

@@ -71,6 +71,8 @@ class VISP_EXPORT vpPanda3DRendererSet : public vpPanda3DBaseRenderer, public vp
    * Thus, if a renderer B depends on A for its render, and if B.getRenderOrder() > A.getRenderOrder() it can rely on A being initialized when B.initFromParent is called (along with the setupCamera, setupRenderTarget).
    */
   void initFramework() VP_OVERRIDE;
+  void initFromParent(std::shared_ptr<PandaFramework> framework, PointerTo<WindowFramework> window) VP_OVERRIDE;
+  void initFromParent(const vpPanda3DBaseRenderer &renderer) VP_OVERRIDE;
 
   /**
    * @brief Set the pose of the camera, using the ViSP convention. This change is propagated to all subrenderers
@@ -134,7 +136,7 @@ class VISP_EXPORT vpPanda3DRendererSet : public vpPanda3DBaseRenderer, public vp
    */
   void addNodeToScene(const NodePath &object) VP_OVERRIDE;
 
-  void setRenderParameters(const vpPanda3DRenderParameters &params) VP_OVERRIDE;
+  virtual void setRenderParameters(const vpPanda3DRenderParameters &params) VP_OVERRIDE;
 
   void addLight(const vpPanda3DLight &light) VP_OVERRIDE;
 
@@ -147,6 +149,8 @@ class VISP_EXPORT vpPanda3DRendererSet : public vpPanda3DBaseRenderer, public vp
    */
   void addSubRenderer(std::shared_ptr<vpPanda3DBaseRenderer> renderer);
 
+  void enableSharedDepthBuffer(vpPanda3DBaseRenderer &sourceBuffer) VP_OVERRIDE;
+
   /**
    * @brief Retrieve the first subrenderer with the specified template type.
    *
@@ -164,6 +168,7 @@ class VISP_EXPORT vpPanda3DRendererSet : public vpPanda3DBaseRenderer, public vp
     }
     return nullptr;
   }
+
   /**
    * @brief Retrieve the subrenderer with the specified template type and the given name.
    *
@@ -185,12 +190,25 @@ class VISP_EXPORT vpPanda3DRendererSet : public vpPanda3DBaseRenderer, public vp
     return nullptr;
   }
 
+  void beforeFrameRendered() VP_OVERRIDE
+  {
+    for (std::shared_ptr<vpPanda3DBaseRenderer> &renderer: m_subRenderers) {
+      renderer->beforeFrameRendered();
+    }
+  }
+
+  void afterFrameRendered() VP_OVERRIDE
+  {
+    for (std::shared_ptr<vpPanda3DBaseRenderer> &renderer: m_subRenderers) {
+      renderer->afterFrameRendered();
+    }
+  }
+
 protected:
   void setupScene() VP_OVERRIDE { }
 
   void setupCamera() VP_OVERRIDE { }
 
-private:
   std::vector<std::shared_ptr<vpPanda3DBaseRenderer>> m_subRenderers;
 };
 END_VISP_NAMESPACE