Offscreen rendering

NPBG uses a technique called offscreen rendering to train models on headless hosts. It allows to generate datasets on-line to save disk space and avoid generating new datasets when the neural network input data format has changed.

DynamicDataset

To fit descriptor points we use DynamicDataset class. This class internally calls OpenGL procedures to render point clouds on-line when data loader iterates trough the dataset. DynamicDataset has to be provided with:

• scene data, i.e. point cloud (point position, color, normal etc.), projection matrix and view matrices for each data image in the dataset.
• rendering format, for example you can specify point colors and point ids with format string uv_1d_ps1, colors_ps1, where uv_1d stands for an integer point id, ps1 for point of size 1px on distance 1 (splatting mode) and colors for an RGB point color usually obtained from SfM algorithm.
• target image paths (those the scene was reconstructed from) and optionally masks and point labels.

DynamicDataset supports basic augmentations like random zoom and random crop. Basically to get an augmented point cloud rendering DynamicDataset modifies the projection matrix to account for viewport zoom or shift in XY plane (see _get_intrinsics). The target images are augmented using perspective warping (see _warp).

MultiscalRenderer

MultiscaleRenderer class is responsible for rendering point cloud attributes at multiple scales. If you define the rendering format string like uv_1d_p1, uv_1d_p1_ds1, uv_1d_p1_ds2, uv_1d_p1_ds3, uv_1d_p1_ds4 then it will render 5 images with spatial sizes decreasing by a factor of 2, f.e. 512x512, 256x256, 128x128, 64x64 and 32x32. MultiscaleRenderer is also responsible for setting the right shader attributes when rendering dataset items (be that point clouds or meshes).

OffscreenRender

OffscreenRender abstracts OpenGL function calls and renders a given scene to an OpenGL texture buffer, a Pytorch tensor or a Numpy array. Its method render requires a scene (an instance of class NNScene) with set up view and projection matrices, rendering mode (points / triangles) and other shader attributes.

NNScene

NNScene class is a convenient wrapper around an OpenGL program shader (vertex and fragment shaders). It assigns scene's projection, model and view matrices, point attributes like color, normal, position, id as well as mesh attributes like texture UV coords. It also control rendering modes, whether to render a colors, normals, point ids/mesh UV, sets the point size etc. See setup_scene for an example how NNScene is filled with data usually loaded using load_scene_data.