Skip to content

Google Summer of Code GSoC ideas

Jump to bottom Edit New page
Frédéric Devernay edited this page Oct 8, 2015 · 12 revisions

This page contains ideas for development projects to be part of Google Summer of Code.

There are two main parts:

  • things that have to be done in the main application (called the host), which require skills in Qt and C++.
  • OpenFX plugins that have to be developped, which require skills in C++, computer graphics and image processing.

Natron host

Github project:

Natron heavily uses Qt, both for GUI and threads management. Natron still uses Qt 4, because the python bindings are still not available in Qt5. We will do the transition as soon as they become available.

The Natron code is split into two main parts: Engine and Gui. Engine manages the images, projects (including loading and saving), parameters, interface with OpenFX plugins (via the HostSupport library), and the rendering itself. Gui manages the graphical interfaces to all these things.

Rotoscoping

Curves editing

  • Curves consisting of many samples, such as tracking results or results from external algorithms, or imported ascii data, should be convertible to piecewise cubic curves for easier editing. Curves will be easier to edit if there are no cusp points at the junction (i.e. keyframes) between cubics (derivatives should be continuous). Several methods may be proposed and tested. Multi-dimensional parameters should share the same keyframes for all dimensions for easier editing.

Vectorscope, Waveform, Image analysis

  • Natron would benefit from vectorscope and waveform displays to complement the Histogram display.
  • Image analysis and picking can be improved by adding the possibility to draw a profile of the image values along a segment drawn by the user.

Color selecting dialog/widget

  • We need better color selection dialog (and widgets) than the platform-native one. For example, we need to be able to select negative values.

OpenFX plugins

Github projects:

An OpenFX plugin is usually a few hundred lines of code and follows the same global pattern.

There are several documentation resources available:

Convolve

We need an FFT-based generic convolution node, preferably using fftw3.

Two inputs: Source and Filter, with optional mask.

Lens aperture generators

We need basic basic lens aperture generators for Bokeh effects (circle, blades, with or without chromatic aberration, etc.). They can be combined with Convolve to get a Defocus effect.

The Flare node in Nuke does that, and a lot more.

Zdefocus

The goal of depth-dependent defocus is, given a depth image and a all-in-focus color image, to blur each pixel in the color image depending on its depth. Many algorithms are available in the litterature have been proposed, but some of them produced coarse approximations of the result. The implementation in Natron should at least handle correctly occlusions by foreground objects. We propose the following method:

  • extract "depth-slices" from the image, using the depth image. a depth slice is black and transparent everywhere except where the depth is within some range, where it is the original RGBA data.
  • blur each slice with the proper blur size (FFT-based convolution is necessary for a proper Bokeh effect). The in-focus slice is not blurred.
  • merge-over each slice, from the back to the from. That way, each blurred slice may occlude objects in the back.

Retiming / Warping / Distortion / Motion blur / Shutter unrolling

  • All these functionalities require the same basic tool: given a displacement function D(x,y) defined at each pixel (x,y) in the image, giving the displacement in pixels to apply at that point, compute the displaced image (this is sometimes called "pushing pixels"). Retiming uses the optical flow from and to the next image to compute an intermediate image by blending "pushed" images.
  • Pushing can be implemented easily in OpenGL by mesh warping. Mesa3D can be used for software rendering when OpenGL is not available.
  • Warping uses a function which can be given by given by lens distortion, a low-resolution deformation grid, a spline function, etc., to push pixels.
  • Motion blur is obtained by blending the generated images obtained over a range of displacements corresponding to the shutter time. It could work either by generating a fixed number of images over the shutter period and blending them, or by using an adaptative algorithm (see how it is implemented in ofxsTransform3x3.cpp using Metropolis sampling).
  • Lens distortion and convertion from/to polar coordinates are easy to implement once these tools are available.
  • Shutter unrolling is used to correct images taken with a rolling shutter camera (such as most CMOS-based cameras, including RED). In these images, each line is taken at a different time, causing deformation artifacts. Shutter unrolling consists in using the optical flow to generate a virtual image taken with a global shutter.

Pixel pushing may be implemented by drawing a warped mesh. A triangular mesh renderer has to be used (either an existing one with a compatible open-source licence, or one has to be written).

Corresponding Nuke nodes: Oflow, Kronos, Vectorblur, GridWarp, SplineMorph, LensDistortion, RollingShutter...

G'MIC meta-plugin

Work was started https://github.com/MrKepzie/openfx-gmic on writing a G'MIC meta-plugin, which should contain hundreds of plugins similar to what is available in GIMP's G'MIC plugin. There is also an interactive application with G'MIC, called ZArt, which implements a reduced number of effects. Maybe we should start with the plugins implemented by ZArt first.

Natural image matting (without green/blue screen)

http://www.wisdom.weizmann.ac.il/~levina/papers/Matting-Levin-Lischinski-Weiss-PAMI.pdf

http://www.wisdom.weizmann.ac.il/~levina/papers/Matting-Levin-Lischinski-Weiss-CVPR06.pdf

Matlab code available: http://people.csail.mit.edu/alevin/matting.tar.gz

Eduardo S. L. Gastal and Manuel M. Oliveira, Shared Sampling for Real-Time Alpha Matting, Eurographics, 2010

KNN matting: http://dingzeyu.li/projects/knn/index.html

For a comparison of matting methods: http://www.alphamatting.com/

Available implementation(s): vfx-matting

This implementation may either use CImg or OpenCV (but may also use neither), and may also use an external library such as CSparse (or any other sparse linear system solver with a compatible licence) for sparse system solving. All pointers to OpenFX documentation and guides are given above on this page. The OpenFX plugin should take two inputs: an image containing the strokes/samples (e.g. red for foreground, green/blue for background, black for unclassified), and the image for which to compute the matte. On output, the alpha channel of the image is set to the matte.

A CImg-based implementation would start from a fork of "openfx-misc", and an OpenCV-based implementation would start from a fork of "openfx-opencv".

See also:

SlitScan

A node that applies a per-pixel time offset. Historical reference: invented by Douglas Trumbull for 2001 (see video). For an example, see this video by Adrien M / Claire B.

The default time map is a linear function of y (as in the original slitscan), which is 0 at the top and 1 at the bottom of the image, but there can be an optionnal single-channel "TimeMap" input.

This plugin requires to render many frames on input, which may fill up the host cache, but if more than 4 frames required on input, Natron renders them on-demand, rather than pre-caching them (see @39d9960). Time offsets that "fall between" frames should either be mapped to the nearest frame (shutter=0), or averaged over several frames (shutter=1 corresponds to linear interpolation, higher shutter may blend more than two frames).

The parameters are:

  • offset for the TimeMap (default = -0.5)
  • gain for the TimeMap (default = 10)
  • absolute, a boolean indicating that the time map gives absolute frames rather than relative frames
  • frame range, only if TimeMap is connected, because the frame range cannot be guessed without looking at the image (default = -5..5). If "absolute" is checked, this frame range is absolute, else it's relative.

FFmpeg reader/writer enhancements

Audio

We should have an AudioRead-like node to show the audio waveform in the curve editor, and play it when flipbooking.

We should be able to mux an audio track in an output video, either from an audio file, or from another video file. See the ffmpeg.c source in the ffmpeg distribution for hints on how to do it.

Presets

done The writer should be simpler to use, with a few useful presets:

  • Create XDCAM HD422 files in .mov or .mxf
  • Create XDCAM IMX/D-10 files in .mov or .mxf
  • Create AVID DNxHD files in .mov
  • Create DVCPROHD files in .mov or .mxf
  • Create ProRes 422 or 4444 files in .mov

References:

Add a custom footer
Add a custom sidebar
Clone this wiki locally