Ported to Nuke
- It works very poorly for movement since it blinking too much.
- It works poorly for very strong intensities and very marked shadows like that of the sun.
Code for the paper: Intrinsic Image Decomposition via Ordinal Shading, Chris Careaga and Yağız Aksoy, ACM Transactions on Graphics, 2023
Project Page | Paper | Video | Supplementary | Data
We propose a method for generating high-resolution intrinsic image decompositions, for in-the-wild images. Our method relies on a carefully formulated ordinal shading representation, and real-world supervision from multi-illumination data in order to predict highly accurate albedo and shading.
Try out our pipeline on your own images!
The inherently under-constrained and scale-invariant nature of the intrinsic decomposition makes it a challenging problem. Shading, which represents complex interactions in the scene, is difficult for neural networks to predict. Compounded by the scarcity of dense ground-truth data, state-of-the-art models fail at high resolutions in real-world scenarios.
Our method focuses on generating high-resolution shading estimations, rather than attempting to estimate shading and albedo separately. Since shading values are unbounded, we develop a representation of shading values called "inverse shading" which maps the shading values into the zero-one range. This creates a balanced distribution of values in a well-defined range that is desirable for training neural networks.
Rather than directly regressing the inverse shading values, we relax the problem and aim to predict ordinal shading values. To do this, we train our network using shift- and scale-invariant loss functions. This simplifies the task of shading estimation as the model does not need to estimate precise values that satisfy the core intrinsic decomposition model
Our ordinal estimations exhibit specific behaviors at different resolutions. At low resolutions, the model can generate globally coherent predictions, but the outputs lack details. At high resolutions, the model can predict fine local details, but at the cost of global coherency.
To generate a final shading estimation we combine two ordinal estimations, at low and high resolutions, with the input image and send them through a second network. We use the final shading estimation, and the input image in order to compute our estimated albedo. This allows us to compute losses on both shading and albedo while using only a single network.
We train our method on multiple rendered datasets. In order to generate real-world supervision for our method we use multi-illumination data. Using our pipeline we estimate the albedo for each image in a given multi-illumination scene. By taking the median across these albedo estimations, small errors are removed resulting in a single accurate albedo. We use these 25,000 pseudo-ground-truth pairs as training data and continue training our pipeline.
Our method can be used for complex image editing tasks such as recoloring and relighting
- Download and unzip the latest release from here.
- Copy the extracted Cattery folder to .nuke or your plugins path.
# Linux
git clone https://github.com/vinavfx/Intrinsic-for-Nuke.git
cd ./Intrinsic-for-Nuke
conda create -n intrinsic python=3.9
conda activate intrinsic
pip install .
wget https://github.com/compphoto/Intrinsic/releases/download/v1.0/final_weights.pt
python ./intrinsic_nuke.py
# Convert with CatFileCreator.nk@ARTICLE{careagaIntrinsic,
author={Chris Careaga and Ya\u{g}{\i}z Aksoy},
title={Intrinsic Image Decomposition via Ordinal Shading},
journal={ACM Trans. Graph.},
year={2023},
}
This implementation is provided for academic use only. Please cite our paper if you use this code or any of the models.
The methodology presented in this work is safeguarded under intellectual property protection. For inquiries regarding licensing opportunities, kindly reach out to SFU Technology Licensing Office <tlo_dir ατ sfu δøτ ca> and Dr. Yağız Aksoy <yagiz ατ sfu δøτ ca>.