About affine transformation

[Korace0v0]

Hi, thanks for your great work.

I am confused about the `Affine' model, which is used to transfrom 2D rgb image. I could not understant it's effect. Why do we really need this transformation?

It seems affine transformation is only applied to rgb_sky_blend, while depth is not transformed. Depth is then supervised by Lidar. As I can understand, if Affine model is used to transfrom 2D image, does depth also need to be transformed to align with rgb?

Thanks for your help!

[ziyc]

Hi @Korace0v0, This is a very good question.

Affine transformation was used in Urban Radiance Field (section 4.1.1) to handle inconsistent exposure of images.

A similar exposure situation occurs in driving data. For images within the same sequence, the exposure, color balance might differ across cameras and frames. These inconsistencies could hurt the reconstruction quality. The affine transformation is used to encode camera-specific features related to color balance, exposure etc. This way, the inconsistent exposures won't be fitted by the scene representation.

[Korace0v0]

I understand it. Thank you very much!

[pierremerriaux-leddartech]

Hi @ziyc ,
i think there is a issue with Affine module gradient propagation, due to 0 mlp init. I proposed a PR here #38
thanks
Pierre