key points extraction in textureless region

[zhaoxin111]

Excellent job, but I have a question. When there is a large area of sky in the picture with little texture, can the key point detection model based on CNN be able to effectively extract the key points?

[Annbless]

Sorry for the late reply. I am afraid that the CNN-based keypoint detection model cannot extract enough keypoints in such areas without texture and edges. I visualized the keypoints extracted in the case you mentioned. The CNN model can extract keypoints on the edges of clouds, but not in texture-free regions such as the sky. However, this does not affect the final performance of the video stabilization, because the distortion of the texture-free regions is insignificant.

[zhaoxin111]

Thank you for your reply, I noticed that in the ablation study, replacing the FAST keypoint detector and the KLT tracker with RFNet and PWCNet respectively, the score of stability, distortion and cropping is evey close to the DUT results. Is that mean the keypoint detector and tracker are the key resulting in good performence in DUT? Will traditional method like MeshFlow also use deep learning-based feature extractors and trackers also have a significant performance improvement?

[Annbless]

Not exactly. Please refer to Figures 3 and 5 in the paper. The distortion metric only describes the global artifacts as the frame ratio changes, but it does not measure the artifacts that affect visual quality. The deep learning-based motion estimation and trajectory smoothing module proposed in this paper is specifically designed to correct such artifacts and thus polish the visual experience. Moreover, this is the reason why we use additional metrics to measure the performance of the motion estimation module. Does this answer solve your problem?

[zhaoxin111]

Your answer basically answered my doubts. I will read your paper again, and there should be new gains.