Pointing Data Usage Issue

[youthhoo]

When I finetuning the pretrained model, I find the pixmo pointing data was divided into four parts, i.e., pixmo_points, pixmo_points_counting, pixmo_points_high_freq and pixmo_points_high_freq_counting. I wonder the reason to do that, and if it can improve the improvement of pointing.

[chrisc36]

The parts are divided into counting vs. not-counting, and then into (high-frequency vs low-frequency). The high-frequency data was collected through a slightly different pipeline that encourage annotators to focus on high-frequency objects. Counting/non-counting just formats the inputs differently.

In practice, I don't think it would make a big difference to merge all these parts into one big pointing dataset. It is just that way in the code due to how the data was collected, and to conveniently have both pointing and counting examples built from each point annotation.

The exception is that using these splits effectively up-samples the high-frequency data bit because smaller datasets are upsampled slightly, and the high-frequency dataset is small, although this is not something we have actually ablated.