Finetuning Vista

[hungdche]

Thank you for the excellent work! I am wondering if you are finetuning from the first phase of Vista, or from their final configuration.

[yunzhiy]

We finetune from the final checkpoint, the cross attention layer used for action control can be directly ignored.

[hungdche]

Thanks! By ignoring, do you remove it from their implementation, or just pass in a zero tensor?

[yunzhiy]

I just remove these parameters.

[hungdche]

Thank you! Final question. At the beginning of the training, did you experience similar issue as this, where the sampled images look like pure noise?

OpenDriveLab/Vista#13 (comment)

[yunzhiy]

I haven't met issues like this. I think maybe you can try setting use_ema to TRUE in the config file.