Noisy results on Pandaset.

[hungdche]

Thank you for the great open-sourced project! I tried running inference on this given image from Pandaset, and the result looks a bit off. I am wondering if I am doing something wrong, and would appreciate your help!

Given condition image:

Outputted prediction.

IMG_000000.mp4

Command:

python sample.py --dataset IMG --low_vram --n_rounds 6

Changes after cloning the repo

• In /configs/inference/vista.yaml

--- a/configs/inference/vista.yaml
+++ b/configs/inference/vista.yaml
-   en_and_decode_n_samples_a_time: 14
-   num_frames: &num_frames 25
+   en_and_decode_n_samples_a_time: 1
+   num_frames: &num_frames 5

• In vwm/modules/encoders/modules.py, as suggested by this

--- a/vwm/modules/encoders/modules.py
+++ b/vwm/modules/encoders/modules.py
-   emb_out = embedder(batch[embedder.input_key])
+   emb_out_1s = [embedder(batch[embedder.input_key][i].unsqueeze(0)) for i in range(batch[embedder.input_key].shape[0])]
+   emb_out = torch.concat(emb_out_1s, 0)
+   # emb_out = embedder(batch[embedder.input_key])