How to prepare the dataset of AitW

[durianer-D]

I am very interested in your work, so I tried to follow your experiments. When I was preparing the dataset, I found that the downloaded AitW dataset file was in TensorFlow format, but the json file you provided would read the .png image. I don't know how to relate the encoded image in TensorFlow to the .png image in the json file you provided.

for example:
{"conversations": [{"from": "human", "value": "Please identify the labels of each marked region in the image. "}, {"from": "gpt", "value": "Region 1: Screenshot Text ('31')\nRegion 2: Screenshot Text ('Google Calendar')\nRegion 3: Screenshot Text ('Make the most of every day.')\nRegion 4: Screenshot ICON (V_FORWARD)\nRegion 5: Screenshot ICON (NAV_BAR_RECT)\nRegion 6: Screenshot ICON (V_BACKWARD)"}], "bbox": [[176, 248, 51, 43], [125, 446, 161, 21], [118, 490, 168, 14], [374, 636, 7, 29], [315, 695, 14, 21], [81, 695, 7, 21]], "image": "aitw/google_apps/images/google_apps_3619507004205175709_3.png"}

I want to know how to get the image name of google_apps_3619507004205175709_3.png from AitW dataset.

Can you help me? Looking forward to your reply very much.

[SiyuanHuang95]

Hi, thanks for your interest in our project.

You can refer to this official demo: https://github.com/google-research/google-research/blob/master/android_in_the_wild/demo.ipynb