KeyError: '11183447'

[bupt-cist]

Hi, I uploaded test.json to evalai (test2019 phase) and got the following issues:

Traceback (most recent call last):
File "/code/scripts/workers/submission_worker.py", line 336, in run_submission
submission_metadata=submission_serializer.data,
File "/tmp/tmpnr9tlxl6/compute/challenge_data/challenge_225/main.py", line 96, in evaluate
output["result"].append({tier: getScores(questions, questions, predictions, tier, kwargs['submission_metadata']['method_name'])})
File "/tmp/tmpnr9tlxl6/compute/challenge_data/challenge_225/main.py", line 315, in getScores
predicted = predictions[qid]
KeyError: '11183447'

I find that the question id '11183447' is in validation split, not in the test split.
So, it is strange that there is a keyerror here in test2019 phase.

[Cadene]

I had the same issue.

To evaluate results on GQA, please submit a JSON file which contains answer to each question from the submission_all_questions.json. The submission format should be the following:

source

[bupt-cist]

I want to participate in test2019. Does that mean I have to submit a JSON file which contains BOTH answers for submission_all_questions.json and test_all_questions.json? @Cadene

[Cadene]

submission only

[chenfei-wu]

You are right. '11183447' is exactly the first question_id in submission_all_questions.json. Thanks a lot! @Cadene

[dorarad]

Hi, sorry for the late reply! you should submit answers for all the questions in submission_all_questions.json. However, we will evaluate you based only on the split you choose to submit to (we use the others just to verify overall validity of the submission file, following evalAI practice).