Need suggestion for finetuning this model for DOC-VQA task

[shubhamReal]

Since this model is very good at OCR. I was thinking I could finetune it for general document understanding. I finetuned the model for docvqa using swift but results were not great.
Now I am thinking to do the SFT training with cauldron and docmatx dataset first and then do the finetuning on DOC-VQA.

i need opinion of Authors of this paper

[Ucas-HaoranWei]

Hello, I don't think this model will be good at DocVQA:

1. VQA task benefits from the LLM, the larger the better. Yet GOT is only 0.5B.
2. DocVQA needs more vision tokens (e.g., the compression ratio of image token and text token may be 1:1) to make it easier for LLM to do QA-task. However, the number of image tokens of GOT is only 256 and the image compression rate is too high.