Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Internvl2TemplateWithAngles template #1784

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

DYF-AI
Copy link

@DYF-AI DYF-AI commented Aug 21, 2024

PR type

  • More Models or Datasets Support

PR information

Template A is added to support entering image angles,and the dataset format is as follows:

{"query": "please describe the image.", "response": "An old man combing the hair of a young boy.", "images": ["/mnt/g/dongyongfei786/custom-swift/examples/data_processing/output/train/235812.jpg"], "angles": [0]}

add custom templates:
--template internvl2-angle

@Jintao-Huang
Copy link
Collaborator

Thank you very much for your PR. Could you please describe the reason for submitting it?😊

@DYF-AI
Copy link
Author

DYF-AI commented Aug 23, 2024

Thank you very much for your PR. Could you please describe the reason for submitting it?😊

Hello, I'm glad to receive your reply. The reason for this PR is that when extracting multimodal image information, the usual practice is to classify the images into angles (such as 0, 90, 180, 270, etc.), and then input the corrected angle images into the model. Based on our experience, this approach works better with a small number of samples for fine-tuning training. The original data format did not support the input of image angles, so we are submitting this PR.         

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants