smol improvements to support more flexible usage

[andimarafioti]

What does this PR do?

This PR improves a few smol issues we had with Idefics 3:

1. We couldn't use the model with images larger than 5*364. This was the default max_image_size. The method where this was computed took a parameter as input, but it was never used. It would also raise an error if we wanted to resize to a larger size. I changed this for a default value of 4k resolution, as this is already considerably larger than what we trained on, ie, anything larger is pretty outrageous.
2. We couldn't train with datasets that contained grayscale images since the input_data_format wasn't properly parsed. I fixed this by switching around the processing order. Now, if the images are grayscale, I add a channel to the end or start of the images. Then, the input_data_format can be correctly inferred if it is none.
3. Finally, when converting to pil_image, we were not passing the input_data_format. For images that have 4 channels, this was breaking the processing. Since we already have the input_data_format in these functions, I added it.

• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

• vision models: @qubvel

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[kashif]

thanks @andimarafioti LGTM, I can load the dataset nicely now

[qubvel]

Hi @andimarafioti! Thanks for the update, just a note regarding converting to RGB

[qubvel]

There is a method that converts PIL images to RGB format (see do_convert_rgb param in other image processors). It's better to use it + extend to numpy arrays if needed.

[andimarafioti]

Yes I know, the order of the conversion here is actually important as RGB conversion before image_splitting was hurting the model performance quite a bit D:

[andimarafioti]

Which is why I'm not converting to RGB until later in the pipeline

[qubvel]

Yes I know, the order of the conversion here is actually important as RGB conversion before image_splitting was hurting the model performance quite a bit D:

haha, that's interesting, do you have any clarification for such a behavior? 😄

[andimarafioti]

My observation is that the order here makes the resulting images are slightly different (if you just plot the differences you see large values on edges in the images). They look the same to me, but clearly not to the model since it was trained with a different pipeline that more closely resembles this.

[qubvel]

Ok, got it, thanks for clarifying 🤗

[mfarre]

@andimarafioti LGTM: I reran vlmevalkit with the current changes and I can confirm there is no regression

[zucchini-nlp]

Agreed that RGB better be done if do_convert_rbg by updating the helper function from utils, otherwise LGTM

[andimarafioti]

To clarify, here I'm not changing where or if RGB conversion happens.

[qubvel]

Yeah, the idea is that we don’t actually need this code because the image will be converted to RGB. However, since maintaining this order is important for quality, it seems fine to me.