About FT Qwen2.5

[lyons66]

Hi, I noticed that in the qwen2_vl.py, we only have Qwen2VLForConditionalGeneration from transformers. However, for Qwen2.5, it requires the dev version of transformers (4.49.0) to import Qwen2_5_VLForConditionalGeneration.

So, how could I apply the fine-tuning for Qwen2.5-7b?

[zjysteven]

Tagging Yuqian @lavinal712 (who contributed the integration of Qwen2.5) for help.

[lavinal712]

I'm currently trying to modify the version of Transformers, but I've encountered a tensor dimension mismatch issue, which might be caused by the latest code in Transformers. I will fix this issue as soon as possible.

[lavinal712]

The upgrade of the Transformers library has made it impossible to fine-tune many models. The current code temporarily does not fully support the fine-tuning of Qwen2.5-VL. We apologize for the inconvenience.

[zjysteven]

@lyons66 The new branch of this repo transformers-4.49.0.dev0 should support finetuning of Qwen2.5-VL. Please have a try and let us know if there are any problems.

[lyons66]

@zjysteven Hi, I have successfully launched the ft for Qwen2.5-vl-7B. Noticing that after upgrading to 4.49.0.dev0 there are two things that need updating:

1. in train.py, the import of deepspeed should be: from transformers.integrations import deepspeed
2. in the loaders/qwen2_vl.py, I need to change this:
   Qwen2VLForConditionalGeneration -> Qwen2_5_VLForConditionalGeneration

But I still got an image features token size mismatch problem:
[rank0]: File "/home/liuliu/miniconda3/envs/lmms-finetune/lib/python3.10/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1743, in forward
[rank0]: raise ValueError(
[rank0]: ValueError: Image features and image tokens do not match: tokens: 1009, features 1369

This issue wasn't there when fine-tuning Qwen2 on the same datasets. My training sets have different image sizes, but the patching codes previously worked well. I don't know why this Qwen 2.5 became sensitive then.

[zjysteven]

There's actually a little bit more than 1, 2 that needed updates, but we've got all of them covered under the branch transformers-4.49.0.dev0 of this repo. Just to confirm, were you using that branch?

If not, would you try again under that branch? Also this time please use our example_scripts/example_image.sh (with MODEL_ID changed from llava-1.5-7b to qwen2.5-vl-3b-instruct). On my end this example training using celeba images runs well. Knowing whether it runs successfully for you can help us better isolate the issue.

[lyons66]

Thanks for your reply, and I didn't notice that branch.
However, after checking out the transformers-4.49.0.dev0 branch, I still got the same problem while processing my images

[rank0]: return inner()
[rank0]: File "/home/liuliu/miniconda3/envs/lmms-finetune/lib/python3.10/site-packages/torch/nn/modules/module.
py", line 1793, in inner
[rank0]: result = forward_call(*args, **kwargs)
[rank0]: File "/home/liuliu/miniconda3/envs/lmms-finetune/lib/python3.10/site-packages/peft/tuners/tuners_utils
.py", line 197, in forward
[rank0]: return self.model.forward(*args, **kwargs)
[rank0]: File "/home/liuliu/miniconda3/envs/lmms-finetune/lib/python3.10/site-packages/transformers/models/qwen
2_5_vl/modeling_qwen2_5_vl.py", line 1743, in forward
[rank0]: raise ValueError(
[rank0]: ValueError: Image features and image tokens do not match: tokens: 1009, features 1369

---

I also tried your example

and got the model well fine-tuned:

but when I switched to my own datasets(which has been applied to the fine-tuning in LLaVA1.6 and Qwen2, I still got the error above.

[zjysteven]

What is the MODEL_MAX_LEN you are using, 1024? In this case it seems quite likely that the image tokens are truncated due to insufficient MODEL_MAX_LEN, and as a result there is that mismatch. Try increasing MODEL_MAX_LEN to 2048 (or even larger depending on your data).

[lyons66]

It works. Thanks a lot for pointing out that!