Raise UserWarning in RewardTraining if PEFT target_modules="all-linear"

[JohnGiorgi]

What does this PR do?

I ran into a nasty edge case that I don't think is documented anywhere when using the RewardTrainer to train a reward model. If a peft_config is provided and target_modules="all-linear" (a wild card that means: adapt all linear layers except for the output) then the output layer of the reward model, which is often newly initialized and is used to score the chosen/rejected completions, will go un-adapted (and therefore un-trained). Performance will be impacted as you might expect, e.g. here's two runs of mine with target_modules="all-linear" and target_modules=None (the default):

This PR simply raises a UserWarning in RewardModel in this case. You could almost argue that raising an error would be warranted, but I wonder if there is some scenario in which the output layer is already trained, and a user wants to just adapt some intermediate layers with LoRA.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[JohnGiorgi]

I am handling both cases of peft_config being dict and being a PeftConfigobject.

It is type-hinted as a dict in the RewardModel docstring, but I can see a PeftConfig object is passed to RewardTrainer here.

[JohnGiorgi]

Closing because I wasn't able to reproduce this on closer inspection... I can also see model.score.parameters() is updated after training even with target_modules=all-linear...