[PPO] feat: Add LoRA support for PPO

[StephenXie]

This PR adds LoRA (Low-Rank Adaptation) support for PPO (#159)

Changes

• Added LoRA support to actor and critic configuration (see [SFT] feat: Add LoRA support for SFT #127)
• Merge the PEFT adapter before serving the model with vLLM and unmerge afterward.

Features

• Configurable LoRA rank and alpha parameters
• Target module specification for selective adaptation
• Compatible with FSDP sharding strategy

Some known issues:

• Merge Ref and Actor when LoRA is on requires modifying ppo_trainer logic, we need some help
• No thorough testing yet
• Line 80 of fsdp_vllm.py needs to be cleaned up
  params = OrderedDict((k.replace(".base_layer.", "."), v) for k, v in params.items() if not ".lora_" in k)

[Jiayi-Pan]

Relevant thread

#159

Jiayi-Pan/TinyZero#15

[vermouth1992]

Summon full params may cause OOM. @PeterSH6 Is there a better approach, that can merge lora weights in sharded form, or at least, one parameter after another to support large models?

[Shirley-Kokane]

The merge_adapter is not releasing the same original model structure as before. Is there any other way to merge and get the original base model structure?