two-stage fine-tuning with LoRa

[tynyanovnik]

Hello,
We want to perform two-stage fine-tuning with LoRa (SFT, then DPO). We have the first LoRA adapter after SFT, and we want to obtain a second adapter after DPO. Is there a way to continue training the same adapter for DPO using swift, without merging the original model weights with the adapter after SFT?

[Jintao-Huang]

When using DPO, directly train with resume_from_checkpoint.