DDP for Finetuning

[psinger-prior]

Describe the workflow you want to enable

Finetuning currently does not support distributed training

Describe your proposed solution

Implement DDP for distributed finetuning

Describe alternatives you've considered, if relevant

No response

Additional context

No response

Impact

None