RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:6!

[Yongxv]

I'm having this problem running 05_dpo, does anyone know how to fix it?

[gschoeni]

It looks like maybe you have multiple GPUs that it is trying to transfer data between? Does changing device_map to “cuda:0” instead of “auto” fix it?

[Yongxv]

It looks like maybe you have multiple GPUs that it is trying to transfer data between? Does changing device_map to “cuda:0” instead of “auto” fix it?

I don't think this will work since this is a machine with 8 rtx3090.Then I tried it, got out of memory：torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.65 GiB. GPU 0 has a total capacty of 23.70 GiB of which 2.62 GiB is free. Including non-PyTorch memory, this process has 21.08 GiB memory in use. Of the allocated memory 19.69 GiB is allocated by PyTorch, and 165.99 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Maybe i still need to use multiple GPUs, so how to modify it?

[Yongxv]

It looks like maybe you have multiple GPUs that it is trying to transfer data between? Does changing device_map to “cuda:0” instead of “auto” fix it?

I don't think this will work since this is a machine with 8 rtx3090.Then I tried it, got out of memory：torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.65 GiB. GPU 0 has a total capacty of 23.70 GiB of which 2.62 GiB is free. Including non-PyTorch memory, this process has 21.08 GiB memory in use. Of the allocated memory 19.69 GiB is allocated by PyTorch, and 165.99 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF Maybe i still need to use multiple GPUs, so how to modify it?

To be precise, when using a single GPU, then out of memory

[gschoeni]

Ah I see, so we'd need to update the training code for multiple devices. I don't have a machine with that configuration to test on but feel free to make a PR if you figure it out!