Very low GPU utility while training #2
Description
Dear team,
I am running into very low gpu utility while training with ADE20 which I think is slowing down the training speed by a lot.
I believe it is caused by: https://github.com/neuralcodinglab/viseon/blob/d84a31eb52ef8743a8f487997fcc87e964f9027c/local_datasets.py#L109C16-L109C69
In the dataset class.
return x.detach().to(self.device),t.detach().to(self.device)
This should be done in the getitem function but rather after loading a full batch with CPU (with pytorch dataloader, which supports multithread loading by setting num_works = num_cpu_cores), then we send the batch of data to the GPU in the training loop.
Thanks for this amazing end-to-end visual neuroprosthetics simulator btw!