CUDA Out of memory when there is plenty available

Question

I'm having trouble with using Pytorch and CUDA. Sometimes it works fine, other times it tells me RuntimeError: CUDA out of memory. However, I am confused because checking nvidia-smi shows that the used memory of my card is 563MiB / 6144 MiB, which should in theory leave over 5GiB available.

However, upon running my program, I am greeted with the message: RuntimeError: CUDA out of memory. Tried to allocate 578.00 MiB (GPU 0; 5.81 GiB total capacity; 670.69 MiB already allocated; 624.31 MiB free; 898.00 MiB reserved in total by PyTorch)

It looks like Pytorch is reserving 1GiB, knows that ~700MiB are allocated, and is trying to assign ~600MiB to the program—but claims that the GPU is out of memory. How can this be? There should be plenty of GPU memory left given these numbers.

Answer 1

You need empty torch cache after some method(before error)

torch.cuda.empty_cache()