Tensorflow 2.14.0 (with CUDA) not registering CUDA?

Question

I'm trying to use Tensorflow with my GPU. My system is Fedora Linux 38, NVIDIA drivers 535.113.01 (currently latest) working as expected on my system.

I created a Python environment with Python 3.9 (AFAIK version 3.11 won't work to install TF 2.14.0 using pip).

On the activated environment I install TF as follows:

python3.9 -m pip install tensorflow[and-cuda]

I can see that pip installs Tensorflow and many required libraries (cublas, cuda, cull and others).

All seems ok but then when I import tensorflow I get this error:

>>> import tensorflow as tf
2023-10-22 01:58:31.798579: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-10-22 01:58:31.798611: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-10-22 01:58:31.798638: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-10-22 01:58:31.804107: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.

I then used the docker image from Tensorflow: tensorflow/tensorflow:latest-gpu, as a last option, but this showed exactly the same error

Other information

• Python version: version should be >=3.9 and < 3.11 (I tried with many versions in between)

• No local installation of CUDA/CUdNN: I deleted CUDA & CUdNN, used "find / -iname cuda" to make sure there is no trace

• I did reinstalled the drivers and tried version 520 too, same problem

I'm trying to figure out what is going on, when Tensorflow says "...one has already been registered 2023-10-22" for cuDNN, cuBLAS, cuFFT to be able to understand at leat what to look for but I can't find anything useful on the web.

Answer 1

After 72hs of trial and error with no luck, including reinstalling various versions of the nvidia driver, I reinstalled the operating system and followed the process:

• Install drivers (no cuda!)
• Create python env
• Install tensorflow[and-cuda] (using Python 3.9

Now the importing Tensorflow, although it shows the same warning, will use the GPU:

physical_devices = tf.config.experimental.list_physical_devices('GPU')
if len(physical_devices) > 0:
    print("We got a GPU")
    tf.config.experimental.set_memory_growth(physical_devices[0], True)
else:
    print("Sorry, no GPU for you...")

This shows that I finally have my GPU up and running... now I can go and have another 72hs of wrestling with data...

Answer 2

After several days of upgrading and downgrading libraries I found a combination that works for me

• CUDA 12.5
• cudnn 8.9
• Tensor Flow 2.16.1

More specifically I installed CUDA by sudo apt-get install cuda I downloaded and installed the deb files libcudnn8-dev_8.9.7.29-1+cuda12.2_amd64.deb and libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb from https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/ and installed tensorflow by pip install tensorflow==2.16.1 (without [and-cuda])

My OS is Linux Mint 21.1 Cinnamon and my python version 3.10.12