Could not load library libcublasLt.so.12. Error: libcublasLt.so.12: cannot open shared object file: No such file or directory, Conv2D TensorFlow

Question

I am trying to setup TensorFlow with GPU support in WSL2. I am following this guide.

When I run this code:

>>> from tensorflow import keras
>>> import numpy as np
>>> t = np.ones([5,32,32,3])
>>> c = keras.layers.Conv2D(32, 3, activation="relu")
>>> c(t)

I get this error:

2023-07-09 09:59:38.820408: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:39.031437: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:39.031864: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:39.034068: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:39.034535: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:39.034921: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:40.590457: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:40.590941: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:40.591052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1722] Could not identify NUMA node of platform GPU id 0, defaulting to 0.  Your kernel may not have been built with NUMA support.
2023-07-09 09:59:40.591459: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:982] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-07-09 09:59:40.591526: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3858 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5
Could not load library libcublasLt.so.12. Error: libcublasLt.so.12: cannot open shared object file: No such file or directory
Aborted

The confusing thing is that when I run this code:

>>> from tensorflow import keras
>>> import numpy as np
>>> t = np.ones([5,32,32,3])
>>> c = keras.layers.Dense(32, activation="relu")
>>> c(t)

I get an output and no errors.

• I have tried reinstalling Cuda, CuDNN
• I have tried installing everything in a fresh install of wsl ubuntu 20.04 & 22.04.2
• I have tried Tensorflow 2.10, 2.11, 2.12 and 2.13
• I also tried apt install libcublasLt

Nothing worked

Environment:

• Windows 11 Home
• WSL 2
• Intel i7-9750h
• Nvidia RTX 2060 Notebook
• Tensorflow 2.12.1
• Python 3.9
• WSL2 Ubuntu 20.04
• Cuda 11.8
• CuDNN 8.6

I am also running it in a conda evironment

Answer 1

Like you, I was using TensorFlow with CUDA 11 when I suddenly started to see these errors asking for CUDA 12 libraries.

After searching a little I found that the libcublas-12-0 Ubuntu package does provide all the required files:

/usr/local/cuda-12.0/targets/x86_64-linux/lib/
 libcublas.so.12 -> libcublas.so.12.0.2.224
 libcublasLt.so.12 -> libcublasLt.so.12.0.2.224
 libnvblas.so.12 -> libnvblas.so.12.0.2.224

However, I did not want to mess up my perfectly fine CUDA 11 installation, so I just manually downloaded the package and extracted the required files into the current CUDA directory:

# install CUDA repo: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#network-repo-installation-for-ubuntu
sudo apt update
sudo apt download libcublas-12-0
mkdir contents
dpkg-deb -xv libcublas-12-0_<VERSION>_<ARCH>.deb contents/
sudo mv contents/usr/local/cuda-12.0/targets/x86_64-linux/lib/* /usr/local/cuda/lib64/
rm -rf contents

Still no clue why TensorFlow needs these libraries in the first place ...

Answer 2

Installing the CUDA 12 version of libcublas worked for me, per this discussion.

sudo apt-get install libcublas-12-0