TensorRT - TensorFlow deserialization fails with Serialization Error in verifyHeader

Question

I'm running the nvcr.io/nvidia/tensorflow:19.12-tf2-py3 docker image with the following runtime information:

• Tensorflow 2.0.0 (tf.__version__)
• Python 3.6 (!python --version)
• TensorRT 6.0.1 (!dpkg -l | grep nvinfer)
• cuda 10.2

I have built a model in TensorFlow 2.0 and converted+saved it to a dir:

1/
├── assets/
|   └── trt-serialized-engine.TRTEngineOp_0
├── variables/
|   ├── variables.data-00000-of-00002
|   ├── variables.data-00001-of-00002
|   └── variables.index
└── saved_model.pb

Now when I try deserializing the cuda engine with the TensorRT python API:

import tensorrt as trt
TRT_LOGGER = trt.Logger(trt.Logger.VERBOSE)

serialized_engine = './tmp/unet-FP32/1/assets/trt-serialized-engine.TRTEngineOp_0'
# serialized_engine = './tmp/unet-FP32/1/saved_model.pb'

trt_runtime = trt.Runtime(TRT_LOGGER)

with open(serialized_engine, 'rb') as f:
    engine = trt_runtime.deserialize_cuda_engine(f.read())

I receive the following error messages:

[TensorRT] ERROR: ../rtSafe/coreReadArchive.cpp (31) - Serialization Error in verifyHeader: 0 (Magic tag does not match)
[TensorRT] ERROR: INVALID_STATE: std::exception
[TensorRT] ERROR: INVALID_CONFIG: Deserialize the cuda engine failed.

I do the saving and loading on the exact same machine, inside the exact same docker container. Am I wrong to assume that 'trt-serialized-engine.TRTEngineOp_0' contains the actual serialized model?

I have also tried doing it with the uff-parserm, but the uff shipped in the NVidia container is incompatible with tensorflow 2.0.

Any ideas how to deserialize my trt engine?

Answer 1

If the engine was created and ran on different versions, this may happen. TensorRT engines are not compatible across different TensorRT versions. Go to this reference for more info.