Writing to two different files with HDF5

Question

I've a small library in C that makes use of HDF5 to write data (v. 1.8.14) under Windows. That lib is then used by a C# app that does some other stuff and then needs to write quite a lot of data.

I now need to launch two instances of the application with the idea of having each instance saving to two distinct files placed on two different hard disks. However, I've performance issues and data loss. By inspecting the use of disks, it seems that data is written serially (first HD is busy while the second is not, then the second becomes busy and the first not and so on), which is as having a single disk (and a single disk is not fast enough for twice the data).

So what shall I do to write to two different files from two different processes? Do I need to use Parallel HDF5? And the solution you are going to propose will also work if I later want to write to two different files from within the same process? Please provide details and links to relevant resources

Answer 1

The HDF5 global lock is not shared across separate processes on a single machine. Each process that opens an HDF5 file has its own instance of the HDF5 library and thus maintains its own separate global lock. This means that the lock only applies within a single process and does not affect other processes accessing the same file.

So did you mean actual processes on the same OS or hardware machine? Or did you mean processes in shared memory such as multiple threads?

• Could you possibly have a typo and are hitting the same file on disk?
• Could it be that your data acquisition process is the bottle neck and is effectively serializing the writes to the two files?

You mention later needing to write to two files in the same process, which is completely different that the first specification.