Appending data to the same dataset in hdf5 in matlab

Question

I have to put all the huge data together into a single dataset in hdf5. Now, the thing is, if you try:

>> hdf5write('hd', '/dataset1', [1;2;3])
>> hdf5write('hd', '/dataset1', [4;5;6], 'WriteMode', 'append')
??? Error using ==> hdf5writec
writeH5Dset: Dataset names must be unique when appending data.

As you can see, hdf5write will complain when you tried to append data to the same dataset. I've looked around and see one possible workaround is to grab your data from the dataset first, then concatenate the data right in matlab environment. This is not a problem for small data, of course. For this case, we are talking about gigabytes of data, and Matlab starts yelling out out of memory.

Because of this, what are my available options in this case?

Note: we do not have h5write function in our matlab version.

Answer 1

I believe the 'append' mode is to add datasets to an existing file.

hdf5write does not appear to support appending to existing datasets. Without the newer h5write function, your best bet would be to write a small utility with the low-level HDF5 library functions that are exposed with the H5* package functions.

To get you started, the doc page has an example on how to append to a datatset.

Answer 2

You cannot do it with hdf5write, however if your version of Matlab is not too old, you can do it with h5create and h5write. This example is drawn from the doc of h5write:

Append data to an unlimited data set.

h5create('myfile.h5','/DS3',[20 Inf],'ChunkSize',[5 5]);
for j = 1:10
    data = j*ones(20,1);
    start = [1 j];
    count = [20 1];
    h5write('myfile.h5','/DS3',data,start,count);
end
h5disp('myfile.h5');

For older versions of Matlab, it should be possible to do it using the Matlab's HDF5 low level API.