How often shoud a file be opened and closed when constantly writing data to a file (SDCard)?

Question

I am developing a CAN-BUS logger with an ESP32. The data is written to a SDCard with fprintf.

I know I have to use fopen() to open the file and later fclose() to close the file again.

My question is how often should I open and close the file? Open it just one time and then maybe an hour later close it? Or open it, write 100 values, close it, open it again?

I don't want to lose a lot of data. The ESP32 will be on when the motorcycle with the CAN-BUS is running. And if the ignition is turned off then the ESP32 will have no power anymore. I don't mind if maybe data from the last 5 seconds is lost. But I don't want that i.e. 10 minutes of data is lost.

I also saw the fflush() function. Should I use that regularly i.e. every 10 seconds? And then maybe it's no problem if the file is never closed?

More info: I could design the device to make sure the power is on long enough so that fclose() is executed (no power failure before that). But I guess this is not really necessary if I don't mind that the last seconds of data is lost.

I put this question into StackOverflow and not electrical engineering because this is about writing the code for that project.

I searched and found similar questions here but I did not really find an answer to this question.

Answer 1

The Linux manpage for fflush includes a really important warning:

NOTES Note that fflush() flushes only the user-space buffers provided by the C library. To ensure that the data is physically stored on disk the kernel buffers must be flushed too, for example, with sync(2) or fsync(2).

sync or fsync are part of the Posix interface, which might or might not be similar to the underlying file i/o interfaces on your ESP32. But the warning is probably still worth heeding.

The C standard has this wording for fflush, which makes it clear that all that is guaranteed is that fflush flushes out the buffers maintained by the C library, similar to the wording in the Linux manpage:

… the fflush function causes any unwritten data for that stream to be delivered to the host environment to be written to the file…

So if you want data to be actually committed to disk, fflush is not generally sufficient.