Erlang: designing a service component that's immutable but slow to start

Question

I am developing a web service with Erlang. Low latency is a goal. The service provides several views of a set of files. Building these views takes time, since I have to read in the files, parse them as CSV, store their fields in records, etc.

The naive approach would be to re-read these files and do the necessary transformations for every request, so the web application is entirely stateless. But I am concerned about latency.

Another approach would be to create a server (implementing the gen_server behavior) that prepares these views at startup and stores them in-memory as LoopData, then returns the views as needed to the web workers. But this reduces concurrency, since this server processes one request at a time.

Is there a design pattern in Erlang that supports doing some expensive initialization at startup, and yet allows concurrent access to the initialized data? (The key being that this data is immutable, so I am not concerned about mutual exclusion.)

Answer 1

You can use one or more ETS tables to hold your data.

Populate tables in your initialization phase, then have every process read from it.

You can abstract initialization and reading from tables into a modules, just to make sure you can switch the implementation later (like using mnesia or something else later).

Note that ETS tables are in memory (by default), and if you need more complicated queries, there is mnesia.