How to properly create, observe and destroy lots of pre-defined jobs at runtime?

Question

I plan to create a web service which will allow the users to spawn a large number, let's say 3k-5k, of certain types of jobs. There'll be a few different, pre-defined types of jobs.

defmoudle Job1 do
  def do_work1(a, b, c) do

  end
end

defmoudle Job2 do
  def do_work2(a, b, c) do

  end
end

# defmoudle Job10 do
# .......

Each one will take on average 5-10, up to 30 minutes to complete, if this is relevant.

It's important that the user should be able to create them at runtime, view the active ones, view the status of each, optionally pause if possible, view the progress if possible, and terminate the jobs.

Should use DynamicSupervisor for this? How would I start a job? As Task.async(..) ... await(..)? As Task.start(..)? Or anything else?

And I probably wouldn't need to make each job a GenServer, right?

---

Without overcomplication. Only by the means of OTP. Oban won't suit me in this case.

Answer 1

I don't know why you don't want to use Oban, but if you must, then

• add max_children: n when you call DynamicSupervisor.init to limit the number of concurrently running jobs.
• add restart: :temporary for your GenServer.
• when the user decides to kill a job, use Process.exit(pid, :kill) to brutely kill the process, otherwise it may not be killed instantly.
• use a registry for naming your GenServer processes so that you can relatively easily find your processes.
• use another GenServer process to monitor the job.
• depending on what you mean by "view the status", you may have to implement some sort of telemetry.
• if you want to pause a running job, then the job should be split into batches. The job should send itself a message after each batch to start the next batch.

Anyway, it's not impossible, but can be pretty hard to implement.