Sidekiq process to have a individual log file

Question

Currently I run the sidekiq like

bundle exec sidekiq

It logs the output to log/sidekiq.log. But I want to have individual file for each process it runs.

For example if first process run then sidekiq_1.log should be created.

Answer 1

I assume you are running a Rails project. If you are using the new ActiveJob framework (available in Rails 4.2) you can add a before_perform callback and change the logger before a job is performed. Thus each job will log to its own file.

Example:

class MyJob < ActiveJob::Base
  queue_as :default

  before_enqueue do |job|
    logger = ActiveSupport::Logger.new("#{self.class}-#{job.job_id}.log")
  end

  def perform(*args)
    # Do something later
  end
end

If you want log to different files for a couple of Jobs you can create a base class and inherit from there.

Example:

class BaseJob < ActiveJob::Base
  before_enqueue do |job|
    logger = ActiveSupport::Logger.new("#{self.class}-#{job.job_id}.log")
  end
end

class MyJob < BaseJob
  queue_as :default

  def perform(*args)
    logger.info 'Performing MyJob'
  end
end

class AnotherJob < BaseJob
  queue_as :default

  def perform(*args)
    logger.info { 'Performing AnotherJob' }
  end
end

If you use Sidekiq::Worker you can do the same thing using Server-side middleware which runs 'around' job processing:

class Sidekiq::Middleware::Server::PerJobLogger
  def call(worker, job, queue)
    # Change logger
    Sidekiq.logger = ActiveSupport::Logger.new("#{job[:id]}.log")

    yield
  end
end

and then register the middleware with Sidekiq:

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add Sidekiq::Middleware::Server::PerJobLogger
  end
end

Be careful tho, with the "log per job" approach you might run out of file descriptors, especially if your system processes a lot of jobs.

NOTE: I haven't tested the Sidekiq::Worker solution. Even tho the Logger is thread safe I haven't checked how Sidekiq uses it. But I believe you'll be OK.

Answer 2

Specify a log file for each process then

bundle exec sidekiq --logfile log/sidekiq_1.log
bundle exec sidekiq --logfile log/sidekiq_2.log

Reading help helps

[0] % sidekiq --help
    ...
    -L, --logfile PATH               path to writable logfile
    ...