Elastic Beanstalk - how to run specific ebextensions according the the environment and Sidekiq Worker

Question

I'll be running a Rails application on Elastic Beanstalk and I'll have both Web and Sidekiq Worker environments. The problem is that, since they both share the same code, I need to run some specific ebextensions on the worker environment (to initialize the worker process and everything) and some specific scripts on the web (to initialize the app server and everything).

1. How do I run specific ebextensions according the the environment on EB or separate the two scripts and tell Elastic Beanstalk to run according to the environment.

2. Should sidekiq worker need to be in the same APP of the web server but on different ENV? for example: my_app env_staging env_staging_worker env_prod env_prod_worker

3. Should Sidekiq worker be deployed as a worker or webapp EB environment? (AWS does not support by default a sidekiq worker).

4. How do you connect the Sidekiq worker to application, for example env_staging to env_staging_worker and env_prod for env_prod_worker?

I would appreciate if you could give details on the architecture or any help you can provide.

Answer 1

If you want to stick to Sidekiq as your worker backend, then you don't need to launch an extra Elastic Beanstalk worker environment. You could start the Sidekiq process within your web environment. Please, have a look at this question and answers from a similar situation.

Concerning the actual idea and architecture behind Elastic Beanstalk's web and worker environment, I'll try to summarize Amazon's official documentation:

A Web environment hosts the HTTP server and web application that responds to the HTTP requests from your users.

A Worker environment hosts an application (can be the same as the web application, but does not have to) which executes background jobs, long running tasks, etc.

The Web and the Worker environment should be connected via an Amazon SQS message queue (an extra AWS feature/service). Your web applications should send messages to this queue, and the worker environment will consume the messages from this queue. When you launch a worker environment, you can choose which SQS queue it should be connected to. Elastic Beanstalk will automatically install and start a daemon which consumes messages from this queue and transform them into HTTP POST requests. These requests are sent to the localhost and a path that you can also define. The request's body will contain the content of the message from the queue. Your application should parse the content and trigger the background job accordingly.

You see, this would make Sidekiq actually obsolete, since the SQS queue will handle the queuing logic.

If you are not constrained to use Sidekiq and your application is written with Rails >= 4.2 then you can use the Active Elastic Job gem for your background tasks. It takes care of all the message sending and parsing, and allows you to keep your deployment setup simple. No need to write and maintain ebextension scripts.