Can I coordinate rollback of Chef's deploy resource across multiple servers?

Question

I'm moving from Capistrano to Chef to deploy a Rails app (with deploy_revision), and an issue of best practice (or common practice) is unclear to me. I haven't found much through Googling around.

With Capistrano and the "push" model, when deploying an application across a number of servers, it's straightforward to recognize when there has been a deployment failure and to rollback the deployment across all servers at once. Capistrano also puts up a maintenance page on each app server, then doesn't take that maintenance page down unless I've either successfully deployed to all servers or rolled back the deployment.

With Chef and the "pull" model, each server may be fetching its updates at a slightly different time. I might wind up with the database server updating the application code and running database migrations several minutes earlier than the application servers. So I really don't have a great way of identifying a failure and ensuring that the build is rolled back to the last successful deployed version (on all servers).

I know that I have some options here:

1. Don't daemonize chef-client. Run it nightly or hourly with cron. (I still can't roll back across all of my servers in the event that some are successful and just one hits a problem.)
2. Don't daemonize chef-client. Don't use deploy_revision. Run it via knife ssh or capistrano-chef. (I am now back to a "push" model instead of a "pull" model, which is part of the appeal of Chef.)
3. Daemonize chef-client and write custom recipes that talk to each other. (Not enthusiastic about it.)
4. Relax. Don't worry about it. Let the errors happen and then fix them. (Not thrilled here, either.)

I could start building any of these, but before I invest a lot of time in building, I was hoping to learn what has worked for large deployments in the field. What have you done?

Answer 1

Pull is suited to addressing the problem of configuration drift. For deployment on demand (and potential rollback), push is better. Check these out (I haven't used them):

https://github.com/etsy/deployinator

http://www.rackspace.com/blog/rackspace-open-sources-dreadnot/

Answer 2

I've deployed ROR apps via chef's deploy_revision strategy with a bit of success. The simplest thing to do to overcome the timing-control limitations of the pull strategy is to write a deploy script that boils down to:

• Stop chef-client (To ensure the daemon isn't processing things)
• Run chef-client manually (this will also restart the daemon)

or if you're lazy, just run chef-client and don't worry about the daemon.

Then use knife-ssh to execute these commands on the applicable nodes. The end result is that you have all your deploy-related config managed by chef but you can fire off a deploy on an as-needed basis.

For bonus points, package all this up in a Jenkins task. You'll need to configure jenks as a chef client, and make a "Deploy to Staging" task inside your app's project. Now you can click to run the task and your non-chef-speaking teammates can do deploys easily. And for even more bonus points, setup jenkins to auto-deploy after a successful build!