Has anyone figured out a way to run the same cucumber scenario on multiple browsers/web drivers?

Question

I'm using cucumber + capybara for some web automation testing. I'd love to be able to wire up my own tag (something like @all_browsers before the scenario) and have it run against a list of web drivers I set (celerity, selenium on firefox, ie and chrome). I don't want to have to write the scenario 4 different times with 4 different tags out front. I've looked into trying to do this with a new driver I register via:

Capybara.register_driver :all_browsers do |app|
 # What would even work in here? I don't think anything will.
end  

And then following it up with:

Before('@all_browsers') do
 # Same problem here.
end

But I'm not quite sure what to put in that Before method that might actually work.

I've tried using cucumber hooks, specifically:

Around('@all_browsers') do |scenario, block|
  Capybara.current_driver = :selenium_firefox
  block.call

  Capybara.current_driver = :selenium_chrome
  block.call
  # etc
end

But this doesn't behave as I had hoped. It uses the same driver and runs the scenario twice with it.

Following along the hook lines, there's this from the cucumber documentation:
You may also provide an AfterConfiguration hook that will be run after Cucumber has been configured. This hook will run only once; after support has been loaded but before features are loaded. You can use this hook to extend Cucumber, for example you could affect how features are loaded...
This may be a potential path to go down for this, but I've not managed to come up with anything that works here either.

I've looked into custom formatters, but they really only look like they do exactly that - format the output, not so much designate how the features are actually run.

I've looked into overriding cucumber's feature runner, but that doesn't look easy or friendly to do.
Help please? Anyone?

Answer 1

So, I wound up rolling my own solution to this. Not sure if it was the best or most elegant approach, but I actually just wound up:

1. Abstracting all common environment stuff into env.rb
2. Using Cucumber profiles which would require a specific environment file (such as firefox.rb) that required env.rb and then set the default driver for Capybara to the appropriate driver.
3. Wrote a big ol' thor class with tasks that bundle up a bunch of cucumber commands and call out to run the bad boy with the proper profile.
4. Wrote an 'all_browsers' task which bundles up the commands, then calls out to each specific driver task, so I can now have one task that runs any set of scenarios I supply on all the supported drivers.

Working like a charm and I think might have actually wound up better in the end than anything I was trying above, as within the Thor file I was able to add things like a benchmarking option, as well as whether or not to split the feature run up into multiple threads. Still curious if anyone else came up with a solution for this though.

cucumber.yaml:
Here, the all_features file just does a glob of everything ending in .feature, because if I pulled in the entire features directory it would pull in everything beneath it, including all the profile files, etc, which isn't what I wanted since each profile file sets the default capybara driver to a different value. Once you specify -r as an option to cucumber, all autoloading of any file is halted.

default: --format pretty

chrome: --format pretty -r features/support/profiles/chrome.rb -r features/all_features -r features/step_definitions

firefox: --format pretty -r features/support/profiles/firefox.rb -r features/all_features -r features/step_definitions

celerity: --format pretty -r features/support/profiles/celerity.rb -r features/all_features -r features/step_definitions

firefox.rb (the 'profile' file):

require File.dirname(__FILE__) + "/../env.rb"

Capybara.configure do |config|
  config.default_driver = :selenium_firefox
end

selenium_firefox.rb (where I register the driver, and set some tag capability which I've wound up not needing now, as the @selenium_firefox tag was part of my original attempt at this posted in the question):

# Register a specific selenium driver for firefox
Capybara.register_driver :selenium_firefox do |app|
  Capybara::Driver::Selenium.new(app, :browser => :firefox)
end

# Allows the use of a tag @selenium_firefox before a scenario to run it in selenium with firefox
Before('@selenium_firefox') do
  Capybara.current_driver = :selenium_firefox
end

feature_runner.thor:

require 'benchmark'

class FeatureRunner < Thor
  APP_ROOT = File.expand_path(File.dirname(__FILE__) + "/../")

  # One place to keep all the common feature runner options, since every runner in here uses them.
  # Modify here, and all runners below will reflect the changes, as they all call this proc.
  feature_runner_options = lambda { 
    method_option :verbose, :type => :boolean, :default => true, :aliases => "-v"
    method_option :tags, :type => :string
    method_option :formatter, :type => :string
    method_option :other_cucumber_args, :type => :string
  }


  desc "all_drivers_runner", "Run features in all available browsers"
  method_option :benchmark, :type => :boolean, :default => false
  method_option :threaded, :type => :boolean, :default => true
  feature_runner_options.call # Set up common feature runner options defined above
  def all_drivers_runner
    if options[:threaded]
      feature_run = lambda { 
        thread_pool = []

        t = Thread.new do |n|
          invoke :firefox_runner
        end
        thread_pool << t

        t = Thread.new do |n|
          invoke :chrome_runner
        end
        thread_pool << t

        t = Thread.new do |n|
          invoke :celerity_runner
        end
        thread_pool << t

        thread_pool.each {|th| th.join}
      }
    else
      feature_run = lambda { 
        invoke "feature_runner:firefox_runner", options
        invoke "feature_runner:chrome_runner", options
        invoke "feature_runner:celerity_runner", options
      }
    end

    if options[:benchmark]
      puts "Benchmarking feature run"
      measure = Benchmark.measure { feature_run.call }
      puts "Benchmark Results (in seconds):"
      puts "CPU Time: #{measure.utime}"
      puts "System CPU TIME: #{measure.stime}"
      puts "Elasped Real Time: #{measure.real}"
    else
      feature_run.call
    end
  end

  desc "firefox_runner", "Run features on firefox"
  feature_runner_options.call # Set up common feature runner options defined above
  def firefox_runner
    command = build_cucumber_command("firefox", options)
    run_command(command, options[:verbose])
  end

  desc "chrome_runner", "Run features on chrome"
  feature_runner_options.call # Set up common feature runner options defined above
  def chrome_runner
    command = build_cucumber_command("chrome", options)
    run_command(command, options[:verbose])
  end

  desc "celerity_runner", "Run features on celerity"
  feature_runner_options.call # Set up common feature runner options defined above
  def celerity_runner
    command = build_cucumber_command("celerity", options)
    run_command(command, options[:verbose])
  end

  private
  def build_cucumber_command(profile, options)
    command = "cd #{APP_ROOT} && ./bin/cucumber -p #{profile}"
    command += " --tags=#{options[:tags]}" if options[:tags]
    command += " --formatter=#{options[:formatter]}" if options[:formatter]
    command += " #{options[:other_cucumber_args]}" if options[:other_cucumber_args]
    command
  end

  def run_command(command, verbose)
    puts "Running: #{command}" if verbose
    output = `#{command}`
    puts output if verbose
  end

end

Where everything wound up, in relation to the root directory:

.
|____cucumber.yml
|____features
| |____all_features.rb
| |____google_search.feature
| |____step_definitions
| | |____google_steps.rb
| | |____web_steps.rb
| |____support
| | |____custom_formatters
| | | |____blah.rb
| | |____env.rb
| | |____paths.rb
| | |____profiles
| | | |____celerity.rb
| | | |____chrome.rb
| | | |____firefox.rb
| | |____selenium_drivers
| | | |____selenium_chrome.rb
| | | |____selenium_firefox.rb
| | | |____selenium_ie.rb
| | | |____selenium_remote.rb
| | |____selenium_drivers.rb
|____tasks
| |____feature_runner.thor
| |____server_task.rb  

Output of thor -T

feature_runner
--------------
thor feature_runner:all_drivers_runner  # Run features in all available browsers
thor feature_runner:celerity_runner     # Run features on celerity
thor feature_runner:chrome_runner       # Run features on chrome
thor feature_runner:firefox_runner      # Run features on firefox  

Now I can run something like:
thor feature_runner:all_drivers_runner --benchmark
This would run all features on all capybara drivers in a thread for each driver, benchmnarking the results.

Or
thor feature_runner:celerity_runner
This would run all features only on celerity.

But I can now also supply some other options to the thor command which get passed onto cucumber such as:
--tags=@all_browsers
--formatter=hotpants
--other_cucumber_args="--dry-run --guess --etc"

What a feature file can now look like:

Feature: Start up browser
  @all_browsers
  Scenario: Search Google
   Given I am on the home page
   When I fill in the search bar with "Capybara"
   And I press "Search"
   Then I should see "Capybara"

Seems like a lot of setup, but now if I tag a feature with @all_browsers, I can build out a suite to test against all capybara drivers, in a multi-threaded environment, with one thor command:
thor feature_runner:all_drivers_runner --threaded --tags=@all_browsers

Or build out a smoke test suite that runs in celerity:
thor feature_runner:celerity_runner --tags=@smoke_test