Testing - nomenclature

Question

Classes are often tested by using tests written in the following sort of syntax, which is provided by a large set of testing frameworks (e.g. Ruby's Unit::Test; or as in this example, MiniTest):

class TestLolcat < MiniTest::Unit::TestCase
  def setup
    @lolcat = Lolcat.new
  end   
  def test_that_lolcat_can_have_cheezburger
    assert_equal true, @lolcat.i_can_has_cheezburger?
  end
end

or instead by using tests written in a second sort of syntax, which is provided by an overlapping set of testing frameworks (e.g. RSpec; or as in this example, MiniTest again):

describe Lolcat do
  before do
    @lolcat = Lolcat.new
  end
  it "must be able to have cheezburger" do
    @lolcat.i_can_has_cheezburger?.must_equal true
  end
end

(There are probably other syntax families which are also suitable for testing class methods and attributes, but these are the two I'm interested in.)

What I want to know is: what is the correct name for each of these two syntax families?

If you want to know more about why I'm asking, see below the line.

---

The reason I'm asking this question is that searching the Web has not yielded an obvious consensus. For instance, the MiniTest documentation refers to the first syntax above as "unit test" syntax and the second as "spec" syntax. By contrast, Michael Hartl describes class method and attribute tests written in the second sort of syntax as "unit tests". Where tests written in this syntax test higher-level functionality that results from the interaction of multiple classes, he calls them "integration tests".

I've also seen people describe the first and second sorts of syntax as "Test::Unit style" and "RSPec-esque", respectively, or "TDD" syntax and "BDD" syntax. Other names for the second sort of syntax include "'should'-like syntax" and "it syntax".

My understanding (which vaguely suggests Hartl is correct) is as follows:

• TDD is a practice, not a testing syntax. Specifically, it's the practice of typically writing a failing test (e.g. a unit test or an integration test), then code to make the test pass, then refactoring if appropriate, and then restarting the cycle.
• BDD is also a practice, but it does make some limited prescriptions about testing syntax. Specifically, it's the practice of following TDD but using "should" as the first word in test names, and nesting testing code to provide context if appropriate.
• Unit testing is a practice, not a testing syntax. Specifically the practice of testing individual units of code (e.g. class methods or attributes)
• Unit tests are tests written in the course of unit testing, regardless of syntax employed.
• Integration testing is a practice, not a testing syntax. Specifically the practice of testing higher-level functionality that results from the interaction of multiple classes.

However, this still doesn't entirely clear things up. Evidently, I'm no expert in the nomenclature of testing practices, test types or test syntaxes. The best names I've been able to come up with for the two types of syntax I've given examples of above are, "'assert' syntax" and "'it ... do' syntax", respectively. Unless these names are in widespread use, I need your advice, fellow StackOverflow users!

Answer 1

I'm not sure there is an accepted definition for each.

I like to think of them in these ways.

the mini-test version is more of an assertation

the rspec version is more of a specification

The latter was concocted to steer you away from the emphasis on 'test' and more to driveout the requirements from a stakeholder point of view.

Answer 2

It might clarify things if we shared a bit of history about BDD and TDD.

Back in 2003, the unit testing framework of choice for TDD was JUnit. Dan North started writing JBehave as a replacement for JUnit, using "should" instead of "test" - so, BDD syntax. Now that doesn't look very similar to what you're doing (and what RSpec does) in Ruby, but nonetheless, RSpec was created as a Ruby version of the same thing. JBehave never nested testing code; it just had "should" for unit-level examples and then some other gubbins for running full-system scenarios. Dan migrated the scenario runner to Ruby, where it became the RSpec story runner, then eventually Cucumber.

So, the "it... should" syntax is definitely unit-level BDD. Most people are familiar with BDD as full-system scenarios, but it isn't where BDD started. Nowadays BDD isn't really one practice; it's a mini-methodology that goes right up to project visioning - just in case you come across anything confusing like that. And JBehave 2.0 was rewritten to work with scenarios and steps, and lost the unit-level testing along with a mocking framework, because JUnit and Mockito worked well by then. But, nonetheless, that's where "should" started.

Now... as soon as you use "must" instead of "should", you lose the sense of uncertainty that "should" provides. Using "should" allows you to ask lots of questions, the most important of which is "should it?" So, anything which uses "must" instead of "should" isn't really BDD syntax any more; it's a hybrid.

Incidentally, we try to avoid the word "test" when we talk about BDD, so they're not unit tests written in the course of testing; they're examples of how a class will behave, written in the course of designing the code.