Why does my HUnit test suite fail but pass successfully in Cabal?

Question

If I have test/Test.hs with

module Main where

import Test.HUnit

test1 :: Test
test1 = TestCase $ assertEqual "Should be one" 1 5

test2 :: Test
test2 = TestCase $ assertEqual "Shold both be zero" 0 0

main :: IO Counts
main = runTestTT $ TestList [test1, test2, test1]

and a .cabal with

test-suite my-test
    type:               exitcode-stdio-1.0
    hs-source-dirs:     test
    main-is:            Test.hs
    build-depends:      base >= 4.8.1.0 && <4.9,
                        HUnit >= 1.3
    default-language:   Haskell2010

and I run cabal test --show-details='always' then I get

Test suite my-test: RUNNING...
### Failure in: 0
test/Test.hs:6
Should be one
expected: 1
 but got: 5
### Failure in: 2
test/Test.hs:6
Should be one
expected: 1
 but got: 5
Cases: 3  Tried: 3  Errors: 0  Failures: 2
Test suite my-test: PASS

Why does my test suite pass when I've had failures? Likewise, if I cabal sdist I get no warning that my tests have failed.

Answer 1

According to the Cabal users' guide,

Test suites using the exitcode-stdio-1.0 interface are executables that indicate test failure with a non-zero exit code when run; they may provide human-readable log information through the standard output and error channels.

You've defined

main :: IO Counts
main = runTestTT $ TestList [test1, test2, test1]

This runs tests, prints out test information, and then always exits successfully. If you want Cabal to know the test has failed, you need to capture the Counts, check for errors and failures, and exit with a non-zero status if you find such.

import System.Exit

main :: IO ()
main = do
  results <- runTestTT $ TestList [test1, test2, test1]
  if (errors results + failures results == 0)
    then
      exitWith ExitSuccess
    else
      exitWith (ExitFailure 1)

The test-framework package provides convenient defaultMain functions that do this sort of thing; you may want to consider that approach.

You should note that the exitcode-stdio-1.0 interface is considered semi-deprecated; the Cabal maintainers recommend switching to their rather more Haskellian detailed-0.9 interface.