unexpected results: microbenchmark

Question

I've always been bugged a bit by the lack of accuracy I see bench marking with system.time and rbenchmark (in that the precision of the timing may be lacking) and saw Hadley reference the microbenchmark package recently. So I decided to give it a whirl as seen below. I pitted mean against f <- function(x) {sum(x)/length(x)} and expected mean to do way better than f but the results, as I understand them, do not indicate this to be true.

1. Am I misunderstanding the results?
2. Is f actually faster than mean?
3. Is microbenchmark still in a beta phase and this needs to be ironed out?

I'm running R2.15 on a win 7 machine (as microbenchmark does timings differently depending on your OS).

The Results

Unit: microseconds
     expr    min     lq median     uq    max
1    f(x) 19.130 20.529 20.529 20.996 286.00
2 mean(x) 28.927 29.860 30.327 30.327 672.31

The Code

library(microbenchmark)

x <- 1:10000
f <- function(x) {sum(x)/length(x)}
mean(x)

res <- microbenchmark(
    mean(x), 
    f(x), 
times=1000L)

print(res)
boxplot(res)

Answer 1

I could be wrong, but this doesn't seem all that surprising to me. Before mean.default can call .Internal(mean(x)) it has to check 3 if statements, calculate the length of x, and then check another if statement. And the difference in times it fairly small.

Calling .Internal(mean(x) directly is slightly faster still:

library(microbenchmark)

x <- 1:10000
f1 <- function(x) {sum(x)/length(x)}
f2 <- function(x) {.Internal(mean(x))}

res <- microbenchmark(
    mean(x), 
    f1(x),
    f2(x), 
times=1000L)

print(res)

Unit: microseconds
     expr    min      lq  median      uq     max
1   f1(x) 32.195 32.4605 32.8850 33.4645 106.997
2   f2(x) 21.840 22.0580 22.2015 22.6270  55.316
3 mean(x) 35.393 35.9840 36.1860 36.4420  91.203

Answer 2

I think you'll find that if you bump up the size of X by a factor of 10 you'll see more consistent results. To be honest I'd be surprised if you really can get micro-second timing accuracy on a computer with a multi-tasking operating system.

You might also consider:

• Are you running on a laptop or a machine which has automatic CPU frequency scaling?
• Warmup?
• Pinning your process to one core.