Simulate from a random guess binary classification Model

Question

I needed to simulate data from a random guess binary classifier such that

$p(success) = 0.55$ and $p(failure) = 1-0.55$.

So I simulated the data from a Bernoulli distribution as follows and compared the results with the actual response data.

set.seed(123)
#predicted response
y_pred=replicate(50,rbinom(1,1,prob = 0.55))

#actual response
y=sample(rep(0:1,each=25))

table(y,y_pred)
 

  y_pred
y    0  1
  0 13 12
  1 10 15

Did I do this correctly? Any guidance would be really helpful.

Answer 1

Looks reasonable for me. You don't need replicate, though. And personally I would set a global n:

set.seed(123)
## predicted response
n <- 50
y_pred <- rbinom(n, 1, prob=.55)
## calculate actual probability of predicted response
sum(y_pred) / length(y_pred)
# [1] 0.54

## actual response
y <- sample(rep(0:1, each=n/2))
## calculate actual probability of actual response
sum(y) / length(y)
# [1] 0.5

table(y, y_pred)
#    y_pred
# y    0  1
#   0 13 12
#   1 10 15

However, the actual probability of your predicted response at such small n can have large random fluctuations (i.e. depends on the seed), particularly with small n. Let's put the code into a function for a minute to show that.

n <- 50

sfun <- function() {
  y_pred <- rbinom(50, 1, prob=.55)
  sum(y_pred) / length(y_pred)
}

set.seed(383159)
sfun()
# [1] 0.62  ## 13% off!
set.seed(82809)
sfun()
# [1] 0.44  ## 20% off!

What you could do is to use a repeat loop, that breaks if the result is within a set tolerance. (Note, that this will run forever, when tol is set too small!)

tol <- .01
set.seed(123)
n <- 50
repeat({
  y_pred <- rbinom(n, 1, prob=.55)
  pr1 <- sum(y_pred) / length(y_pred)
  if (pr1 <= .55 + tol & pr1 >= .55 - tol)
    break
  })
y_pred
# [1] 1 0 1 0 0 1 1 0 0 1 0 1 0 0 1 0 1 1 1 0 0 0 0 0 0 0 1 0 1 1 0 0 0 0
# [35] 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 0
sum(y_pred) / length(y_pred)
# [1] 0.54  ## ok!