How do I run apply on a data.table?

Question

I have a data.table with columns 2 through 20 as strings with spaces (e.g., "Species Name"). I want to run str_replace() on all those columns simultaneously so all the "Species Name" become "Species_Name". I can either do:

data.table(apply(as.data.frame(dt[,2:dim(dt)[2], with=F]), 2,                                 function(x){ str_replace(x," ","_") })) 

or if I keep it as a data.table object, then I can do this one column at a time:

dt[,SpeciesName := str_replace(SpeciesName, " ", "_") 

How do I do this for all columns 2 through the end similar to the one of the above?

Answer 1

Completely rewritten on 2015-11-24, to fix an error in previous versions.

Also added more modern options on 2019-09-27

You have a few options.

1. Process all of the target columns with an embedded call to lapply(), using := to assign the modified values in place. This relies on :='s very handy support for simultaneous assignment to several column named on its LHS.

2. Use a for loop to run through the target columns one at a time, using set() to modify the value of each one in turn.

3. Use a for loop to iterate over multiple "naive" calls to [.data.table(), each one of which modifies a single column.

These methods all seem about equally fast, so which one you use will be mostly a matter of taste. (1) is nicely compact and expressive. It's what I most often use, though you may find (2) easier to read. Because they process and modify the columns one at a time, (2) or (3) will have an advantage in the rare situation in which your data.table is so large that you are in danger of running up against limits imposed by your R session's available memory.

library(data.table)  ## Create three identical 1000000-by-20 data.tables DT1 <- data.table(1:1e6,            as.data.table(replicate(1e6, paste(sample(letters, nr, TRUE),                                              sample(letters, nr, TRUE))))) cnames <- c("ID", paste0("X", 1:19)) setnames(DT1, cnames) DT2 <- copy(DT1); DT3 <- copy(DT1)  ## Method 1 system.time({ DT1[, .SDcols=cnames[-1L], cnames[-1L] :=    lapply(.SD, function(x) gsub(" ", "_", x, fixed=TRUE)), ] }) ##   user  system elapsed  ##  10.90    0.11   11.06   ## Method 2 system.time({     for(cname in cnames[-1]) {         set(DT2, j=cname, value=gsub(" ", "_", DT2[[cname]], fixed=TRUE))     } }) ##   user  system elapsed  ##  10.65    0.05   10.70   ## Method 3 system.time({     for(cname in cnames[-1]) {         DT3[ , (cname) := gsub(" ", "_", get(cname), fixed=TRUE)]     } }) ##   user  system elapsed  ##  10.33    0.03   10.37  

For more details on set() and :=, read their help page, gotten by typing ?set or ?":=".