Getting the index of an iterator in R (in parallel with foreach)

Question

I'm using the foreach function to iterate over columns of a data.frame. At each iteration, I would like to get the index of the iterator (i.e. the index or the name of the column considered) and the column itself.

However, the following code, which seems fine in first place, doesn't work because i has no names or colnames attributes.

 foreach(i=iter(base[1:N],by='col')) %dopar% c(colnames(i),i)

Now, if you wonder why I'm not iterating over indexes, the reason is that I'm using the %dopar% tool and I don't want to send the whole base to all workers, but only the columns each of them require.

Question : How can I get the index of an iterator ?

Thank you

Answer 1

I would just specify a second iteration variable in the foreach loop that acts as a counter:

library(foreach)
library(iterators)

df <- data.frame(a=1:10, b=rnorm(10), c=runif(10))
r <- foreach(d=df, i=icount()) %do% {
    list(d=d, i=i)
}

The "icount" function from the iterators package will return an unbounded counting iterator if no arguments are used, so this example works regardless of the number of columns in the data frame.

You could also include the column name as a third iteration variable:

r <- foreach(d=df, i=icount(), nm=colnames(df)) %do% {
    list(d=d, i=i, nm=nm)
}

Answer 2

Here are a couple of possibilities:

Modify the iter function (or write your own) so that instead of sending just the value of the column it includes the names or other information)

You could iterate over the indexes, but use a shared memory tool (such as the Rdsm package) so that each process only needs to grab the part of the data frame it needs rather than distributing the entire data frame.

You could convert your base data frame into a list where each element contains the corresponding column of base along with the column name, then iterate over that list (so the entire element is sent, but not the other elements).