Using 'fastmatch' package in R

Question

I have to find indices for 1MM numeric values within a vector of roughly 10MM values. I found the package fastmatch, but when I use the function fmatch(), I am only returning the index of the first match.

Can someone help me use this function to find all values, not just the first? I realize this is a basic question but online documentation is pretty sparse and fmatch has cut down the computing time considerably.

Thanks so much!

---

Here is some sample data - for the purposes of this exercise, let's call this data frame A:

              DateTime     Address       Type     ID
1  2014-03-04 20:21:03   982076970          1  2752394
2  2014-03-04 20:21:07 98174238211          1  2752394
3  2014-03-04 20:21:08 76126162197          1  2752394
4  2014-03-04 20:21:16  6718053253          1  2752394
5  2014-03-04 20:21:17 98210219176          1  2752510
6  2014-03-04 20:21:20  7622877100          1  2752510
7  2014-03-04 20:21:23  2425126157          1  2752510
8  2014-03-04 20:21:23  2425126157          1  2752510
9  2014-03-04 20:21:25   701838650          1  2752394
10 2014-03-04 20:21:27 98210219176          1  2752394

What I wish to do is to find the number of unique Type values for each Address. There are several million rows of data with roughly 1MM unique Address values... on average, each Address appears about 6 times in the data set. And, though the Type values listed above are all 1, they can take any value from 0:5. I also realize the Address values are quite long, which adds to the time required for the matching.

I have tried the following:

uvals <- unique(A$Address)
utypes <- matrix(0,length(uvals),2)
utypes[,1] <- uvals

for (i in 1:length(unique(Address))) {
    b <- which(uvals[i] %in% A$Address)
    c <- length(unique(A$Type[b]))
    utypes[i,2] <- c
}

However, the code above is not very efficient - if I am looping over 1MM values, I estimate this will take 10-15 hours.

I have tried this, as well, within the loop... but it is not considerably faster.

b <- which(A$Address == uvals[i])  

I know there is a more elegant/faster way, I am fairly new to R and would appreciate any help.

Answer 1

This can be done using unique function in data.table, followed by an aggregation. I'll illustrate it using more or less the sample data generated by @Chinmay:

Create sample data:

set.seed(100L)
dat = data.frame(
         address = sample(1e6L, 1e7L, TRUE), 
           value = sample(1:5, 1e7L, TRUE, prob=c(0.5, 0.3, 0.1, 0.07, 0.03))
      )

data.table solution:

require(data.table) ## >= 1.9.2
dat.u = unique(setDT(dat), by=c("address", "value"))
ans   = dat.u[, .N, by=address]

Explanation:

• The setDT function converts a data.frame to data.table by reference (which is very fast).
• unique function operated on a data.table evokes the unique.data.table method, which is incredibly fast compared to base:::unique. Now, we've only unique values of type for every address.
• All that's left to do is to aggregate or group-by address and get the number of observations that are there in each group. The by=address part groups by address and .N is an in-built data.table variable that provides the number of observations for that group.

Benchmarks:

I'll create functions to generate data as data.table and data.frame to benchmark data.table answer againstdplyr solution (a) proposed by @beginneR, although I don't see the need for arrange(.) there and therefore will skip that part.

## function to create data
foo <- function(type = "df") {
    set.seed(100L)
    dat = data.frame(
             address = sample(1e6L, 1e7L, TRUE), 
               value = sample(1:5, 1e7L, TRUE, prob=c(0.5, 0.3, 0.1, 0.07, 0.03))
          )
    if (type == "dt") setDT(dat)
    dat
} 

## DT function
dt_sol <- function(x) {
    unique(x, by=c("address", "value"))[, .N, by=address]
}

## dplyr function
dplyr_sol <- function(x) {
    distinct(x) %>% group_by(address) %>% summarise(N = n_distinct(value))
}

The timings reported here are three consecutive runs of system.time(.) on each function.

## benchmark timings in seconds
##        pkg   run-01   run-02   run-03                                 command
## data.table     2.4       2.3      2.4  system.time(ans1 <- dt_sol(foo("dt")))
##      dplyr    15.3      16.3     15.7   system.time(ans2 <- dplyr_sol(foo()))

For some reason, dplyr automatically orders the result by the grouping variable. So in order to compare the results, I'll also order them in the result from data.table:

system.time(setkey(ans1, address)) ## 0.102 seconds
identical(as.data.frame(ans1), as.data.frame(ans2)) ## TRUE

So, data.table is ~6x faster here.

Note that bit64:::integer64 is also supported in data.table - since you mention the address values are too long, you can also store them as integer64.

Answer 2

You can try creating an index of your 10MM values and sort that. Then looking for your 1MM values in that indexed vector should be faster.

For example, using data.table package you can do that by using setkey function which indexes given column of data.table.

require(data.table)

set.seed(100)

dat <- sample(1:1e+07, size = 1e+07, replace = T)
searchval <- sample(dat, size = 1e+06)

DT <- data.table(dat, index = seq_along(dat))
setkey(DT, dat)
DT
##                dat   index
##        1:        1  169458
##        2:        1 4604823
##        3:        1 7793446
##        4:        2 5372388
##        5:        3 2036622
##       ---                 
##  9999996:  9999996 1271426
##  9999997:  9999998  530029
##  9999998: 10000000  556672
##  9999999: 10000000 6776063
## 10000000: 10000000 6949665


lookup <- data.table(val = searchval)
setkey(lookup, val)
lookup
##              val
##       1:       2
##       2:      16
##       3:      24
##       4:      33
##       5:      36
##      ---        
##  999996: 9999970
##  999997: 9999973
##  999998: 9999988
##  999999: 9999996
## 1000000: 9999998

Now you can lookup all the values from lookup in DT by simply using

DT[lookup]
##              dat   index
##       1:       2 5372388
##       2:      16  537927
##       3:      16 1721233
##       4:      24 7286522
##       5:      33 7448516
##      ---                
## 2000298: 9999973 8008610
## 2000299: 9999988 3099060
## 2000300: 9999988 7996302
## 2000301: 9999996 1271426
## 2000302: 9999998  530029