deleting NA values from huge rasterfile

Question

I am currently attempting to remove NA values from a huge raster file (1.9*10^7 observations). In these rasters 99.9% are NA values. My aim is to remove NA and create a .csv file conataining all non-NA values. my attempt is as follows:

# Load packages
packs = c('raster', 'rgdal')
sapply(packs, FUN = 'require', character.only = TRUE)
xy <- xyFromCell(raster, 1:ncell(raster))
v <- as.data.frame(raster)
xyv <- data.frame(xy, v)
rm(xy,v)
xyv <- na.omit(xyv)
write.csv(xyv, file ="raster.csv", row.names = F)

When i execute na.omit() R/Rstudio gives an error message that it has encountered a fatal error and terminates. Is there a simpler and faster solution to execute this?

Answer 1

You can use the rasterToPoints function for that.

library(raster)
r <- raster()
r[50:52] <- 1:3
xyv <- rasterToPoints(r)

write.csv(xyv, file ="raster.csv", row.names = FALSE)

Answer 2

Whenever I see a large array with mostly missing values, I think "sparse matrix" as an efficient way to hold the data. If the non-missing data in your raster are all non-zero, then using a sparse matrix is straightforward. If there are zeros in the data, then one extra step (included below) is needed.

First lets create a large raster with mostly NA's. And also create a matrix from it.

my.raster <- raster(nrows=1e3, ncols=1e4, xmn=0, xmx=10, vals=NA)
my.raster[sample(1:(1e3*1e4), 100)] <- as.integer(runif(100,0,100))
my.matrix <- as.matrix(my.raster)

Sparse matrices only store the non-zero elements, so to make this sparse we need to change NA's to zeroes. In case the data may already contain zeroes that we don;t want to lose track of, we store the locations of the zeroes before making the matrix sparse.

library(Matrix)
zeros <- data.frame(xyFromCell(my.raster, which(my.matrix == 0)), val=0)
my.matrix[is.na(my.matrix)] <- 0
sp <- as(Matrix(my.matrix, sparse=T), "dgTMatrix") # use triplet form of sparse matrix

Now the values are in sp@x, and the coordinates are stored in @i and @j. So, to save to .csv

my.df <- data.frame(x = xFromCol(my.raster, sp@j), y = yFromRow(my.raster, sp@i), val=sp@x)
my.df <- rbind(zeros, my.df)
write.csv(my.df, file ="raster.csv", row.names = F)