Efficiently listing files in a directory having very many entries

Question

I need to recursively read a directory structure, but I also need to perform an additional step once I have read through all entries for each directory. Therefore, I need to write my own recursion logic (and can't use the simplistic filepath.Walk routine). However, the ioutil.ReadDir and filepath.Glob routines only return slices. What if I'm pushing the limits of ext4 or xfs and have a directory with files numbering into the billions? I would expect golang to have a function that returns an unsorted series of os.FileInfo (or, even better, raw strings) over a channel rather than a sorted slice. How do we efficiently read file entries in this case?

All of the functions cited above seem to rely on readdirnames in os/dir_unix.go, and, for some reason, it just makes an array when it seems like it would've been easy to spawn a gothread and and push the values into a channel. There might have been sound logic to do this, but it's not clear what it is. I'm new to Go, so I also could've easy missed some principle that's obvious to everyone else.

This is the sourcecode, for convenience:

func (f *File) readdirnames(n int) (names []string, err error) {
    // If this file has no dirinfo, create one.
    if f.dirinfo == nil {
        f.dirinfo = new(dirInfo)
        // The buffer must be at least a block long.
        f.dirinfo.buf = make([]byte, blockSize)
    }
    d := f.dirinfo

    size := n
    if size <= 0 {
        size = 100
        n = -1
    }

    names = make([]string, 0, size) // Empty with room to grow.
    for n != 0 {
        // Refill the buffer if necessary
        if d.bufp >= d.nbuf {
            d.bufp = 0
            var errno error
            d.nbuf, errno = fixCount(syscall.ReadDirent(f.fd, d.buf))
            if errno != nil {
                return names, NewSyscallError("readdirent", errno)
            }
            if d.nbuf <= 0 {
                break // EOF
            }
        }

        // Drain the buffer
        var nb, nc int
        nb, nc, names = syscall.ParseDirent(d.buf[d.bufp:d.nbuf], n, names)
        d.bufp += nb
        n -= nc
    }
    if n >= 0 && len(names) == 0 {
        return names, io.EOF
    }
    return names, nil
}

Answer 1

ioutil.ReadDir and filepath.Glob are just convenience functions around reading directory entries.

You can read directory entries in batches by directly using the Readdir or Readdirnames methods, if you supply an n argument > 0.

For something as basic as reading directory entries, there's no need to add the overhead of a goroutine and channel, and also provide an alternate way to return the error. You can always wrap the batched calls with your own goroutine and channel pattern if you wish.