How to create an AVRO file using Go?

Question

I'm trying to create an AVRO file using Go. So far I tried a couple of libraries and I have some code.

The problem is that I can work with the data but don't know how to serialize it to store it. Here's the code I got from github.com/hamba/avro with some small modifications.

import (
  "fmt"
  "github.com/hamba/avro"
  "log"
)

type SimpleRecord struct {
        A int64  `avro:"a"`
        B string `avro:"b"`
}

func main() {
    schema, err := avro.Parse(`{
        "type": "record",
        "name": "simple",
        "namespace": "hamba",
        "fields" : [
            {"name": "a", "type": "long"},
            {"name": "b", "type": "string"}
        ]
    }`)
    if err != nil {
        log.Fatal(err)
    }

    in := SimpleRecord{A: 27, B: "foo"}

    data, err := avro.Marshal(schema, in)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(data)
}

This block of code prints:

[54 6 102 111 111]

This line corresponds to the avro encoding of the data. And it seems like this is all I need to store, but I don't know how to create the file itself.

I tried:

mode := int(0644)
    permissions := os.FileMode(mode)
    err = ioutil.WriteFile("file.avro", data, permissions)
    if err != nil {
        log.Fatal(err)
    }

And it generates a file. However, when I try to read it as an AVRO file using Python fastavro library, I get the error ValueError: cannot read header - is it an avro file?.

But according to the docs (https://godoc.org/github.com/hamba/avro#example-Marshal): "Marshal returns the Avro encoding of v." Marshal(schema Schema, v interface{}) ([]byte, error), so data should be of type []byte.

Answer 1

Avro defines the data encoding format only which can be packaged as messages or files. So, for file storage should use Avro OCF - Avro Object Container Files. Here is a working hamba avro ocf encoder example.

In my code I've encoded multiple rows to upload it to BigQuery (error checks, init, and close are omitted for clarity):

f, err := os.Open("/your/avro/file.avro")
enc, err := ocf.NewEncoder(schema, w, ocf.WithCodec(ocf.Snappy))
for _, item := range items {
    enc.Encode(item)
}