How do we set maximum_bad_records when loading a Bigquery table from dataflow?

Question

Is there a way to set the maximum number of bad records when writing to BigqueryIO? It seems to keep the default at 0.

Answer 1

At this time, unfortunately, we don't provide a way to directly set the value of configuration.load.maxBadRecords in relation to BigQueryIO in Cloud Dataflow.

As a workaround, you should be able to apply a custom ParDo transform that filters "bad records" before they are passed to BigQueryIO.Write. As a result, BigQuery shouldn't get any "bad records". Hopefully, this helps.

If the ability to control configuration.load.maxBadRecords is important to you, you are welcome to file a feature request in the issue tracker of our GitHub repository.