Dataflow Flex template job is Queued

Question

I am trying to reproduce this tutorial to run a Flex Template on Dataflow.

When I submit the job, I can see it in the console but it's not started and marked as Queued. Does this mean that the job is submitted in a FlexRS mode ? How can I start directly the job after submitting it ?

Answer 1

The "Queued" status for Flex Template jobs means that your container is running on a VM to build the pipeline and start the job. If the job stays in Queued for more than a few minutes, that indicates the this process got stuck. You can view the logs for this VM in the Dataflow UI in the "Job Logs" section.

Answer 2

This looks like a bug, where an upstream dependency (pyarrow?) started requiring git in order to build, but the base image does not currently include git. I have filed an issue here: https://issuetracker.google.com/issues/176570473