Google cloud_sql_proxy unable to connect to instance, stream error, protocol_error

Question

I've been successfully using the Google cloud_sql_proxy on multiple Compute Engine instances for some time, until today, one instance at a time, the proxy started to show the following error pattern:

2017/05/30 13:28:07 New connection for "project-id-1234:us-central1:sql_instance"
2017/05/30 13:28:07 couldn't connect to "project-id-1234:us-central1:sql_instance": Post https://www.googleapis.com/sql/v1beta4/projects/project-id-1234/instances/sql_instance/createEphemeral?alt=json: stream error: stream ID 1; PROTOCOL_ERROR
2017/05/30 13:28:41 New connection for "project-id-1234:us-central1:sql_instance"
2017/05/30 13:28:41 Thottling refreshCfg(project-id-1234:us-central1:sql_instance): it was only called 33.490705951s ago
2017/05/30 13:28:41 couldn't connect to "project-id-1234:us-central1:sql_instance": Post https://www.googleapis.com/sql/v1beta4/projects/project-id-1234/instances/sql_instance/createEphemeral?alt=json: stream error: stream ID 1; PROTOCOL_ERROR

When trying to connect directly to MySQL (while using the proxy) I get error 2013 (HY000):

ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0 "Internal error/check (Not system error)"

What I've tried

• Restarting the cloud_sql_proxy yielded a temporary fix until finally both my Compute Engine instances are unable to connect to my Cloud SQL instance and the proxies show only this result.
• Restarting the Cloud SQL instance and both Compute Engine instances.
• Eliminating the proxy: I added the appropriate networks to my SQL instance's Authorized Networks, and updated all applications to use the public IP. This restored functionality to my production apps, but now I'm using a public connection instead of local/proxy.

Some research

• I came across a similar issue relating to Google Cloud SQL that yielded the same MySQL error above, but it appears to have only affected connecting to Cloud SQL from external, non GCE/GKE networks.
• A few others have reported the same issue also started for them this morning on the Google Cloud SQL Discuss group.

Answer 1

My team started seeing the same issue appear today, with GKE managed servers. Same as you saw: restarts of servers and DB did nothing.

We tried doing an update of the version of Google Cloud Proxy we were using from v1.05 to v1.09 and the problem went away (for now).

I know that's not much of an explanation but give it a try to see if that helps you.