GCP API Gateway to secure internal/external communication for API deployed on App Engine

Question

I am building a microservice architecture and I need help with internal/external communication.

I have microservices which are deployed on GCP App Engine Flex and have GCP API Gateway that sits in front of them. API Gateway handles external communication authentication using a JWT token sent in request header signed via service account private key.

On App Engine, we have configured Ingress (Internal + Load Balancer), so the App Engine's appspot URL are blocked externally. Each service has load balancer on which IAP is enabled and only API Gateway's service account has IAP-Secured Web App User role to pass request to LB.

My questions are :

• Should GCP API Gateway be used for internal service to service communication ?
• Since we have ingress (Internal + Load Balancer) enabled on App Engine and appspot URL are only accessible inside GCP project, can these URL be used for internal service to service communication ? Is this secure / recommended approach ?

Which of the above 2 suits well for the architecture to manage secure communication. Also, if possible, please suggest some alternatives.

Update : Adding flow diagram for both approaches

Answer 1

If you use ingress internal + LB for internal communication that means only the traffic coming from the VPC (of the current project) or the traffic coming from LB (of the current project) will be able to reach the service. Keep in ming that even if you set your traffic to internal, the IP is ALWAYS publicly accessible. There is simply an additional check perform on the traffic origin.

If you have another service on App Engine flex in your project, it should use either the LB (possible) or the VPC (route the traffic to the VPC even if it's a public URL -> That latest case is possible with Cloud Functions Cloud Run and App Engine standard (egress control feature, route all the traffic to the serverless VPC connector), but you can't with flex environment.

In addition, API Gateway can only reach public URL, and therefore you can only use the LB to reach your App Engine flex, and not the "internal" VPC traffic.