Google Cloud Speech to text API call fails due to invalid authentication credentials on Android

Question

I'm trying to use the google cloud speech to text API in my android client. I have enabled the API in the Google Cloud and in the console I have generate a new key like this: "Create credentials" -> "Service account" (added details) -> "Create and Continue". This has generated a JSON file that contains this params:

{
  "type": "service_account",
  "project_id": "",
  "private_key_id": "",
  "private_key": "",
  "client_email": "",
  "client_id": "",
  "auth_uri": "",
  "token_uri": "",
  "auth_provider_x509_cert_url": "",
  "client_x509_cert_url": "",
  "universe_domain": ""
}

I have added this JSON file in the raw dir in my android project (this is just a POC so not to worried about security at this point)

Next is the android code:

val req = RecognizeRequest.newBuilder()
               .setConfig(RecognitionConfig.newBuilder()
                        .setEncoding(RecognitionConfig.AudioEncoding.AMR_WB)
                        .setLanguageCode("en-US")
                        .setSampleRateHertz(16000)
                        .build())
                    .setAudio(RecognitionAudio.newBuilder()
                        .setContent(fileByteString))
                        .build()


            val speechClient = SpeechClient.create(SpeechSettings.newBuilder()              .setCredentialsProvider{GoogleCredentials.fromStream(this.resources.openRawResource(R.raw.credentials)) }


            val response = speechClient.recognize(req)

Log.d(TAG, "we have a count of ${response.resultsCount} hits on the  audio file")

            val results = response.resultsList

When the speechClient.recognize(req) fun is called, the API sends back this error message:

com.google.api.gax.rpc.UnauthenticatedException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
        at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:116)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
        at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
        at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84)
        at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1127)
        at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
        at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:574)
        at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:544)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:541)
        at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489)
        at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453)
        at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
        at java.lang.Thread.run(Thread.java:764)
     Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
        at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
        at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
        at com.google.cloud.speech.v1.SpeechClient.recognize(SpeechClient.java:252)

Does anyone know how to auth correctly when calling this API please?

I have also tried to generate OAuth & Service Key type of credentials in the Google Cloud Console but I was getting a different error saying something like "Credentials don't contain field `type`"

Answer 1

Please, consider review this related SO question, it is defined for TextToSpeechSettings but I think that the solution provided could be of application here as well.

As suggested, basically you need to create a valid GoogleCredentials object from the content of your service account json file:

// R.raw.credential points to the downloaded credential.json file
InputStream stream = getResources().openRawResource(R.raw.credential); 
GoogleCredentials credentials = GoogleCredentials.fromStream(stream);

And then, use the obtained credentials to initialize your SpeechClient.

For instance, adapting the code from my original answer:

import java.io.IOException;
import java.io.InputStream;
import java.util.List;

import com.google.api.gax.core.CredentialsProvider;
import com.google.api.gax.core.FixedCredentialsProvider;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.cloud.speech.v1.RecognitionAudio;
import com.google.cloud.speech.v1.RecognitionConfig;
import com.google.cloud.speech.v1.RecognizeRequest;
import com.google.cloud.speech.v1.RecognizeResponse;
import com.google.cloud.speech.v1.SpeechClient;
import com.google.cloud.speech.v1.SpeechRecognitionResult;
import com.google.cloud.speech.v1.SpeechSettings;
import com.google.protobuf.ByteString;

public class RecognitionActivity extends Activity {

  // Your activity definition

  private void performRecognition(ByteString fileByteString) throws IOException {
    // R.raw.credential points to the downloaded credential.json file
    // I tested my code locally in a laptop using
    // GoogleCredentials credentials = GoogleCredentials.fromStream(new FileInputStream("/path/to/credentials.json"));
    InputStream stream = getResources().openRawResource(R.raw.credential);
    GoogleCredentials credentials = GoogleCredentials.fromStream(stream);
    // You can use ServiceAccountCredentials as well instead of the line above:
    // ServiceAccountCredentials credentials = ServiceAccountCredentials.fromStream(stream);
    CredentialsProvider credentialsProvider = FixedCredentialsProvider.create(credentials);
    SpeechSettings speechSettings = SpeechSettings.newBuilder()
        .setCredentialsProvider(credentialsProvider)
        .build();

    SpeechClient speechClient = SpeechClient.create(speechSettings);

    // The rest of your code

    RecognizeRequest req = RecognizeRequest.newBuilder()
        .setConfig(RecognitionConfig.newBuilder()
            .setEncoding(RecognitionConfig.AudioEncoding.AMR_WB)
            .setLanguageCode("en-US")
            .setSampleRateHertz(16000)
            .build())
        .setAudio(RecognitionAudio.newBuilder()
            .setContent(fileByteString))
        .build();

    RecognizeResponse response = speechClient.recognize(req);

    Log.d(TAG, "we have a count of " + response.getResultsCount() + " hits on the  audio file")

    List<SpeechRecognitionResult> results = response.getResultsList();
    
    // Handle results
  }
}

In kotlin (please, forgive me for any inaccuracy) it would look like this:

private fun performRecognition(fileByteString: ByteString) {
    // R.raw.credential points to the downloaded credential.json file
    val stream: InputStream = getResources().openRawResource(R.raw.credential)
    val credentials = GoogleCredentials.fromStream(stream);
    // val credentials = ServiceAccountCredentials.fromStream(stream)
    val credentialsProvider = FixedCredentialsProvider.create(credentials)
    val speechSettings = SpeechSettings.newBuilder()
            .setCredentialsProvider(credentialsProvider)
            .build()

    val speechClient = SpeechClient.create(speechSettings)

    // The rest of your code
    val req = RecognizeRequest.newBuilder()
            .setConfig(RecognitionConfig.newBuilder()
                    .setEncoding(RecognitionConfig.AudioEncoding.AMR_WB)
                    .setLanguageCode("en-US")
                    .setSampleRateHertz(16000)
                    .build())
            .setAudio(RecognitionAudio.newBuilder()
                    .setContent(fileByteString))
            .build()

    val response = speechClient.recognize(req)

    Log.d(TAG, "we have a count of ${response.resultsCount} hits on the  audio file")

    val results = response.resultsList

    // Handle results
}

Please, be sure that you followed the necessary steps for setting up the Speech API, especially those related to configuring the service account.

The service account must be able to interact with the Speech API, assigning a convenient role to it such as Cloud Speech Client:

As described in the aforementioned documentation you can do it from the GCP Console or using the command line.