Why does my Lambda function time out even though the API Gateway callback has already been called?

Question

I have an AWS API Gateway method that proxies requests through to AWS Lambda. However, it errors after three seconds with the following in the logs:

Endpoint response body before transformations: {"errorMessage":"2017-09-05T16:30:49.987Z 922186c0-9257-11e7-9db3-51921d5597a2 Task timed out after 3.00 seconds"}

Thus, I went on to check my Node 6.10 AWS Lambda function to see why it was timing out. I added logging statements before and after every function call. Surprisingly, it did everything it's supposed to do: called the API Gateway callback, and run a query against the database after that. All that takes 0.6s, and as far as I'm aware there's no other code left to run. Nevertheless, it appears to keep on running for the rest of the three seconds and then timing out. (This is, I think, because I'm leaving a connection to the database open.)

The logs statements I placed before and after the callback call indicate that the that call is executed in under half a second. Yet, that response doesn't seem to make it to API Gateway, whereas the error after three seconds does.

What could be potential reasons for this, and how can I debug it?

Answer 1

By default calling the callback() function in a NodeJS Lambda function does not end the function execution. It will continue running until the event loop is empty. A common issue with NodeJS Lambda functions continuing to run after callback is called occurs when you are holding on to open database connections. You haven't posted any code, so I can't give specific recommendations, but you would need to determine if you are leaving database connections open in your code or something similar.

Alternatively, you can change the behavior such that the execution ends as soon as the callback function is called by setting callbackWaitsForEmptyEventLoop = false on the context object.

Answer 2

Your API gateway has a fixed timeout of 29 seconds. Most queries do complete within this timeframe. Increase your lambda execution timeout to anywhere between 30 Sec to 3 Min 00 Sec. Use context.succeed() instead of callback. This worked for me.

const mysql = require('mysql');
const connection = mysql.createConnection({
  host     : 'your_mysql_host',
  user     : 'your_mysql_user',
  password : 'your_mysql_password',
  database : 'your_mysql_db'
});

exports.handler = (event, context) => {
  var userId = event.params.querystring.userid;
  const sql = 'SELECT * FROM users where USER_ID=' + userId;

  var response = {
    "statusCode": 200, 
    "body": "body_text_goes_here"
  }

  if(userId){
    connection.query(sql, function (error, results, fields) {
      if (error) {
        context.succeed(error);
      } else {
        response.body = results;
        context.succeed(response);
      }
    });
  }
}