Error: Connecting to a SFTP server with ssh2-sftp-client in a AWS lambda function throws a time out

Question

I'm trying to connect to a SFTP-Server and list the documents in the /ARCHIVE Folder. Credentials are stored in a .env file. When I run this on my local machine it works and lists the documents.

async function main (event){
    let Client = require('ssh2-sftp-client');
    let sftp = new Client();
    const dotenv = require('dotenv');
    dotenv.config();

    const ftpOptions = {
        host: process.env.FTP_HOST,
        port: process.env.FTP_PORT,
        username: process.env.FTP_USER,
        password: process.env.FTP_PASSWORD,
        debug: console.log
    }

    await sftp.connect(ftpOptions);
    let documentList = await sftp.list('/ARCHIVE');
    console.log(documentList);
    sftp.end();
}

But if I try it in my AWS lambda function, it tries to connect and times out. The env variables are loaded and avaiable before the sftp.connect(ftpOptions) is executed. The logs show that the function is trying to connect to the server, but can't even login with the credentials.

Function Logs:
START RequestId: 20d3c3b7-19d8-49cf-8997-1f2331eb0b6e Version: $LATEST
2020-06-04T08:27:12.837Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    INFO    Debugging turned on
2020-06-04T08:27:12.860Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    INFO    DEBUG: Local ident: 'SSH-2.0-ssh2js0.4.10'
2020-06-04T08:27:12.898Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    INFO    DEBUG: Client: Trying ftp_foo.com on port 22 ...
2020-06-04T08:27:32.929Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    INFO    CLIENT[sftp]: Connection attempt 1 failed. Trying again.
2020-06-04T08:27:33.931Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    INFO    DEBUG: Local ident: 'SSH-2.0-ssh2js0.4.10'
2020-06-04T08:27:33.931Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    INFO    DEBUG: Client: Trying ftp_foo.com on port 22 ...
2020-06-04T08:27:53.951Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    INFO    CLIENT[sftp]: Exhausted all connection attempts. Giving up
2020-06-04T08:27:53.957Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    ERROR   Invoke Error    {"errorType":"Error","errorMessage":"connect: Timed out while waiting for handshake after 2 attempts","code":"ERR_GENERIC_CLIENT","custom":true,"stack":["Error: connect: Timed out while waiting for handshake after 2 attempts","    at Object.formatError (/var/task/node_modules/ssh2-sftp-client/src/utils.js:62:18)","    at Client.connectErrorListener (/var/task/node_modules/ssh2-sftp-client/src/index.js:98:21)","    at Object.onceWrapper (events.js:417:26)","    at Client.emit (events.js:310:20)","    at Timeout._onTimeout (/var/task/node_modules/ssh2/lib/client.js:697:14)","    at listOnTimeout (internal/timers.js:549:17)","    at processTimers (internal/timers.js:492:7)"]}
2020-06-04T08:27:53.959Z    20d3c3b7-19d8-49cf-8997-1f2331eb0b6e    ERROR   Unhandled Promise Rejection     {"errorType":"Runtime.UnhandledPromiseRejection","errorMessage":"Error: end: No SFTP connection available","reason":{"errorType":"Error","errorMessage":"end: No SFTP connection available","code":"ERR_NOT_CONNECTED","custom":true,"stack":["Error: end: No SFTP connection available","    at formatError (/var/task/node_modules/ssh2-sftp-client/src/utils.js:62:18)","    at Object.haveConnection (/var/task/node_modules/ssh2-sftp-client/src/utils.js:612:20)","    at /var/task/node_modules/ssh2-sftp-client/src/index.js:1248:19","    at new Promise (<anonymous>)","    at SftpClient.end (/var/task/node_modules/ssh2-sftp-client/src/index.js:1236:12)","    at downloadPDFs (/var/task/index.js:40:14)","    at processTicksAndRejections (internal/process/task_queues.js:97:5)"]},"promise":{},"stack":["Runtime.UnhandledPromiseRejection: Error: end: No SFTP connection available","    at process.<anonymous> (/var/runtime/index.js:35:15)","    at process.emit (events.js:310:20)","    at processPromiseRejections (internal/process/promises.js:209:33)","    at processTicksAndRejections (internal/process/task_queues.js:98:32)"]}
[ERROR] [1591259273978] LAMBDA_RUNTIME Failed to post handler success response. Http response code: 403.

Is there maybe something, that happens on my local machine to provide the connection which I didn't implemented in the lambda function?

Answer 1

The issue could be that your Lambda function does not have access to the SFTP host.

Is this SFTP server hosted outside of AWS or within your account ?

If it's hosted in the internet and not by you on AWS, you can do the following:

1. Create a public VPC [Skip this if you already have a public VPC]
2. Add a NAT Gateway in that public VPC [Also skip this if you already have a NAT Gateway]
3. In the VPC where your Lambda function resides, add a route table which redirects traffic to the NAT Gateway by setting the destination as 0.0.0.0/0 and the target as your NAT Gateway which we created in step 2.

You can find a detailed answer on how to add internet connection to your Lambda function by visiting this AWS article: How do I give internet access to my Lambda function in a VPC?

If the SFTP server is already hosted by you in AWS, you can simply add access to it by using AWS security group.

Check this article on how to connect Lambda with another AWS service (it's using RDS but you can follow the same logic with your SFTP server): Configuring a Lambda function to access Amazon RDS in an Amazon VPC

Also, make sure you set the Lambda timeout to be long enough to allow connecting and exchange data with an external SFTP server.