Redis connection lost without any indication

Question

I'm using a very simple redis pub-sub application, in which I have a redis server in AWS and a nodejs based redis client that is located inside office LAN that subscribes to some channel.

This worked great until the network changed and it seems that some device is now interfering with outgoing connections (I also started receiving socket hangups on outbound SSH connections which I mitigated with the ServerAliveInterval 60 setting in the SSH config).

After the network change, whenever the redis client application is executed, it creates a redis client, subscribes to some channel and acts upon published messages in that channel. It works okay for several minutes, but then it stops receiving any messages.

I registered the redis client to all known connection events (including the "error" event), I added a "retry_strategy" handler and also modified the configuration to have "socket_keepalive" and "socket_initialdelay" to 10 seconds (see code below).

Nevertheless, no event is triggered when the connection is interfered.

When the application stops receiving the messages, I see that the connection on the redis port is still valid:

dev@server:~> sudo netstat -tlnpua | grep 6379
tcp        0      0 10.43.22.150:52052      <server_ip>:6379     ESTABLISHED 27014/node

I also captured a PCAP on port 6379 on which I don't see any resets or TCP errors, and it seems that from the connection perspective everything is valid.

I tried running another nodejs application from within the LAN in which I create a client that connects to the AWS redis server, registers to all events and only publishes messages once in a while. After several minutes (in which the connection breaks), I try publishing another command and the error event handler is indeed triggered:

> client.publish("channel", "ANOTHER TRY")
true
> Error: Redis connection to <server_hostname>:6379 failed - read ECONNRESET
Redis connection ended
Redis reconnecting
Redis connected
Redis connection is ready

So if I try publishing via the client after the connection was interfered, the connection event callbacks are indeed called and I can run some kind of reconnection logic.

But in the scenario in which I subscribe and wait for publishes to the channel, no connection event handler is called and the application is basically broken.

Application code:

const redis = require('redis');

const config = { "host": <hostname>, "port": 6379, "socket_keepalive": true, 
"socket_initdelay": 10};

config.retry_strategy =  function (options) {
    console.log("retry strategy. error code: " + (options.error ? 
options.error.code : "N/A"));
    console.log("options.attempt", options.attempt, "options.total_retry_time", 
options.total_retry_time);
    return 2000;
}

const client = redis.createClient(config);

client.on('message', function(channel, message) {
    console.log("Channel", channel, ", message", message);
});

client.on("error", function (err) {
    console.log("Error " + err);
});

client.on("end", function () {
    console.log("Redis connection ended");
});

client.on("connect", function () {
    console.log("Redis connected");
});

client.on("reconnecting", function () {
    console.log("Redis reconnecting");
});

client.on("ready", function () {
    console.log("Redis connection is ready");
});

const channel = "channel";
console.log("Subscribing to channel", channel);
client.subscribe(channel);

I'm using [email protected] and node v8.11.3.

Answer 1

The solution for this issue is quite sad.

First, there is indeed some network device between the redis client and server, which drops inactive connections after some timeout. It seems that this timeout is really low (several minutes).

Redis has a socket_keepalive configuration which is enabled by default, and its default value is Node.js's default socket keep alive value (which is set for 2 hours if i'm not mistaken).

As can be seen above, I used a socket_initdelay configuration parameter that should have changed this default value, but unfortunately the code that uses this parameter isn't in the redis npm package but rather in node-redis.

To summarize: There is no configuration setting to change the keep alive timeout value in [email protected] (latest version when writing this post). You can either:

1. Use node-redis which accepts the socket_initdelay setting.

2. Modify the timeout manually by running the following:

const client = redis.createClient();
client.on("connect", function () {
    client.stream.setKeepAlive(true, <timeout_value_in_milliseconds>);
}