How do I adjust redis settings to play well with Heroku-redis?

Question

I have a couple settings configurations that incorporate some kind of Redis backend and they're all different.

This one is required by django-channels:

# REDIS BACKEND
redis_host = os.environ.get('REDIS_HOST', 'localhost')
# Channel layer definitions
# http://channels.readthedocs.org/en/latest/deploying.html#setting-up-a-channel-backend
CHANNEL_LAYERS = {
    "default": {
        "BACKEND": "channels_redis.core.RedisChannelLayer",
        "CONFIG": {
            "hosts": [(redis_host, 6379)],
        },
    },
}  

and makes use of this channels_redis library.

I also have this:

CACHES = {
    'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': f'{env("REDIS_URL", default="redis://127.0.0.1:6379")}/{0}',
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
            # Mimicing memcache behavior.
            # http://niwinz.github.io/django-redis/latest/#_memcached_exceptions_behavior
            'IGNORE_EXCEPTIONS': True,
        }
    }
}  

cache storage which comes default thanks to Cookiecutter-django and uses the django-redis backend.

I'm now trying to upload the app to Heroku, but Heroku requires that we use their Heroku-Redis addon.

Upon deploying my app, I'm getting what I believe are redis issues--the same errors that occur during local development which are solved by running redis-server in a terminal, are popping up once the app is deployed on Heroku.

Unfortunately, the only proof of this I can provide is this output:

2018-03-30T07:50:42.366099+00:00 heroku[router]: at=info method=GET path="/chat/loadhistory/" host=paira.herokuapp.com request_id=b52c5600-56c5-4711-b743-85df254bf9bb fwd="121.129.196.176" dyno=web.1 connect=0ms service=609ms status=101 bytes=145 protocol=https
2018-03-30T07:50:45.402850+00:00 heroku[router]: at=info method=GET path="/chat/stream/" host=paira.herokuapp.com request_id=a10fe599-0960-4836-af91-694b8041fc92 fwd="121.129.196.176" dyno=web.1 connect=0ms service=1016ms status=101 bytes=145 protocol=https
2018-03-30T07:50:45.696224+00:00 app[web.1]: 10.79.192.24:30767 - - [30/Mar/2018:16:49:48] "WSCONNECT /chat/loadhistory/" - -
2018-03-30T07:50:45.696234+00:00 app[web.1]: 10.79.192.24:30767 - - [30/Mar/2018:16:49:49] "WSDISCONNECT /chat/loadhistory/" - -
2018-03-30T07:50:45.696236+00:00 app[web.1]: 10.5.185.134:42175 - - [30/Mar/2018:16:49:49] "WSCONNECTING /chat/stream/" - -
2018-03-30T07:50:45.696238+00:00 app[web.1]: 10.5.185.134:42175 - - [30/Mar/2018:16:49:49] "WSCONNECT /chat/stream/" - -
2018-03-30T07:50:45.696239+00:00 app[web.1]: 10.5.185.134:42175 - - [30/Mar/2018:16:49:50] "WSDISCONNECT /chat/stream/" - -
2018-03-30T07:50:45.696241+00:00 app[web.1]: 10.43.181.164:17852 - - [30/Mar/2018:16:49:52] "WSCONNECTING /chat/loadhistory/" - -
2018-03-30T07:50:45.696243+00:00 app[web.1]: 10.43.181.164:17852 - - [30/Mar/2018:16:49:52] "WSCONNECT /chat/loadhistory/" - -
2018-03-30T07:50:45.696244+00:00 app[web.1]: 10.43.181.164:17852 - - [30/Mar/2018:16:49:53] "WSDISCONNECT /chat/loadhistory/" - -
2018-03-30T07:50:45.696246+00:00 app[web.1]: 10.11.169.30:26270 - - [30/Mar/2018:16:49:53] "GET /inbox/notifications/api/unread_list/?max=10" 200 475
2018-03-30T07:50:45.696248+00:00 app[web.1]: 10.228.182.163:23985 - - [30/Mar/2018:16:49:53] "WSCONNECTING /chat/stream/" - -
2018-03-30T07:50:45.696249+00:00 app[web.1]: 10.228.182.163:23985 - - [30/Mar/2018:16:49:53] "WSCONNECT /chat/stream/" - -
2018-03-30T07:50:45.696251+00:00 app[web.1]: 10.228.182.163:23985 - - [30/Mar/2018:16:49:54] "WSDISCONNECT /chat/stream/" - -
2018-03-30T07:50:45.696252+00:00 app[web.1]: 10.13.221.137:33008 - - [30/Mar/2018:16:49:56] "WSCONNECTING /chat/loadhistory/" - - 

This is the output given to me by Heroku when I open the page where I'm loading websockets.
In local development, I'd encounter the same output, but they'd be promptly silenced once I run redis-server in a terminal.

A few things to note: Heroku's help center commented on this issue:

"I'm not familiar with using Redis with Django, but if the issue is with your Redis connection string, then I would suggest using the REDIS_URL for this – this string also includes your host, you shouldn't need to set that as a separate config variable."

So with my own environment variables, I set the REDIS_URL to the one provided by Heroku. Although they say I shouldn't need to "set [it] as a separate config variable", I extracted only the HOST from their Redis connection string and set it as a REDIS_HOST environment variable, since it seems the Django-Channels CHANNEL_LAYERS definition requires redis_host.

Edit:
I've tried removing the CACHES settings which did not help.

Edit:
A bit more background on the output--It's the result of me using the reconnecting-websocket library for the frontend alongside Django-channels. So it appears that the WS connection repeatedly attempts to(or successfully) connects, but then is promptly disconnected.

I don't believe this is a websocket issue as it works fine locally. I also sort of confirmed it isn't an issue related to WS timeouts as I got this response from Andrew:

The underlying Daphne code will send PING requests to the client periodically, adjustable with --ping-interval. If it doesn't get a response back before --ping-timeout, it will close the connection.

Heroku have their own loadbalancer, so it's possible that is changing how the sockets behave.

Edit:
OK, so I've narrowed it down: Locally, the app will work only if I run redis-server without setting the REDIS_HOST. It'll work with or without REDIS_URL.

So it breaks if I change REDIS_HOST under CHANNEL_LAYERS to anything other than localhost.
Given that this appears to be the chokepoint, I'm attempting setting REDIS_HOST on Heroku as the host that I extracted from the REDIS_URL string provided by Heroku: redis://h:pf954cfs86918ca88905dcf8fef4546dbc2738d5895b12bc36ed2d058c387ec@ec2-33-201-236-230.compute-1.amazonaws.com:7719
That's the whole string, so I assume the HOST would be only: ec2-33-201-236-230.compute-1.amazonaws.com. Furthermore, I noticed Heroku-Redis provides their redis instance under a different port than 6379. So I also tried adjusting my Django settings like so:

CHANNEL_LAYERS = {
    "default": {
        "BACKEND": "channels_redis.core.RedisChannelLayer",
        "CONFIG": {
            "hosts": [(redis_host, 7719)],
        },
    },
}  

Still doesn't work..

Any ideas?

Answer 1

Ultimately the issue is with semantics, due to a differing/liberal use of terms in the channels documentation.

Short answer:
Set up your CHANNEL_LAYERS like this:

CHANNEL_LAYERS = {
    "default": {
        "BACKEND": "channels_redis.core.RedisChannelLayer",
        "CONFIG": {
            "hosts": ["redis://h:954c886918c238905dc2c322c34546bd9dbc2738d32523b12bc36ed2d058c387ec@ec2-34-211-446-320.compute-1.amazonaws.com:7719"],
        },
    },
}  

Long answer:
Here and here, "redis-server-name" and "localhost" actually refer to the entire redis URI string beginning with "redis://" and ending with the more traditional definition of "host"--it includes "redis://" + user + ":password" + "@host:".

So here where it says "host", you need to place more than just the host. Or, you can opt to skip the tuple and pass the entire URI as a string.

On second glance, I haven't figured out what exactly Channels wants from you if you opt to use the tuple. I'll be going with the URI string for now.