How to scale up a gRPC pub/sub chat service?

Question

I'm studying how to make a chat service with gRPC. I notice that most of the examples store all the subscriber's connections into a List data structure. When the chatroom got a new message, the server will loop through that List and send new messages.

Stores subscribers in source code's variable subscribers.

My question is that when the subscriber number grows up and storing all the subscribers in the HashMap seems like a bad idea because that will cost too much memory? I have tried to store those connections in Redis but it might be impossible due to the connection is not serializable.

My Idea is when I deploy multiple pod instances in Kubernetes. Each chat service becomes independent. How could I get the subscribers correctly when they are distributed on different servers?

Answer 1

Here are a couple of aspects that I'm thinking:

• Any chat system has some limits in terms of chatroom size. How many subscribers do you want to support? If you want to support tens (or even hundreds) of subscribers, I believe it works well with an in-memory list of subscribers, since it's much faster than having the subscriber's list in an external cache

• When you need to send a message to the entire list of subscribers, instead of looping on that list, you can do that in parallel, using an IO-optimized thread-pool.
• Deploying your chat-service in a pod is a good idea, you need the ability to scale horizontally, but you also need a smart-gateway in front of your service, to route the requests from a given user, to the pod that stores its information (if the subscribers list is stored in memory). Otherwise, if this info is stored externally (in a cache), your chat service can be fully stateless.