Socket authentication and encryption

Question

This includes multiple questions but since they are somewhat related I wanted to post them together.

What I'm doing: I'm writing a server-client application where users have to login in order to communicate with the server. Right now I'm using udp (and yes I'm sure I want to use udp) with some minor modifications.

First part:

What would be the best way to store a user connection?

My ideas:

1. Create a container storing all addresses of all clients which are allowed to connect (after successful login)
2. Create a container storing all session ids (session id will be sent with every packet)

Other ideas appreciated (especially if they are already being used)

My concerns:

1. Could someone change the address of the sender of a packet? (I assume yes)
2. Session ids could get sniffed out. (I remember some big company names having this problem)

Second part:

Nevertheless I would have to encrypt my packets. In case of (2) the encryption could be related to the session id so that only a packet from a user can be decrypted with the correct session key corresponding to that client (AES like, just to provide an example).

This would require an appropriate algorithm which is fast (there may be 30-50 packets with 256 bytes sent per second from a single client)

1. Which algorithm would be appropriate for this (RSA seems to be a little bit too slow)?
2. How would this algorithm work? (Only a very short summary but a source for further information is appreciated)
3. Would it encrypt the packet so that it would be as big as the original packet or would it be bigger so that I have to write some kind of caching mechanism on the server side which would assemble these packets?

Oh and btw. I don't need an explanation on public/private keys, handshakes etc. It might be important to know that I would use this algorithm in a commercial product (licence-wise).

Answer 1

This is hard to answer without a specific application in mind, but I'll try to give some generic hints:

Create a container storing all addresses of all clients which are allowed to connect (after successful login)

This simply will not work because of NAT, which is sadly still in use and actually even increasing because of IPv4 exhaustion. You'd at least need src-ip+src-port. Even then, considering mobile users you might never want to use IP as a session ID. An average smartphone will switch between cellular and WiFi networks quite easily, most often leading to a complete restart of the IP-stack, so there is no way to correlate the new traffic with the previous traffic. This might or might not be an issue, but I would never use this approach unless you have control over the IP addresses.

Create a container storing all session ids (session id will be sent with every packet)

This is actually the generic solution, your first solution is just a specific implementation where you use the source-ip as a session id. If you're concerned about session-id management, just use UUID's, the odds of collisions between session id's are ridiculously low. Or, when using public/private key encryption you could use a user's public key as a session ID.

An important part here is how to negotiate the session-id's. You might want to let the user choose, you might want to come in with a 'special' session-id (e.g. 0) to let the server choose. What is the best depends on your application.

Could someone change the address of the sender of a packet? (I assume yes)

Sure, this is called a man-in-the-middle attack (if done on an in-transit packet) or ip address spoofing (if sending a packet with a fake IP) and is undetectable for most end-users. Though many networks have protection against this, for example using Reverse Path Forwarding.

Session ids could get sniffed out. (I remember some big company names having this problem)

If encrypted: maybe (see later). If not encrypted: certainly.

About the whole part of your encryption questions then:

In general you're on the right track, you usually want to use a symmetric-key encryption scheme for your regular traffic. AES is a good choice but there are others out there, go do some research.

However, you have a problem with setting up the encryption. In general you need to get the encryption key on both sides securely, without people sniffing them. You could try to send the keys through airmail, but I doubt most users would find that user-friendly and even that isn't really safe.

That's where the asymmetric key encryption schemes come in. You will usually use something like RSA to negotiate the initial connection (session id, encryption key, maybe some accounting, ...) and let a symmetric key take over for the actual traffic. A popular scheme is the Diffie-Hellman key exchange, but again there are more of those out there.

That all being said you can secure your channel quite fine, but a man-in-the-middle attack stays always a concern. It turns out there is actually little you can do to really prevent this as you have no control over one of the sides (the client), if it is an infected machine all bets are off:

1. Use a unique, pre-distributed private key per user that you can verify with incoming sessions. This will make mitm attacks harder if they do not get that key some other way, but non auto generated private keys are often hard to combine with user-friendliness. You have troubles with how to distribute them (darn, how do I handle mim there?), how to store them at the user (oh, he uses a laptop, an iPhone AND an iPad), how to recover them if lost, ...
2. Make sure that all traffic is initiated by the client and encrypted immediately with the server's public key. This is easier as you don't have to distribute a private key. However, a hacker can still replace your server's public key with it's own key, but it's a lot harder and if done right almost comes down to installing a virus on the client computer.
3. Do some sanity checks in your client application. For example, make sure you're connecting to a known pool of server IP's, check if the DNS query is correct, etc etc. It's far from fail-safe but it are simple verification that will discourage potential hackers.
4. Educate your users. This is what many banks do (at least where I live), let them do a regular anti-virus check, only use trusted WiFi-networks, verify the DNS servers, ... . Of course some things are harder to teach then others, but a little common sense gets you a long way.

Oh and finally I do actually want to comment on the UDP part: are you really really sure? Because almost everything of this scheme and even a lot more is covered by TLS, which is integrated in boost asio. If your traffic is that low-rate I can hardly imagine it's an application that needs the advantages that UDP offers, unless you want to secure voip, which is done already, don't reinvent the wheel.