Problem sending binary files via sockets, python

Question

I'm trying to write a program that transfers binary files from the client to the server. Here's the code:

Client (send file)

  def send_file(self,filename):
        print("Sending: " + filename)
        size = self.BUFFER_SIZE
        with open(filename,'rb') as f:
            raw = f.read().decode()
        buffer = [raw[i:i + size] for i in range(0, len(raw), size)]
        for x in range(len(buffer)):
            self.sock.sendall(buffer[x].encode())

        return

Server (recv file)

def recv_file(self, conn, filename):
    packet = ""
    buffer = ""
    while True:
        buffer = conn.recv(self.BUFFER_SIZE)
        packet = packet + str(buffer.decode())
        if not len(buffer) == self.BUFFER_SIZE:
            break
    with open(filename, 'wb') as f:
        f.write(bytes(packet.encode()))
    #print(packet)
    return 

This way I can transfer txt files, but when I have to transfer jpeg or any other type of file, it freezes in the loop. Can someone please explain me why? I'm new to py and i'm trying to learn

Answer 1

It shouldn't freeze if both sides have the same locale encoding, but it could easily die with an exception.

You're reading and sending as binary (good), but inexplicably decode-ing to str, then encodeing back to bytes (bad). Problem is, arbitrary binary data isn't guaranteed to be decodable in any given locale; if your locale encoding is UTF-8, odds are it's not legal. If it's latin-1 it's legal, but pointless.

Worse, if your client and server have different locale encodings, the result of decoding might be different on each side (and therefore the lengths won't match).

Use bytes consistently, don't convert to and from strings, and locale settings won't matter. Your code will also run faster. You also need to actually send the file length ahead of time; your loop is hoping recv will return a short length only when the file is done, but if:

1. The file is an exact multiple of the buffer size, or
2. The socket happens to send data in chunks that don't match the buffer size

you can each get short recv results, by coincidence in case #2, and deterministically in case #1.

A safer approach is to actually prefix your transmission with the file length, rather than hoping the chunking works as expected:

def send_file(self,filename):
    print("Sending:", filename)
    with open(filename, 'rb') as f:
        raw = f.read()
    # Send actual length ahead of data, with fixed byteorder and size
    self.sock.sendall(len(raw).to_bytes(8, 'big'))
    # You have the whole thing in memory anyway; don't bother chunking
    self.sock.sendall(raw)

def recv_file(self, conn, filename):
    # Get the expected length (eight bytes long, always)
    expected_size = b""
    while len(expected_size) < 8:
        more_size = conn.recv(8 - len(expected_size))
        if not more_size:
            raise Exception("Short file length received")
        expected_size += more_size

    # Convert to int, the expected file length
    expected_size = int.from_bytes(expected_size, 'big')

    # Until we've received the expected amount of data, keep receiving
    packet = b""  # Use bytes, not str, to accumulate
    while len(packet) < expected_size:
        buffer = conn.recv(expected_size - len(packet))
        if not buffer:
            raise Exception("Incomplete file received")
        packet += buffer
    with open(filename, 'wb') as f:
        f.write(packet)