How to zip multiple csv files into archive and return it in FastAPI?

Question

I'm attempting to compose a response to enable the download of reports. I retrieve the relevant data through a database query and aim to store it in memory to avoid generating unnecessary files on the server. My current challenge involves saving the CSV file within a zip file. Regrettably, I've spent several hours on this issue without finding a satisfactory solution, and I'm uncertain about the specific mistake I may be making. The CSV file in question is approximately 40 MB in size.

This is my FastAPI code. I successfully saved the CSV file locally, and all the data within it is accurate. I also managed to correctly create a zip file containing the CSV. However, the FastAPI response is not behaving as expected. After downloading it returns me zip with error:

The ZIP file is corrupted, or there's an unexpected end of the archive.

from fastapi import APIRouter, Depends
from sqlalchemy import text
from libs.auth_common import veryfi_admin
from libs.database import database
import csv
import io
import zipfile
from fastapi.responses import Response

router = APIRouter(
    tags=['report'],
    responses={404: {'description': 'not found'}}
)


@router.get('/raport', dependencies=[Depends(veryfi_admin)])
async def get_raport():
    query = text(
        """
            some query
        """
    )

    data_de = await database.fetch_all(query)

    csv_buffer = io.StringIO()
    csv_writer_de = csv.writer(csv_buffer, delimiter=';', lineterminator='\n')

    csv_writer_de.writerow([
        "id", "name", "date", "stock",
    ])

    for row in data_de:
        csv_writer_de.writerow([
            row.id,
            row.name,
            row.date,
            row.stock,

        ])
    csv_buffer.seek(0)

    zip_buffer = io.BytesIO()
    with zipfile.ZipFile(zip_buffer, 'w', zipfile.ZIP_DEFLATED) as zip_file:
        zip_file.writestr("data.csv", csv_buffer.getvalue())

    response = Response(content=zip_buffer.getvalue())
    response.headers["Content-Disposition"] = "attachment; filename=data.zip"
    response.headers["Content-Type"] = "application/zip"
    response.headers["Content-Length"] = str(len(zip_buffer.getvalue()))

    print("CSV Buffer Contents:")
    print(csv_buffer.getvalue())
    return response

Here is also the vue3 code

const downloadReport = () => {
  loading.value = true;
  instance
    .get(`/raport`)
    .then((res) => {
      const blob = new Blob([res.data], { type: "application/zip" });
      const link = document.createElement("a");
      link.href = window.URL.createObjectURL(blob);
      link.download = "raport.zip";
      link.click();
      loading.value = false;
    })
    .catch(() => (loading.value = false));
};
<button @click="downloadReport" :disabled="loading">
      Download Report
</button>

Thank you for your understanding as I navigate through my first question on this platform.

Answer 1

Here's a working example on how to create multiple csv files, then compress them into a zip file, and finally, return the zip file to the user. This answer makes use of code and concepts previously discussed in the following answers: this, this and this. Thus, I would suggest having a look at those answers for more details.

Also, since the zipfile module's operations are synchronous, you should define the endpoint with normal def instead of async def, unless you used some third-party library that provides an async API as well, or you had to await for some coroutine (async def function) inside the endpoint, in which case I would suggest running the zipfile's operations in an external ThreadPool (since they are blocking IO-bound operations). Please have a look at this answer for relevant solutions, as well as details on async / await and how FastAPI deals with async def and normal def API endpoints.

Further, you don't really need using StreamingResponse, if the data are already loaded into memory, as shown in the example below. You should instead return a custom Response (see the example below, as well as this, this and this for more details).

Note that the example below uses utf-16 encoding on the csv data, in order to make it compatible with data that include unicode or non-ASCII characters, as explained in this answer. If there's none of such characters in your data, utf-8 encoding could be used as well.

Also, note that for demo purposes, the example below loops through a list of dict objects to write the csv data, in order to make it more easy for you to adapt it to your database query data case. Otherwise, one could also csv.DictWriter() and its writerows() method, as demosntrated in this answer, in order to write the data, instead of looping through the list.

Working Example

from fastapi import FastAPI, HTTPException, BackgroundTasks, Response
import zipfile
import csv
import io


app = FastAPI()


fake_data = [
  {
    "Id": "1",
    "name": "Alice",
    "age": "20",
    "height": "62",
    "weight": "120.6"
  },
  {
    "Id": "2",
    "name": "Freddie",
    "age": "21",
    "height": "74",
    "weight": "190.6"
  }
]


def create_csv(data: list):
    s = io.StringIO()
    try:
        writer = csv.writer(s, delimiter='\t')
        writer.writerow(data[0].keys())
        for row in data:
            writer.writerow([row['Id'], row['name'], row['age'], row['height'], row['weight']])
        s.seek(0)
        return s.getvalue().encode('utf-16')
    except:
        raise HTTPException(detail='There was an error processing the data', status_code=400)
    finally:
        s.close()


@app.get('/')
def get_data():
    zip_buffer = io.BytesIO()
    try:
        with zipfile.ZipFile(zip_buffer, 'w', zipfile.ZIP_DEFLATED) as zip_file:
            for i in range(5):
                zip_info = zipfile.ZipInfo(f'data_{i}.csv')
                csv_data = create_csv(fake_data)
                zip_file.writestr(zip_info, csv_data)
            
        zip_buffer.seek(0)
        headers = {"Content-Disposition": "attachment; filename=files.zip"}
        return Response(zip_buffer.getvalue(), headers=headers, media_type="application/zip")
    except:
        raise HTTPException(detail='There was an error processing the data', status_code=400)
    finally:
        zip_buffer.close()