How can I send large files in base64 format?

Question

I have a Spring & Angular application and I developed a service to upload and download files.

The files that will be downloaded are fetched from a folder on my server (so I will use FTP to put the files in the folder) then encoded in base64 and displayed to the user.

The user can download the file and I somehow managed to let the user download large files by dividing the long base64 string into small chuncks of 256 bytes (I tested it locally and it works even if it's still pretty slow).

The problem is in my Spring back-end, i can't compress a base64 string and the only way I have to display the files in my architecture is by returning base64 strings but when it comes to large files I get the following error:

    Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space

Here's the snippet of the service I'm using to return the base64 string:

    @Override
    public List<FileDto> loadAll() {
        List<FileDto> files = new ArrayList<FileDto>();
        
        try {
            File folder = new File("uploads"+ File.separator +"adminfolder");
            File[] listOfFiles = folder.listFiles();
            
            for (int i = 0; i < listOfFiles.length; i++) {
                String directory = folder.toString() + File.separator + listOfFiles[i].getName();

                String allegato = BufferedImageUtility.encodeFileToBase64Binary(directory);
                String tipoFile = BufferedImageUtility.getMimeType(directory);
                String estensione = "";
                
                files.add(new FileDto("", allegato, listOfFiles[i].getName(), tipoFile, estensione));
            }
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

And here the encodeFileToBase64Binary() method:

    public static String encodeFileToBase64Binary(String fileName) {
            String encodedString = "";
            
            try {
                File file = new File(fileName);
                byte[] bytes = loadFile(file);
                byte[] encoded = Base64.getEncoder().encode(bytes);
                encodedString = new String(encoded,StandardCharsets.US_ASCII);
            } catch (IOException e) {
                logger.debug("errore encodeFileToBase64Binary, ", e);
                throw new EFWMException(e);
            }
            
            return encodedString;
    }

Answer 1

Instead of loading the whole file into memory with loadFile(file), I would suggest you try to stream it into Base64.

Try this for encoding:

  public static String encodeFileToBase64Binary(final String fileName)
     {
        try(final ByteArrayOutputStream out = new ByteArrayOutputStream())
        {
            Files.copy(Path.of(fileName), Base64.getEncoder().wrap(out));
            return out.toString(StandardCharsets.US_ASCII);
        }
        catch (final IOException e)
        {
            logger.debug("errore encodeFileToBase64Binary, ", e);
            throw new EFWMException(e);
        }
     }