decode large base64 from xml in java: OutOfMemory

Question

I need to write a base64 encoded element of an xml file into a separate file. Problem: the file could easily reach the size of 100 MB. Every solution I tried ended with the "java.lang.OutOfMemoryError: Java heap space". The problem is not reading the xml in general or the decoding process, but the size of the base64 block.

I used jdom, dom4j and XMLStreamReader to access the xml file. However, as soon as I want to access the base64 content of the respective element I get the mentioned error. I also tried an xslt using saxon's base64Binary-to-octets function, but of course with the same result.

Is there a way to stream this base64 encoded part into a file without getting the whole chunk in one single piece?

Thanks for your hints,

Andreas

Answer 1

Apache Commons Codec has a Base64OutputStream, which should allow you to feed the XML data in a scalable way, by chaining the Base64OutputStream with a FileOutputStream.

You'll need a representation of the XML as a String, so you may not even have to read it into a DOM structure at all.

Something like:

PrintWriter printWriter = new PrintWriter(
   new Base64OutputStream(
      new BufferedOutputStream(
         new FileOutputStream("/path/to/my/file")
      )
   )
);
printWriter.write(myXml);
printWriter.close();

If the input XML file is too big, then you should read chunks of it into a buffer in a loop, writing the buffer contents to the output (i.e. a standard reader-to-writer copy).