Spark history logs decompress manually

Question

I have some problems with spark history server and therefore tried to uncompress lz4 logs manually. I installed liblz4-tool and downloaded application log application_1510049252849_0303.lz4. But when uncompressing it with lz4 ~/application_1510049252849_0303.lz4 I got the following error:

Error 44 : Unrecognized header : file cannot be decoded 

And this I have for every spark history lz4 files. What is the solution? Is there a way to open it manually?

Answer 1

I had the same error yesterday while trying to decompress the lz4 event logs. Apparently there seems to be some incompatibilities between hadoop lz4 codec and other lz4 codecs: see https://issues.apache.org/jira/browse/HADOOP-12990.

For what it's worth, spark history server can read compressed event logs in lz4, so you don't need to decompress them manually. Hope this helps.