Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Failed to load native-lzo library when setting up Lzo on Cloudera Hadoop

I just followed the steps in the Cloudera Document, and had GPL Extras Parcel installed on the cluster as well as configured HDFS service via Cloudera Manager. But an error ocurred when trying to read .lzo files on HDFS:

$hadoop fs -text /tmp/Lzo/log.txt.lzo 
INFO lzo.GPLNativeCodecLoader: Loaded native gpl library
WARN lzo.LzoCompressor: java.lang.NoSuchFiledError: lzoCompressLevelFunc
ERROR lzo.LzoCodec: Failed to load/initialize native-lzo library
-text: Fatal internal error
java.lang.RuntimeException: native-lzo library not available

I've read a dozen of posts, and known that it's caused by a failure on JNI loading lzo library, but none of them could properly solve my problem. Following are the efforts I've made:
1.All datanodes had installed lzop.
2.JAVA_LIBRARY_PATH in mapred-site.xml was set to /opt/cloudera/parcels/CDH/lib/hadoop/lib/native, which contains liblzo2.* files.
3.HADOOP_CLASSPATH was set to /usr/local/lib which contains hadoop-lzo.jar files.

What else can I do? Any suggestions would be appreciated!

like image 825
Paul Lam Avatar asked Nov 22 '25 06:11

Paul Lam


1 Answers

Problem solved! It's caused by java.lang.NoSuchFiledError. Hadoop-lzo-0.4.15 do not have the field lzoCompressLevelFunc, and when I switched to hadoop-lzo0.4.20 the WARN and the ERROR were gone.

like image 101
Paul Lam Avatar answered Nov 24 '25 21:11

Paul Lam



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!