在mac os x上,当我使用压缩编解码器是hadoop时,会出现以下错误:
java.lang.IllegalArgumentException: SequenceFile doesn't work with GzipCodec without native-hadoop code!
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:386)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:354)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:476)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:61)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.(ReduceTask.java:569)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:638)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)
但我已经配置了本地库(我已经在本地构建了hadoop,并将本机libs复制到hadoop目录。
hadoop checknative -a
16/06/06 13:16:54 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
16/06/06 13:16:54 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /opt/hadoop-2.5.0-cdh5.3.3/lib/native/libhadoop.dylib
zlib: true /usr/lib/libz.1.dylib
snappy: true /opt/hadoop-2.5.0-cdh5.3.3/lib/native/libsnappy.1.dylib
lz4: true revision:99
bzip2: false
openssl: false org.apache.hadoop.crypto.OpensslCipher.initIDs()V
16/06/06 13:16:54 INFO util.ExitUtil: Exiting with status 1
所以看起来本地lib还可以。
ps:当我使用b2zipcodec时,一切都很好。
pss:也许我的个人资料有助于了解发生了什么?
export JAVA_LIBRARY_PATH=/opt/hadoop-2.5.0-cdh5.3.3/lib/native
export LD_LIBRARY_PATH=/opt/hadoop-2.5.0-cdh5.3.3/lib/native
export HADOOP_HOME=/opt/hadoop-2.5.0-cdh5.3.3
export M2_HOME="/opt/maven/"
export MAVEN_OPTS="-Xms1024m -Xmx3000m -XX:MaxPermSize=228m -Dmaven.surefire.debug=-Xmx3000m"
export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.7.0_72.jdk/Contents/Home"
export PATH=${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:${HCAT_HOME}/bin:${M2_HOME}/bin:${PATH}
更新此配置半年前运行良好。经过一些软件更新后,它坏了。
暂无答案!
目前还没有任何答案,快来回答吧!