sequencefile不能与gzipcodec一起工作

56lgkhnf  于 2021-06-02  发布在  Hadoop
关注(0)|答案(0)|浏览(534)

在mac os x上,当我使用压缩编解码器是hadoop时,会出现以下错误:

java.lang.IllegalArgumentException: SequenceFile doesn't work with GzipCodec without native-hadoop code!
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:386)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:354)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:476)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:61)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.(ReduceTask.java:569)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:638)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)

但我已经配置了本地库(我已经在本地构建了hadoop,并将本机libs复制到hadoop目录。

hadoop checknative -a 

16/06/06 13:16:54 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
16/06/06 13:16:54 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /opt/hadoop-2.5.0-cdh5.3.3/lib/native/libhadoop.dylib
zlib:    true /usr/lib/libz.1.dylib
snappy:  true /opt/hadoop-2.5.0-cdh5.3.3/lib/native/libsnappy.1.dylib
lz4:     true revision:99
bzip2:   false 
openssl: false org.apache.hadoop.crypto.OpensslCipher.initIDs()V
16/06/06 13:16:54 INFO util.ExitUtil: Exiting with status 1

所以看起来本地lib还可以。
ps:当我使用b2zipcodec时,一切都很好。
pss:也许我的个人资料有助于了解发生了什么?

export JAVA_LIBRARY_PATH=/opt/hadoop-2.5.0-cdh5.3.3/lib/native
export LD_LIBRARY_PATH=/opt/hadoop-2.5.0-cdh5.3.3/lib/native
export HADOOP_HOME=/opt/hadoop-2.5.0-cdh5.3.3
export M2_HOME="/opt/maven/"
export MAVEN_OPTS="-Xms1024m -Xmx3000m -XX:MaxPermSize=228m -Dmaven.surefire.debug=-Xmx3000m"
export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.7.0_72.jdk/Contents/Home"
export PATH=${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:${HCAT_HOME}/bin:${M2_HOME}/bin:${PATH}

更新此配置半年前运行良好。经过一些软件更新后,它坏了。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题