我的第一个wordcount mapreduce程序中出现以下错误:
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
cat: `/home/Kumar/DEV/Eclipse/eclipse/Workspace/MyFirstMapReduce/Files/input/': No such file or directory
操作系统和hadoop版本
CentOS
Release 6.6 (Final)
Kernel Linux 2.6.32-504.12.2.el6.x86_64
GNOME 2.28.2
Hadoop 2.6.0 64bit version
bashrc配置
export JAVA_HOME ="/usr/lib/jvm/jdk1.8.0"
export PATH =$PATH:$JAVA_HOME/bin"
export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar
export HADOOP_INSTALL="/home/Kumar/DEV/HDS/hadoop"
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
运行守护程序
3763 SecondaryNameNode
4406 ResourceManager
22264 org.eclipse.equinox.launcher_1.3.0.v20140415-2008.jar
17736 Jps
30584 org.eclipse.equinox.launcher_1.3.0.v20140415-2008.jar
4697 NodeManager
3611 DataNode
4059 NameNode
我从apache站点下面复制了wordcount程序,并按照本教程中给出的步骤进行了操作。当我编译wordcount.java时,它创建了3个类文件:
hadoop com.sun.tools.javac.Main /home/Kumar/DEV/Eclipse/eclipse/Workspace/MyFirstMapReduce/src/WordCount.java
WordCount.class
WordCount$IntSumReducer.class
WordCount$TokenizerMapper.class
当我运行hdfs命令时,它会抛出下面的警告消息和file或directory not found消息,即使file和director存在
hdfs dfs -cat /home/Kumar/DEV/Eclipse/eclipse/Workspace/MyFirstMapReduce/Files/input/file1
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
cat: `/home/Kumar/DEV/Eclipse/eclipse/Workspace/MyFirstMapReduce/Files/input/file1': No such file or directory
hdfs dfs -cat /home/Kumar/DEV/Eclipse/eclipse/Workspace/MyFirstMapReduce/Files/input/
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
cat: `/home/Kumar/DEV/Eclipse/eclipse/Workspace/MyFirstMapReduce/Files/input/': No such file or directory
2条答案
按热度按时间mkh04yzy1#
您尝试使用的文件位于本地linux文件系统上,而不是hdfs上。因此,可以使用以下简单命令:
cat /home/Kumar/DEV/Eclipse/eclipse/Workspace/MyFirstMapReduce/Files/input/file1
如果mapreduce程序需要hdfs上的输入文件,则将相同的文件放在hdfs上,并将hdfs路径放在job中。您可以使用此命令将文件放在hdfs上:hadoop fs -put <input-file-path-on-linux> <path-on-hdfs>
您所指的教程使用mapreduce的旧API。您可以在此处查看当前hadoop版本的相同教程:http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapreducetutorial.html希望这对你有帮助。
piwo6bdm2#
第一条消息只是一个警告,并不麻烦。
出现第二条消息是因为
hadoop -fs
将查看hdfs,而不是本地文件系统。您可能需要检查您的文件是否与一起存在ls
.