我已经在测试单节点集群上设置了基本的hadoop 2.6.0+hive 0.14.0,主要遵循以下指南:
hadoop:http://tecadmin.net/setup-hadoop-2-4-single-node-cluster-on-linux/
Hive:https://cwiki.apache.org/confluence/display/hive/gettingstarted
但是,如果运行wordcount java示例,会出现以下错误:
$ hadoop jar wc.jar WordCount /user/hadoop/HDFSClient.java /user/hadoop/wc.out
16/02/08 12:24:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/02/08 12:24:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/02/08 12:24:56 INFO mapreduce.Cluster: Failed to use org.apache.hadoop.mapred.YarnClientProtocolProvider due to error: Error in instantiating YarnClient
Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
at WordCount.main(WordCount.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
conf/hdfs-site.xml文件:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>0.0.0.0:8000</value>
</property>
</configuration>
conf/yarn-site.xml文件:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>0.0.0.0:8004</value>
</property>
</configuration>
conf/core-site.xml:
<configuration>
<property>
<name>mapred.job.tracker.http.address</name>
<value>50031</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
我是一个有点新的设置hadoop和Hive,并将感谢任何建议或指针上可能的事情看。
2条答案
按热度按时间zengzsys1#
如果您在类路径中使用“hadoopmapreduceclientjobclient.jar”运行应用程序,那么这似乎是可行的。
0s7z1bwu2#
用以下内容替换core-site.xml、mapred-site.xml、hdfs-site.xml和yarn-site.xml文件属性
core-site.xml文件
hdfs-site.xml文件
mapre-site.xml文件
yarn-site.xml文件