由于示例化yarclient时出错,解析mapreduce.cluster未能使用org.apache.hadoop.mapred.yarclientprotocolprovider

kgsdhlau  于 2021-05-29  发布在  Hadoop
关注(0)|答案(2)|浏览(665)

我已经在测试单节点集群上设置了基本的hadoop 2.6.0+hive 0.14.0,主要遵循以下指南:
hadoop:http://tecadmin.net/setup-hadoop-2-4-single-node-cluster-on-linux/
Hive:https://cwiki.apache.org/confluence/display/hive/gettingstarted
但是,如果运行wordcount java示例,会出现以下错误:

$ hadoop jar wc.jar WordCount /user/hadoop/HDFSClient.java /user/hadoop/wc.out
16/02/08 12:24:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/02/08 12:24:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/02/08 12:24:56 INFO mapreduce.Cluster: Failed to use org.apache.hadoop.mapred.YarnClientProtocolProvider due to error: Error in instantiating YarnClient
Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
        at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
        at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
        at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
        at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
        at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
        at WordCount.main(WordCount.java:59)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

conf/hdfs-site.xml文件:

<configuration>

<property>
 <name>dfs.replication</name>
  <value>1</value>
  </property>

<property>
  <name>dfs.name.dir</name>
      <value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
      </property>

<property>
  <name>dfs.data.dir</name>
      <value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
      </property>

<property>
  <name>dfs.namenode.http-address</name>
      <value>0.0.0.0:8000</value>
      </property>

</configuration>

conf/yarn-site.xml文件:

<configuration>

 <property>
   <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce.shuffle</value>
        </property>

 <property>
   <name>yarn.resourcemanager.webapp.address</name>
     <value>0.0.0.0:8004</value>
        </property>
</configuration>

conf/core-site.xml:

<configuration>

<property>
  <name>mapred.job.tracker.http.address</name>
      <value>50031</value>
   </property>
<property>
  <name>fs.default.name</name>
      <value>hdfs://localhost:9000</value>
   </property>
</configuration>

我是一个有点新的设置hadoop和Hive,并将感谢任何建议或指针上可能的事情看。

zengzsys

zengzsys1#

如果您在类路径中使用“hadoopmapreduceclientjobclient.jar”运行应用程序,那么这似乎是可行的。

0s7z1bwu

0s7z1bwu2#

用以下内容替换core-site.xml、mapred-site.xml、hdfs-site.xml和yarn-site.xml文件属性
core-site.xml文件

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>

hdfs-site.xml文件

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>

<property>
<name>dfs.namenode.name.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
</property>

<property>
<name>dfs.datanode.data.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
</property>
</configuration>

mapre-site.xml文件

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

yarn-site.xml文件

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

相关问题