我将hadoop(2.6.0)设置为多机模式:1个namenode+3个datanode。当我使用命令start-all.sh时,它们(namenode、datanode、resourcemanager、node manager)工作正常。我用jps命令检查了它,每个节点的结果如下:
名称节点:
7300资源管理器
6942名称节点
7154次要名称节点
数据节点:
3840数据节点
3924节点管理器
我还上传了hdfs上的示例文本文件:/user/hadoop/data/sample.txt。那一刻绝对没有错误。
但是当我尝试用hadoop示例的jar运行mapreduce时:
hadoop jar hadoop-mapreduce-examples-2.6.0.jar wordcount/user/hadoop/data/sample.txt/user/hadoop/output
我有一个错误:
15/04/08 03:31:26 INFO mapreduce.Job: Job job_1428478232474_0001 running in uber mode : false
15/04/08 03:31:26 INFO mapreduce.Job: map 0% reduce 0%
15/04/08 03:31:26 INFO mapreduce.Job: Job job_1428478232474_0001 failed with state FAILED due to: Application application_1428478232474_0001 failed 2 times due to Error launching appattempt_1428478232474_0001_000002. Got exception: java.net.ConnectException: Call From hadoop/127.0.0.1 to localhost:53245 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy31.startContainers(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:119)
at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:254)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 9 more Failing the application.
15/04/08 03:31:26 INFO mapreduce.Job: Counters: 0
关于配置,请确保namenode可以ssh到datanodes,反之亦然,无需提示密码。我还禁用了ip6并修改了/etc/hosts文件:
127.0.0.1本地主机hadoop
192.168.56.102 hadoop网络
192.168.56.103 hadoop-dn1版本
192.168.56.104 hadoop-dn2
192.168.56.105 hadoop-dn3版本
我不知道为什么mapreduced不能运行,尽管namenode和datanodes工作正常。我在这里几乎被绊倒了,你能帮我找到原因吗??
谢谢
编辑:hdfs-site.xml(namenode)中的配置:
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///usr/local/hadoop/hadoop_stores/hdfs/namenode</value>
<description>NameNode directory for namespace and transaction logs storage.</description>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.use.datanode.hostname</name>
<value>false</value>
</property>
<property>
<name>dfs.namenode.datanode.registration.ip-hostname-check</name>
<value>false</value>
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>hadoop-nn:50070</value>
<description>Your NameNode hostname for http access.</description>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop-nn:50090</value>
<description>Your Secondary NameNode hostname for http access.</description>
</property>
在数据节点中:
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///usr/local/hadoop/hadoop_stores/hdfs/data/datanode</value>
<description>DataNode directory</description>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.use.datanode.hostname</name>
<value>false</value>
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>hadoop-nn:50070</value>
<description>Your NameNode hostname for http access.</description>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop-nn:50090</value>
<description>Your Secondary NameNode hostname for http access.</description>
下面是命令的结果:hadoopfs-ls/user/hadoop/data
hadoop@hadoop用法:~/data$hadoop fs-ls/user/hadoop/data 15/04/09 00:23:27找到2项
-rw-r--r--3 hadoop超群29 2015-04-09 00:22>/user/hadoop/data/sample.txt
-rw-r--r--3 hadoop超群27 2015-04-09 00:22>/user/hadoop/data/sample1.txt
hadoop fs-ls/user/hadoop/output
ls:`/user/hadoop/output':没有这样的文件或目录
2条答案
按热度按时间up9lanfz1#
防火墙问题:
java.net.connectexception:连接被拒绝
此错误可能是由于防火墙问题造成的。在终端中执行此操作:
在继续之前检查文件是否已创建(因为如果出现问题,这将用于恢复防火墙)。
现在,刷新iptable规则(即停止防火墙):
现在试试看,
此命令不应返回任何规则。现在,尝试运行map/reduce作业。
注意:如果要将iptables恢复到以前的状态,请在终端中键入以下内容:
sudo iptables-restore < /usr/iptables-backup/iptables.v4.rules
lg40wkob2#
找到解决方案!!请参阅本文—将数据节点id/名称显示为localhost
主机和从机在/etc/hostname中的主机名都是localhost.localdomain。
我把奴隶的主人的名字改成了奴隶1和奴隶2。成功了。谢谢大家抽出时间。
@请确保namenode和datanodes中的etc/hostname未设置为localhost。只需在终端中键入~#hostname即可查看。您可以使用相同的命令设置新的主机名。
我的主人和工人或奴隶的/etc/hosts是这样的-
worker1的主机名
和工人2
另外,您可能不希望使用带有环回接口的“hadoop”主机名。即。
检查中的这一点(1)https://wiki.apache.org/hadoop/connectionrefused.
谢谢您。