hadoop:启动datanode似乎没有响应

2skhul33  于 2021-06-01  发布在  Hadoop
关注(0)|答案(1)|浏览(426)

我的测试环境

我试图在我的测试环境中部署一个 Hadoop Cluster 基于3个节点:
1个名称节点(主节点:172.30.10.64)
2个数据节点(slave1:172.30.10.72和slave2:172.30.10.62)
我在namenode中配置了主属性文件,在datananodes中配置了从属性文件。

硕士学位论文

主持人:

  1. 127.0.0.1 localhost
  2. 172.30.10.64 master
  3. 172.30.10.62 slave2
  4. 172.30.10.72 slave1
  5. # The following lines are desirable for IPv6 capable hosts
  6. ::1 localhost ip6-localhost ip6-loopback
  7. ff02::1 ip6-allnodes
  8. ff02::2 ip6-allrouters

hdfs-site.xml:

  1. <configuration>
  2. <property>
  3. <name>dfs.replication</name>
  4. <value>2</value>
  5. </property>
  6. <property>
  7. <name>dfs.namenode.name.dir</name>
  8. <value>file:/usr/local/hadoop_tmp/hdfs/namenode</value>
  9. </property>
  10. </configuration>

core-site.xml:

  1. <configuration>
  2. <property>
  3. <name>fs.default.name</name>
  4. <value>hdfs://master:9000</value>
  5. </property>
  6. </configuration>

yarn-site.xml:

  1. <configuration>
  2. <property>
  3. <name>yarn.resourcemanager.resource-tracker.address</name>
  4. <value>master:8025</value>
  5. </property>
  6. <property>
  7. <name>yarn.resourcemanager.scheduler.address</name>
  8. <value>master:8030</value>
  9. </property>
  10. <property>
  11. <name>yarn.resourcemanager.address</name>
  12. <value>master:8050</value>
  13. </property>
  14. </configuration>

mapred-site.xml:

  1. <configuration>
  2. <property>
  3. <name>mapreduce.framework.name</name>
  4. <value>yarn</value>
  5. </property>
  6. <property>
  7. <name>mapreduce.jobhistory.address</name>
  8. <value>master:10020</value>
  9. </property>
  10. </configuration>

我有个文件:

  1. slave1
  2. slave2

主文件:

  1. master

从属文件:

我只添加了一些文件,这些文件与主文件不同。
hdfs-site.xml:

  1. <configuration>
  2. <property>
  3. <name>dfs.replication</name>
  4. <value>2</value>
  5. </property>
  6. <property>
  7. <name>dfs.datanode.data.dir</name>
  8. <value>file:/usr/local/hadoop_tmp/hdfs/datanode</value>
  9. </property>
  10. </configuration>

我的问题

我是从 /usr/local/hadoop/sbin :
./start-dfs.sh&&./start-yarn.sh
这就是我得到的:

  1. hduser@master:/usr/local/hadoop/sbin$ ./start-dfs.sh && ./start-yarn.sh
  2. 18/03/14 10:45:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  3. Starting namenodes on [master]
  4. hduser@master's password:
  5. master: starting namenode, logging to /usr/local/hadoop-2.7.5/logs/hadoop-hduser-namenode-master.out
  6. hduser@slave2's password: hduser@slave1's password:
  7. slave2: starting datanode, logging to /usr/local/hadoop-2.7.5/logs/hadoop-hduser-datanode-slave2.out

所以我打开了slave2的日志文件:

  1. 2018-03-14 10:46:05,494 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/172.30.10.64:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECOND$
  2. 2018-03-14 10:46:06,495 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/172.30.10.64:9000. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECOND$
  3. 2018-03-14 10:46:07,496 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/172.30.10.64:9000. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECOND$

我做了什么

我试过一些东西,但到目前为止没有效果:
从主人到奴隶,从奴隶到奴隶的过程都很好
从主人到奴隶和奴隶之间的ssh工作得很好 hdfs namenode -format 在我的主节点中
重新创建namenode和datanaode文件夹
在我的主虚拟机中打开端口9000
防火墙已禁用: sudo ufw status -->残疾人
我有点迷路,因为一切似乎都很好,我不知道为什么我不克服开始我的hadoop集群。

2guxujil

2guxujil1#

我也许能找到答案:
我从主节点重新生成ssh密钥,然后复制到从节点。它现在似乎起作用了。

  1. # Generate a ssh key for hduser
  2. $ ssh-keygen -t rsa -P ""
  3. # Authorize the key to enable password less ssh
  4. $ cat /home/hduser/.ssh/id_rsa.pub >> /home/hduser/.ssh/authorized_keys
  5. $ chmod 600 authorized_keys
  6. # Copy this key to slave1 to enable password less ssh and slave2 too
  7. $ ssh-copy-id -i ~/.ssh/id_rsa.pub slave1
  8. $ ssh-copy-id -i ~/.ssh/id_rsa.pub slave2

相关问题