我试图在aws上设置的集群上运行wordcount示例。它挂了，只是说运行作业。
我在resourcemanager日志中发现了此错误
我可以通过hdfs ui查看我的所有节点(namenode:50070).
但是，当我试图通过namenode:8088/cluster/nodes 上面说有0个节点？
有什么想法吗？我尝试过编辑yarn-site.xml来指定最小/最大内存和内核，但没有成功。
**编辑以下是nodemanager日志文件中的错误 2018-02-08 19:28:41,110 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 8042 2018-02-08 19:28:41,111 INFO org.mortbay.log: jetty-6.1.26 2018-02-08 19:28:41,246 INFO org.mortbay.log: Extract jar:file:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.9.0.jar!/webapps/node to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp 2018-02-08 19:28:42,777 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:8042 2018-02-08 19:28:42,777 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app node started at 8042 2018-02-08 19:28:42,783 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Node ID assigned is : ec2-34-227-117-73.compute-1.amazonaws.com:39885 2018-02-08 19:28:42,797 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8031 2018-02-08 19:28:42,798 INFO org.apache.hadoop.util.JvmPauseMonitor: Starting JVM pause monitor 2018-02-08 19:28:42,861 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out 0 NM container statuses: [] 2018-02-08 19:28:42,866 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registering with RM using containers :[] 2018-02-08 19:28:43,935 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-02-08 19:28:44,936 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-02-08 19:28:45,937 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-02-08 19:28:46,937 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-02-08 19:28:47,938 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-02-08 19:28:48,939 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
1条答案
按热度按时间dphi5xsq1#
你在理解hadoop时犯了一个常见的错误。hadoop由一个文件系统（hdfs）和一个计算引擎（yarn）组成。数据节点只显示hdfs功能。要运行作业，您需要资源管理器，还需要节点管理器来提供计算能力。
资源管理器的屏幕截图证明了这一点。您没有运行节点管理器，因此没有vCore或内存可供计算。