如何知道执行的从机数量

t8e9dugd  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(358)

我有一个关于Map/减少执行的问题。我有一个wordcount代码，我执行它（我有2个从机和1个主机）。我怎么知道奴隶2被处决了？我怎么能做这样的事 println 在控制台中执行从机时获得通知？
我在函数Map的一部分尝试了这个步骤：

String ip = InetAddress.getLocalHost().getHostAddress().toString();

我使用ip并将其作为密钥发送（值为 new intWritable 因此reduce接收执行map代码的从机的ip并将其写入输出文件。但这并不成功，因为它写的是从机1或从机2的ip地址，而不是2个从机的地址，我使用：

DistributedCache.addCacheFile(link,job.getConfiguration());

用于在所有从属服务器上分发所有文件（而不是将文件的一部分分发给每个从属服务器）。
最后我得到：
节点管理器日志：

2015-05-17 22:35:52,855 INFO SecurityLogger.org.apache.hadoop.ipc.Server: 
Auth successful for appattempt_1431892621101_0005_000001 (auth:SIMPLE)
2015-05-17 22:35:54,294 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1431892621101_0005_01_000002 by user hduser
2015-05-17 22:35:54,579 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1431892621101_0005
2015-05-17 22:35:54,707 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduser   IP=192.168.1.11 OPERATION=Start Container Request   TARGET=ContainerManageImpl  RESULT=SUCCESS  APPID=application_1431892621101_0005    CONTAINERID=container_1431892621101_0005_01_000002
2015-05-17 22:35:55,092 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1431892621101_0005 transitioned from NEW to INITING
2015-05-17 22:35:55,102 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_1431892621101_0005_01_000002 to application application_1431892621101_0005
2015-05-17 22:35:55,145 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1431892621101_0005 transitioned from INITING to RUNNING
2015-05-17 22:35:55,642 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000002 transitioned from NEW to LOCALIZING
2015-05-17 22:35:55,658 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1431892621101_0005
2015-05-17 22:35:55,663 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_INIT for appId application_1431892621101_0005
2015-05-17 22:35:55,663 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got APPLICATION_INIT for service mapreduce_shuffle
2015-05-17 22:35:55,759 INFO org.apache.hadoop.mapred.ShuffleHandler: Added token for job_1431892621101_0005
2015-05-17 22:35:55,909 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://master:54310/tmp/hadoop-yarn/staging/hduser/.staging/job_1431892621101_0005/job.jar transitioned from INIT to DOWNLOADING
2015-05-17 22:35:55,909 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://master:54310/tmp/hadoop-yarn/staging/hduser/.staging/job_1431892621101_0005/job.xml transitioned from INIT to DOWNLOADING
2015-05-17 22:35:56,085 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_1431892621101_0005_01_000002
2015-05-17 22:35:56,734 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-hduser/nm-local-dir/nmPrivate/container_1431892621101_0005_01_000002.tokens. Credentials list: 
2015-05-17 22:36:00,700 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hduser
2015-05-17 22:36:01,816 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-hduser/nm-local-dir/nmPrivate/container_1431892621101_0005_01_000002.tokens to /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005/container_1431892621101_0005_01_000002.tokens
2015-05-17 22:36:01,865 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005 = file:/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005
2015-05-17 22:36:22,696 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://master:54310/tmp/hadoop-yarn/staging/hduser/.staging/job_1431892621101_0005/job.jar(->/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005/filecache/10/job.jar) transitioned from DOWNLOADING to LOCALIZED
2015-05-17 22:36:22,979 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://master:54310/tmp/hadoop-yarn/staging/hduser/.staging/job_1431892621101_0005/job.xml(->/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005/filecache/11/job.xml) transitioned from DOWNLOADING to LOCALIZED
2015-05-17 22:36:22,979 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000002 transitioned from LOCALIZING to LOCALIZED
2015-05-17 22:36:23,752 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000002 transitioned from LOCALIZED to RUNNING
2015-05-17 22:36:23,902 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005/container_1431892621101_0005_01_000002/default_container_executor.sh]
2015-05-17 22:36:26,604 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1431892621101_0005_01_000002
2015-05-17 22:36:29,051 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7030 for container-id container_1431892621101_0005_01_000002: 28.3 MB of 1 GB physical memory used; 1.7 GB of 2.1 GB virtual memory used
2015-05-17 22:36:36,703 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7030 for container-id container_1431892621101_0005_01_000002: 35.3 MB of 1 GB physical memory used; 1.7 GB of 2.1 GB virtual memory used
2015-05-17 22:36:42,008 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7030 for container-id container_1431892621101_0005_01_000002: 66.4 MB of 1 GB physical memory used; 1.7 GB of 2.1 GB virtual memory used
2015-05-17 22:36:50,807 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7030 for container-id container_1431892621101_0005_01_000002: 78.4 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used
2015-05-17 22:36:56,039 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7030 for container-id container_1431892621101_0005_01_000002: 87.3 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used
2015-05-17 22:37:13,734 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7030 for container-id container_1431892621101_0005_01_000002: 233.7 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used
2015-05-17 22:37:17,816 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1431892621101_0005_000001 (auth:SIMPLE)
2015-05-17 22:37:18,024 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1431892621101_0005_01_000002
2015-05-17 22:37:18,094 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduser   IP=192.168.1.11 OPERATION=Stop Container Request    TARGET=ContainerManageImpl  RESULT=SUCCESS  APPID=application_1431892621101_0005    CONTAINERID=container_1431892621101_0005_01_000002
2015-05-17 22:37:18,130 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000002 transitioned from RUNNING to KILLING
2015-05-17 22:37:18,149 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1431892621101_0005_01_000002
2015-05-17 22:37:20,943 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1431892621101_0005_01_000002 is : 143
2015-05-17 22:37:21,116 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7030 for container-id container_1431892621101_0005_01_000002: 0B of 1 GB physical memory used; 0B of 2.1 GB virtual memory used
2015-05-17 22:37:21,721 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000002 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL
2015-05-17 22:37:21,749 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduser   OPERATION=Container Finished - Killed   TARGET=ContainerImpl    RESULT=SUCCESS  APPID=application_1431892621101_0005    CONTAINERID=container_1431892621101_0005_01_000002
2015-05-17 22:37:21,749 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000002 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE
2015-05-17 22:37:21,750 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1431892621101_0005_01_000002 from application application_1431892621101_0005
2015-05-17 22:37:21,769 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1431892621101_0005
2015-05-17 22:37:21,914 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005/container_1431892621101_0005_01_000002
2015-05-17 22:37:22,521 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1431892621101_0005_000001 (auth:SIMPLE)
2015-05-17 22:37:22,637 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1431892621101_0005_01_000003 by user hduser
2015-05-17 22:37:22,639 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduser   IP=192.168.1.11 OPERATION=Start Container Request   TARGET=ContainerManageImpl  RESULT=SUCCESS  APPID=application_1431892621101_0005    CONTAINERID=container_1431892621101_0005_01_000003
2015-05-17 22:37:22,639 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_1431892621101_0005_01_000003 to application application_1431892621101_0005
2015-05-17 22:37:22,708 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000003 transitioned from NEW to LOCALIZING
2015-05-17 22:37:22,709 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1431892621101_0005
2015-05-17 22:37:22,709 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_INIT for appId application_1431892621101_0005
2015-05-17 22:37:22,709 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got APPLICATION_INIT for service mapreduce_shuffle
2015-05-17 22:37:22,710 INFO org.apache.hadoop.mapred.ShuffleHandler: Added token for job_1431892621101_0005
2015-05-17 22:37:22,712 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000003 transitioned from LOCALIZING to LOCALIZED
2015-05-17 22:37:23,245 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000003 transitioned from LOCALIZED to RUNNING
2015-05-17 22:37:23,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005/container_1431892621101_0005_01_000003/default_container_executor.sh]
2015-05-17 22:37:24,119 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1431892621101_0005_01_000003
2015-05-17 22:37:24,119 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1431892621101_0005_01_000002
2015-05-17 22:37:25,044 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1431892621101_0005_01_000002]
2015-05-17 22:37:27,019 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 27.4 MB of 1 GB physical memory used; 1.7 GB of 2.1 GB virtual memory used
2015-05-17 22:37:32,749 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 43.8 MB of 1 GB physical memory used; 1.7 GB of 2.1 GB virtual memory used
2015-05-17 22:37:39,417 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 75 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used
2015-05-17 22:37:46,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 104.5 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used
2015-05-17 22:37:48,835 INFO org.apache.hadoop.mapred.ShuffleHandler: Setting connection close header...
2015-05-17 22:37:52,871 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 147.5 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used
2015-05-17 22:37:56,163 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 151.8 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used
2015-05-17 22:38:33,363 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 157.2 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used
2015-05-17 22:38:40,907 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 157.1 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used
2015-05-17 22:38:57,551 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 161.5 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used
2015-05-17 22:39:02,314 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 164.0 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used
2015-05-17 22:39:07,150 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 164.0 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used
2015-05-17 22:39:10,542 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 164.0 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used
2015-05-17 22:39:13,957 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 161.4 MB of 1 GB physical memory used; 1.8 GB of 2.1 GB virtual memory used
2015-05-17 22:39:16,564 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1431892621101_0005_000001 (auth:SIMPLE)
2015-05-17 22:39:16,794 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1431892621101_0005_01_000003
2015-05-17 22:39:16,795 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000003 transitioned from RUNNING to KILLING
2015-05-17 22:39:16,795 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1431892621101_0005_01_000003
2015-05-17 22:39:16,796 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduser   IP=192.168.1.11 OPERATION=Stop Container Request    TARGET=ContainerManageImpl  RESULT=SUCCESS  APPID=application_1431892621101_0005    CONTAINERID=container_1431892621101_0005_01_000003
2015-05-17 22:39:18,989 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1431892621101_0005_01_000003 is : 143
2015-05-17 22:39:19,847 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000003 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL
2015-05-17 22:39:19,852 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduser   OPERATION=Container Finished - Killed   TARGET=ContainerImpl    RESULT=SUCCESS  APPID=application_1431892621101_0005    CONTAINERID=container_1431892621101_0005_01_000003
2015-05-17 22:39:19,852 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1431892621101_0005_01_000003 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE
2015-05-17 22:39:19,852 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1431892621101_0005_01_000003 from application application_1431892621101_0005
2015-05-17 22:39:19,852 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1431892621101_0005
2015-05-17 22:39:19,954 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005/container_1431892621101_0005_01_000003
2015-05-17 22:39:19,983 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 7093 for container-id container_1431892621101_0005_01_000003: 0B of 1 GB physical memory used; 0B of 2.1 GB virtual memory used
2015-05-17 22:39:22,984 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1431892621101_0005_01_000003
2015-05-17 22:39:29,850 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1431892621101_0005_01_000003]
2015-05-17 22:39:30,206 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1431892621101_0005 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP
2015-05-17 22:39:30,264 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1431892621101_0005
2015-05-17 22:39:30,272 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1431892621101_0005
2015-05-17 22:39:30,289 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1431892621101_0005 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2015-05-17 22:39:30,289 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1431892621101_0005, with delay of 10800 seconds

我希望我能找到例如 nodemanager 示例，但我没有。
我如何知道在我的代码中执行的从机的数量？提前谢谢。

hadoopmapreducecountslaveword

1条答案

按热度按时间
bq9c1y66

bq9c1y661#

数据节点日志：

enter code here
2015-05-17 22:15:53,749 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741854_1030 src: /192.168.1.10:53005 dest: /192.168.1.10:50010
2015-05-17 22:16:02,962 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 329ms (threshold=300ms)
2015-05-17 22:16:19,755 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 303ms (threshold=300ms)
2015-05-17 22:17:00,801 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53005, dest: /192.168.1.10:50010, bytes: 57614344, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1317607313_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741854_1030, duration: 58914540985
2015-05-17 22:17:00,823 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741854_1030, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2015-05-17 22:17:07,237 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741854_1030 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741854 for deletion
2015-05-17 22:17:07,325 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741854_1030 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741854
2015-05-17 22:24:31,972 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741855_1031 src: /192.168.1.10:53023 dest: /192.168.1.10:50010
2015-05-17 22:25:29,693 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53023, dest: /192.168.1.10:50010, bytes: 57614412, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_553195046_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741855_1031, duration: 57499963690
2015-05-17 22:25:29,712 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741855_1031, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2015-05-17 22:25:30,882 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741856_1032 src: /192.168.1.10:53028 dest: /192.168.1.10:50010
2015-05-17 22:25:31,043 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53028, dest: /192.168.1.10:50010, bytes: 131, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_553195046_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741856_1032, duration: 26027070
2015-05-17 22:25:31,045 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741856_1032, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2015-05-17 22:25:31,162 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741857_1033 src: /192.168.1.10:53030 dest: /192.168.1.10:50010
2015-05-17 22:25:31,285 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53030, dest: /192.168.1.10:50010, bytes: 13, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_553195046_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741857_1033, duration: 62104463
2015-05-17 22:25:31,285 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741857_1033, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2015-05-17 22:25:32,384 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741858_1034 src: /192.168.1.10:53032 dest: /192.168.1.10:50010
2015-05-17 22:25:32,642 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53032, dest: /192.168.1.10:50010, bytes: 88165, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_553195046_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741858_1034, duration: 198784861
2015-05-17 22:25:32,642 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741858_1034, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2015-05-17 22:26:03,671 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741859_1035 src: /192.168.1.11:44921 dest: /192.168.1.10:50010
2015-05-17 22:26:03,870 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44921, dest: /192.168.1.10:50010, bytes: 104903, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_589847681_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741859_1035, duration: 141461930
2015-05-17 22:26:03,871 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741859_1035, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2015-05-17 22:26:11,276 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741860_1036 src: /192.168.1.11:44928 dest: /192.168.1.10:50010
2015-05-17 22:26:18,166 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44928, dest: /192.168.1.10:50010, bytes: 33241, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_589847681_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741860_1036, duration: 6865081449
2015-05-17 22:26:18,166 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741860_1036, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2015-05-17 22:26:18,320 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741861_1037 src: /192.168.1.11:44935 dest: /192.168.1.10:50010
2015-05-17 22:26:18,374 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44935, dest: /192.168.1.10:50010, bytes: 347, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_589847681_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741861_1037, duration: 50009828
2015-05-17 22:26:18,375 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741861_1037, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2015-05-17 22:26:18,841 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741862_1038 src: /192.168.1.11:44938 dest: /192.168.1.10:50010
2015-05-17 22:26:18,900 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44938, dest: /192.168.1.10:50010, bytes: 33241, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_589847681_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741862_1038, duration: 53415933
2015-05-17 22:26:18,900 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741862_1038, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2015-05-17 22:26:19,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741863_1039 src: /192.168.1.11:44940 dest: /192.168.1.10:50010
2015-05-17 22:26:19,329 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44940, dest: /192.168.1.10:50010, bytes: 104903, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_589847681_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741863_1039, duration: 157935747
2015-05-17 22:26:19,331 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741863_1039, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2015-05-17 22:26:23,457 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741856_1032 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741856 for deletion
2015-05-17 22:26:23,515 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741856_1032 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741856
2015-05-17 22:26:23,515 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741857_1033 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741857 for deletion
2015-05-17 22:26:23,603 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741857_1033 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741857
2015-05-17 22:26:23,677 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741858_1034 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741858 for deletion
2015-05-17 22:26:23,678 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741859_1035 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741859 for deletion
2015-05-17 22:26:23,680 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741860_1036 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741860 for deletion
2015-05-17 22:26:23,680 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741858_1034 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741858
2015-05-17 22:26:23,681 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741859_1035 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741859
2015-05-17 22:26:23,682 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741855_1031 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741855 for deletion
2015-05-17 22:26:23,683 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741860_1036 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741860
2015-05-17 22:26:23,688 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741855_1031 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741855
2015-05-17 22:31:18,177 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741864_1040 src: /192.168.1.10:53045 dest: /192.168.1.10:50010
2015-05-17 22:31:18,411 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53045, dest: /192.168.1.10:50010, bytes: 2, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-572045485_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741864_1040, duration: 120795963
2015-05-17 22:31:18,412 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741864_1040, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2015-05-17 22:33:03,737 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741865_1041 src: /192.168.1.10:53053 dest: /192.168.1.10:50010
2015-05-17 22:33:03,895 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53053, dest: /192.168.1.10:50010, bytes: 2, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-947295319_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741865_1041, duration: 138647149
2015-05-17 22:33:03,895 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741865_1041, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2015-05-17 22:34:37,656 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741866_1042 src: /192.168.1.10:53059 dest: /192.168.1.10:50010
2015-05-17 22:35:35,213 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53059, dest: /192.168.1.10:50010, bytes: 57614412, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1837746633_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741866_1042, duration: 57228181985
2015-05-17 22:35:35,282 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741866_1042, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2015-05-17 22:35:36,863 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741867_1043 src: /192.168.1.10:53064 dest: /192.168.1.10:50010
2015-05-17 22:35:36,943 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53064, dest: /192.168.1.10:50010, bytes: 126, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1837746633_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741867_1043, duration: 24430351
2015-05-17 22:35:36,943 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741867_1043, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2015-05-17 22:35:37,104 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741868_1044 src: /192.168.1.10:53066 dest: /192.168.1.10:50010
2015-05-17 22:35:37,266 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53066, dest: /192.168.1.10:50010, bytes: 26, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1837746633_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741868_1044, duration: 145913793
2015-05-17 22:35:37,267 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741868_1044, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2015-05-17 22:35:38,798 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741869_1045 src: /192.168.1.10:53068 dest: /192.168.1.10:50010
2015-05-17 22:35:39,276 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53068, dest: /192.168.1.10:50010, bytes: 88156, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1837746633_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741869_1045, duration: 190623489
2015-05-17 22:35:39,278 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741869_1045, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2015-05-17 22:35:48,707 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741870_1046 src: /192.168.1.11:44948 dest: /192.168.1.10:50010
2015-05-17 22:35:48,956 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44948, dest: /192.168.1.10:50010, bytes: 104894, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-824185019_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741870_1046, duration: 178516552
2015-05-17 22:35:48,958 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741870_1046, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2015-05-17 22:37:19,135 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741871_1047 src: /192.168.1.11:44955 dest: /192.168.1.10:50010
2015-05-17 22:37:51,956 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741872_1048 src: /192.168.1.10:53088 dest: /192.168.1.10:50010
2015-05-17 22:38:40,341 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 42910ms (threshold=300ms)
2015-05-17 22:38:40,842 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 493ms (threshold=300ms)
2015-05-17 22:38:41,809 WARN org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 42670ms
GC pool 'PS Scavenge' had collection(s): count=1 time=42530ms
2015-05-17 22:38:41,904 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 495ms (threshold=300ms)
2015-05-17 22:38:50,682 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:382ms (threshold=300ms)
2015-05-17 22:38:58,498 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:346ms (threshold=300ms)
2015-05-17 22:39:03,675 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 407ms (threshold=300ms)
2015-05-17 22:39:15,336 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.10:53088, dest: /192.168.1.10:50010, bytes: 37785075, op: HDFS_WRITE, cliID: DFSClient_attempt_1431892621101_0005_r_000000_0_-581669036_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741872_1048, duration: 82970475818
2015-05-17 22:39:15,346 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741872_1048, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2015-05-17 22:39:16,181 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-567696754-192.168.1.10-1431878010808:blk_1073741852_1028
2015-05-17 22:39:16,236 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-567696754-192.168.1.10-1431878010808:blk_1073741851_1027
2015-05-17 22:39:16,313 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-567696754-192.168.1.10-1431878010808:blk_1073741853_1029
2015-05-17 22:39:19,173 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44955, dest: /192.168.1.10:50010, bytes: 33639, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-824185019_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741871_1047, duration: 119891168848
2015-05-17 22:39:19,174 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741871_1047, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2015-05-17 22:39:20,113 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741873_1049 src: /192.168.1.11:44963 dest: /192.168.1.10:50010
2015-05-17 22:39:20,135 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44963, dest: /192.168.1.10:50010, bytes: 350, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-824185019_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741873_1049, duration: 20566418
2015-05-17 22:39:20,137 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741873_1049, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2015-05-17 22:39:20,455 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741874_1050 src: /192.168.1.11:44966 dest: /192.168.1.10:50010
2015-05-17 22:39:20,528 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44966, dest: /192.168.1.10:50010, bytes: 33639, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-824185019_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741874_1050, duration: 68722185
2015-05-17 22:39:20,528 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741874_1050, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2015-05-17 22:39:20,655 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-567696754-192.168.1.10-1431878010808:blk_1073741875_1051 src: /192.168.1.11:44968 dest: /192.168.1.10:50010
2015-05-17 22:39:20,779 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.1.11:44968, dest: /192.168.1.10:50010, bytes: 104894, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-824185019_1, offset: 0, srvID: 555ac87e-c981-4e93-866a-42f33be6ff6e, blockid: BP-567696754-192.168.1.10-1431878010808:blk_1073741875_1051, duration: 117317991
2015-05-17 22:39:20,850 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-567696754-192.168.1.10-1431878010808:blk_1073741875_1051, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2015-05-17 22:39:21,674 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-567696754-192.168.1.10-1431878010808:blk_1073741870_1046
2015-05-17 22:39:21,790 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-567696754-192.168.1.10-1431878010808:blk_1073741863_1039
2015-05-17 22:39:21,874 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-567696754-192.168.1.10-1431878010808:blk_1073741867_1043
2015-05-17 22:39:25,480 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741866_1042 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741866 for deletion
2015-05-17 22:39:25,481 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741867_1043 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741867 for deletion
2015-05-17 22:39:25,481 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741868_1044 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741868 for deletion
2015-05-17 22:39:25,481 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741869_1045 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741869 for deletion
2015-05-17 22:39:25,482 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741870_1046 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741870 for deletion
2015-05-17 22:39:25,482 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741871_1047 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741871 for deletion
2015-05-17 22:39:25,542 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741866_1042 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741866
2015-05-17 22:39:25,542 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741867_1043 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741867
2015-05-17 22:39:25,542 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741868_1044 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741868
2015-05-17 22:39:25,554 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741869_1045 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741869
2015-05-17 22:39:25,554 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741870_1046 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_1073741870
2015-05-17 22:39:25,554 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-567696754-192.168.1.10-1431878010808 blk_1073741871_1047 file /usr/local/tmp/hdfs/datanode/current/BP-567696754-192.168.1.10-1431878010808/current/finalized/subdir0/subdir0/blk_107374187
赞(0）分享  回复(0）举报  2021-06-03
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com