我已经在8节点集群上部署了hadoop（0.20.203.0rc1）。在上传文件到hdfs之后，我只在一个节点上得到了这个文件，而不是均匀地分布在所有节点上。有什么问题吗？
$HADOOP_HOME/bin/hadoop dfs -copyFromLocal ../data/rmat-20.0 /user/frolo/input/rmat-20.0$HADOOP_HOME/bin/hadoop dfs -stat "%b %o %r %n" /user/frolo/input/rmat-*1220222968 67108864 1 rmat-20.0$HADOOP_HOME/bin/hadoop dfsadmin -reportConfigured Capacity: 2536563998720 (2.31 TB)Present Capacity: 1642543419392 (1.49 TB)DFS Remaining: 1641312030720 (1.49 TB)DFS Used: 1231388672 (1.15 GB)DFS Used%: 0.07%Under replicated blocks: 0Blocks with corrupt replicas: 0Missing blocks: 0-------------------------------------------------Datanodes available: 8 (8 total, 0 dead)Name: 10.10.1.15:50010Decommission Status : NormalConfigured Capacity: 317070499840 (295.29 GB)DFS Used: 24576 (24 KB)Non DFS Used: 131536928768 (122.5 GB)DFS Remaining: 185533546496(172.79 GB)DFS Used%: 0%DFS Remaining%: 58.51%Last contact: Fri Feb 07 12:10:27 MSK 2014Name: 10.10.1.13:50010Decommission Status : NormalConfigured Capacity: 317070499840 (295.29 GB)DFS Used: 24576 (24 KB)Non DFS Used: 131533377536 (122.5 GB)DFS Remaining: 185537097728(172.79 GB)DFS Used%: 0%DFS Remaining%: 58.52%Last contact: Fri Feb 07 12:10:27 MSK 2014Name: 10.10.1.17:50010Decommission Status : NormalConfigured Capacity: 317070499840 (295.29 GB)DFS Used: 24576 (24 KB)Non DFS Used: 120023924736 (111.78 GB)DFS Remaining: 197046550528(183.51 GB)DFS Used%: 0%DFS Remaining%: 62.15%Last contact: Fri Feb 07 12:10:27 MSK 2014Name: 10.10.1.18:50010Decommission Status : NormalConfigured Capacity: 317070499840 (295.29 GB)DFS Used: 24576 (24 KB)Non DFS Used: 78510628864 (73.12 GB)DFS Remaining: 238559846400(222.18 GB)DFS Used%: 0%DFS Remaining%: 75.24%Last contact: Fri Feb 07 12:10:24 MSK 2014Name: 10.10.1.14:50010Decommission Status : NormalConfigured Capacity: 317070499840 (295.29 GB)DFS Used: 24576 (24 KB)Non DFS Used: 131537530880 (122.5 GB)DFS Remaining: 185532944384(172.79 GB)DFS Used%: 0%DFS Remaining%: 58.51%Last contact: Fri Feb 07 12:10:27 MSK 2014Name: 10.10.1.11:50010Decommission Status : NormalConfigured Capacity: 317070499840 (295.29 GB)DFS Used: 1231216640 (1.15 GB)Non DFS Used: 84698116096 (78.88 GB)DFS Remaining: 231141167104(215.27 GB)DFS Used%: 0.39%DFS Remaining%: 72.9%Last contact: Fri Feb 07 12:10:24 MSK 2014Name: 10.10.1.16:50010Decommission Status : NormalConfigured Capacity: 317070499840 (295.29 GB)DFS Used: 24576 (24 KB)Non DFS Used: 131537494016 (122.5 GB)DFS Remaining: 185532981248(172.79 GB)DFS Used%: 0%DFS Remaining%: 58.51%Last contact: Fri Feb 07 12:10:27 MSK 2014Name: 10.10.1.12:50010Decommission Status : NormalConfigured Capacity: 317070499840 (295.29 GB)DFS Used: 24576 (24 KB)Non DFS Used: 84642578432 (78.83 GB)DFS Remaining: 232427896832(216.47 GB)DFS Used%: 0%DFS Remaining%: 73.3%Last contact: Fri Feb 07 12:10:27 MSK 2014
1条答案
按热度按时间fbcarpbf1#
您的文件已使用复制因子写入
1，你的hadoop fs -stat命令输出。这意味着文件下的块将只存在一个块副本。写入的默认复制因子由属性控制
dfs.replication低于$HADOOP_HOME/conf/hdfs-site.xml. 如果未指定，则默认为3，但很可能指定了它的重写，其值为1. 将其值更改回3或者完全删除它（调用默认值）将使所有新的文件写入都使用3默认情况下为复制副本。您还可以使用
-D属性支持的属性传递方法hadoop fs实用程序，例如：hadoop fs -Ddfs.replication=3 -copyFromLocal ../data/rmat-20.0 /user/frolo/input/rmat-20.0您可以使用hadoop fs -setrep实用程序，例如：hadoop fs -setrep 3 -w /user/frolo/input/rmat-20.0带有HDFS复制因子大于1将显示自动分布在多个节点上。HDFS决不会将一个块的多个副本写入同一块DataNode.