无法删除hdfs损坏的文件

8ulbf1ek  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(501)

我无法删除hdfs中存在的损坏文件。namenode已运行到安全模式。区块总数为980个,其中978个已上报。当我运行以下命令时,

  1. sudo -u hdfs hdfs dfsadmin -report

生成的报告是,

  1. Safe mode is ON
  2. Configured Capacity: 58531520512 (54.51 GB)
  3. Present Capacity: 35774078976 (33.32 GB)
  4. DFS Remaining: 32374509568 (30.15 GB)
  5. DFS Used: 3399569408 (3.17 GB)
  6. DFS Used%: 9.50%
  7. Under replicated blocks: 0
  8. Blocks with corrupt replicas: 0
  9. Missing blocks: 0
  10. Missing blocks (with replication factor 1): 0
  11. -------------------------------------------------
  12. Live datanodes (1):
  13. Name: 10.0.2.15:50010 (quickstart.cloudera)
  14. Hostname: quickstart.cloudera
  15. Decommission Status : Normal
  16. Configured Capacity: 58531520512 (54.51 GB)
  17. DFS Used: 3399569408 (3.17 GB)
  18. Non DFS Used: 19777388544 (18.42 GB)
  19. DFS Remaining: 32374509568 (30.15 GB)
  20. DFS Used%: 5.81%
  21. DFS Remaining%: 55.31%
  22. Configured Cache Capacity: 0 (0 B)
  23. Cache Used: 0 (0 B)
  24. Cache Remaining: 0 (0 B)
  25. Cache Used%: 100.00%
  26. Cache Remaining%: 0.00%
  27. Xceivers: 2
  28. Last contact: Tue Nov 14 10:39:58 IST 2017

在执行以下命令时,

  1. sudo -u hdfs hdfs fsck /

输出是,

  1. Connecting to namenode via http://quickstart.cloudera:50070/fsck?ugi=hdfs&path=%2F
  2. FSCK started by hdfs (auth:SIMPLE) from /10.0.2.15 for path / at Tue Nov 14 10:41:25 IST 2017
  3. /hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.default.1509701903728: CORRUPT blockpool BP-1914853243-127.0.0.1-1500467607052 block blk_1073743141
  4. /hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.default.1509701903728: MISSING 1 blocks of total size 83 B..
  5. /hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.meta.1509701932269.meta: CORRUPT blockpool BP-1914853243-127.0.0.1-1500467607052 block blk_1073743142
  6. /hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.meta.1509701932269.meta: MISSING 1 blocks of total size 83 B
  7. Status: CORRUPT
  8. Total size: 3368384392 B (Total open files size: 166 B)
  9. Total dirs: 286
  10. Total files: 966
  11. Total symlinks: 0 (Files currently being written: 3)
  12. Total blocks (validated): 980 (avg. block size 3437126 B) (Total open file blocks (not validated): 2)
  13. ********************************
  14. UNDER MIN REPL'D BLOCKS: 2 (0.20408164 %)
  15. dfs.namenode.replication.min: 1
  16. CORRUPT FILES: 2
  17. MISSING BLOCKS: 2
  18. MISSING SIZE: 166 B
  19. CORRUPT BLOCKS: 2
  20. ********************************
  21. Minimally replicated blocks: 978 (99.79592 %)
  22. Over-replicated blocks: 0 (0.0 %)
  23. Under-replicated blocks: 0 (0.0 %)
  24. Mis-replicated blocks: 0 (0.0 %)
  25. Default replication factor: 1
  26. Average block replication: 0.9979592
  27. Corrupt blocks: 2
  28. Missing replicas: 0 (0.0 %)
  29. Number of data-nodes: 1
  30. Number of racks: 1
  31. FSCK ended at Tue Nov 14 10:41:26 IST 2017 in 774 milliseconds
  32. The filesystem under path '/' is CORRUPT

有人能帮忙纠正损坏的块,或者删除它们吗?提前谢谢。

voase2hg

voase2hg1#

据说namenode处于安全模式,首先关闭它。

  1. hdfs dfsadmin -safemode leave

然后执行其中一个命令

  1. hdfs fsck / | egrep -v '^\.+$' | grep -v replica | grep -v Replica

  1. hdfs fsck hdfs://quickstart.cloudera:50070/ | egrep -v '^\.+$' | grep -v replica | grep -v Replica

输出与

  1. /path/to/filename.fileextension: CORRUPT blockpool BP-1016133662-10.29.100.41-1415825958975 block blk_1073904305
  2. /path/to/filename.fileextension: MISSING 1 blocks of total size 15620361 B

在您的情况下,已列出损坏的文件。所以执行下面的命令

  1. hdfs dfs -rm /hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.default.1509701903728
  2. hdfs dfs -rm /hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.meta.1509701932269.meta

不要进入安全模式。继续工作。哎呀!!

展开查看全部

相关问题