hdfs无法读取数据(got error,status message opreadblock)

gc0ot86w  于 2021-07-13  发布在  Hadoop
关注(0)|答案(1)|浏览(1460)

我已经做了haddop的升级和降级操作,然后旧数据就不能打开了。你能帮我检查一下这个问题吗
https://hadoop.apache.org/docs/r2.10.1/hadoop-project-dist/hadoop-hdfs/hdfsrollingupgrade.html#namenode_-滚动升级

[root@master current]# hdfs dfs -cat /test/test.csv
21/03/02 14:42:17 WARN hdfs.BlockReaderFactory: I/O error constructing remote block reader.
java.io.IOException: Got error, status message opReadBlock BP-1289313299-192.168.1.26-1533200460191:blk_1073762237_21423 
received exception java.io.IOException: BlockId 1073762237 is not valid., 
for OP_READ_BLOCK, self=/10.10.202.26:47930, remote=/10.10.202.26:50010, 
for file /test/test.csv, for pool BP-1289313299-192.168.1.26-1533200460191 block 1073762237_21423
    at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:140)
    at org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:456)
    at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:424)
    at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:818)
    at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:69
...
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1289313299-192.168.1.26-1533200460191:blk_1073762237_21423 file=/test/test.csv
    at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:983)
    at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:642)
    at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)
    at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)
    at java.io.DataInputStrea

-rw-r--r--   1 root   supergroup        359 2018-08-21 11:19 /test/test.csv

[root@master current]# hdfs fsck /test/
Connecting to namenode via http://master:50070/fsck?ugi=root&path=%2Ftest
FSCK started by root (auth:SIMPLE) from /10.10.202.26 for path /test at Tue Mar 02 15:05:23 CST 2021
...................................................Status: HEALTHY
 Total size:    6720241197 B
 Total dirs:    17
 Total files:   51
 Total symlinks:        0
 Total blocks (validated):  85 (avg. block size 79061661 B)
 Minimally replicated blocks:   85 (100.0 %)
 Over-replicated blocks:    0

文件下载失败,请在此处输入图像说明

yb3bgrhw

yb3bgrhw1#

确保时间正确并在所有服务器上同步。确保datanode文件在linux文件系统上具有正确的权限。
尝试:

hadoop fsck /test/ -files -blocks
hadoop fsck /test/  -list-corruptfileblocks

在某些情况下,更改hdfs-site.xml文件:

<property> 
<name>dfs.client.use.datanode.hostname</name> 
<value>true</value> 
</property> ​

帮助解决了问题。

相关问题