hdfs get在特定客户机节点上失败

eblbsuwk  于 2021-06-04  发布在  Hadoop
关注(0)|答案(2)|浏览(417)

我对hdfs有个奇怪的问题。虽然对现有文件的get操作在访问hdfs群集的所有客户端上都像一个符咒一样工作,但在一个客户端上失败:
工作主持人:

[user@host1]$ hadoop fs -ls /path/to/file.csv
found 1 items
-rw-r--r--   3 compute supergroup       1628 2013-12-10 12:22 /path/to/file.csv
[user@host1]$ hadoop fs -get /path/to/file.csv /tmp/test.csv
[user@host1]$  cat /tmp/test.csv 
48991,24768,2013-12-10 00:00:00,1,0.0001,0.0001

非工作主机:

[user@host2]$ hadoop fs -ls /path/to/file.csv
Found 1 items
-rw-r--r--   3 compute supergroup       1628 2013-12-10 12:22 /path/to/file.csv
[user@host2]$ hadoop fs -get /path/to/file.csv /tmp/test.csv
get: java.lang.NullPointerException
[user@host2]$  cat /tmp/test.csv 
cat: /tmp/test.csv: No such file or directory

在工作主机上使用java hdfs客户端:

[user@host1]$ java -jar hadoop_get-1.0-SNAPSHOT-jar-with-dependencies.jar hdfs://my.namenode:port  /path/to/file.csv
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
48991,24768,2013-12-10 00:00:00,1,0.0001,0.0001

在非工作主机上使用java hdfs客户端:

[user@host2]$ java -jar hadoop_get-1.0-SNAPSHOT-jar-with-dependencies.jar hdfs://my.namenode:port  /path/to/file.csv
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException

at org.apache.hadoop.ipc.Client.call(Client.java:1225)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:154)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:957)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:947)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:171)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:138)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:131)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1104)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:246)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:79)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:711)
at my.namespace.client.Client.main(Client.java:34)
9lowa7mx

9lowa7mx1#

你在用cdh4吗?从cdh3升级后,我们也遇到了同样的问题。
尝试研究问题主机的反向dns查找名称-我们发现问题主机和没有问题的主机的区别只是在dns解析方面。修好后-一切正常。

vptzau2j

vptzau2j2#

我们通过部署客户端配置、刷新集群和重新启动hdfs解决了这个问题。

相关问题