java/scala远程hdfs使用

7fhtutme  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(304)

我正在尝试连接到远程hdfs群集。我已经阅读了一些文档和入门指南,但没有找到一个最好的解决方案。情境:我在-something.com上有hdfs。我可以通过ssh连接到它,一切正常。
但我要做的是,把文件从它拿到我的本地机器上。
我所做的:
我已经在conf文件夹中创建了core-site.xml(我正在创建play!应用程序)。在这里,我将fs.default.name配置更改为hdfs://-something.com:8020(不确定端口)。然后我试着做一个简单的测试:

val conf = new Configuration()
conf.addResource(new Path("conf/core-site.xml"))
val fs = FileSystem.get(conf)
val status = fs.listStatus(new Path("/data/"))

我发现了一些错误:

13:56:09.012 [specs2.DefaultExecutionStrategy1] WARN  org.apache.hadoop.conf.Configuration - conf/core-site.xml:a attempt to override final parameter: fs.trash.interval;  Ignoring.
13:56:09.012 [specs2.DefaultExecutionStrategy1] WARN  org.apache.hadoop.conf.Configuration - conf/core-site.xml:a attempt to override final parameter: hadoop.tmp.dir;  Ignoring.
13:56:09.013 [specs2.DefaultExecutionStrategy1] WARN  org.apache.hadoop.conf.Configuration - conf/core-site.xml:a attempt to override final parameter: fs.checkpoint.dir;  Ignoring.
13:56:09.022 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.fs.FileSystem - Creating filesystem for hdfs://xxx-something.com:8021
13:56:09.059 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.conf.Configuration - java.io.IOException: config()
    at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:226)
    at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:213)
    at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:53)
    at org.apache.hadoop.net.NetUtils.<clinit>(NetUtils.java:62)

提前谢谢!
更新:可能是端口错误。现在我把它设置为22,我仍然得到同样的错误,但3次之后它确实会说:

14:01:01.877 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.ipc.Client - Connecting to xxx-something.com/someIp:22
14:01:02.187 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.ipc.Client - IPC Client (47) connection to xxx-something.com/someIp:22 from britva sending #0
14:01:02.188 [IPC Client (47) connection to xxx-something.com/someIp:22 from britva] DEBUG org.apache.hadoop.ipc.Client - IPC Client (47) connection to xxx-something.com/someIp:22 from britva: starting, having connections 1
14:01:02.422 [IPC Client (47) connection to xxx-something.com/someIp:22 from britva] DEBUG org.apache.hadoop.ipc.Client - IPC Client (47) connection to xxx-something.com/someIp:22 from britva got value #1397966893

之后:

Call to xxx-something.com/someIp:22 failed on local exception: java.io.EOFException
java.io.IOException: Call to xxx-something.com/someIp:22 failed on local exception: java.io.EOFException
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
    at org.apache.hadoop.ipc.Client.call(Client.java:1071)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
    at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:122)
    at HdfsSpec$$anonfun$1$$anonfun$apply$3.apply(HdfsSpec.scala:33)
    at HdfsSpec$$anonfun$1$$anonfun$apply$3.apply(HdfsSpec.scala:17)
    at testingSupport.specs2.MyNotifierRunner$$anon$2$$anon$1.executeBody(MyNotifierRunner.scala:16)
    at testingSupport.specs2.MyNotifierRunner$$anon$2$$anon$1.execute(MyNotifierRunner.scala:16)
Caused by: java.io.EOFException
    at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:807)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:745)

这是什么意思?

nhn9ugyo

nhn9ugyo1#

您需要在运行name节点(hdfs master)的服务器上的$hadoop\u home/conf/core-site.xml中找到fs.default.name属性,以获得正确的端口。可能是8020,也可能是别的什么。你应该用这个。确保您和服务器之间没有防火墙,不允许在端口上进行连接。

相关问题