请查看附件中的代码片段。我用这段代码将文件从hdfs下载到我的本地文件系统-
Configuration conf = new Configuration();
FileSystem hdfsFileSystem = FileSystem.get(conf);
Path local = new Path(destinationPath);
Path hdfs = new Path(sourcePath);
String fileName = hdfs.getName();
if (hdfsFileSystem.exists(hdfs))
{
hdfsFileSystem.copyToLocalFile(false, hdfs, local, true);
logger.info("File " + fileName + " copied to local machine on location: " + destinationPath);
}
else
{
logger.error("File " + fileName + " does not exist on HDFS on location: " + sourcePath);
}
运行此命令将产生以下输出-
Exception in thread "main" java.lang.UnsupportedOperationException: Not implemented by the DistributedFileSystem FileSystem implementation
at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:217)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2624)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
at com.flipkart.ads.importer.HdfsToLocalImporter.importFile(HdfsToLocalImporter.java:35)
at com.flipkart.ads.importer.HdfsToLocalImporter.main(HdfsToLocalImporter.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
Process finished with exit code 1
我错过了什么?
4条答案
按热度按时间t8e9dugd1#
断然的
我没想到。这是一个不匹配的问题。hadoop代码和hadoop commons提供了相同的jar,我已经包含了这两个依赖项。
wlwcrazw2#
你失踪了
conf.addResources
. 您已初始化Configuration
对象,但它仍然是空的。查看这里的文档
编辑:尝试以下操作:
conf.addResource(new Path("<absolute file path>"))
rqmkfv5c3#
检查您的构建路径。如果您在那里看到hadoop core*****.jar,请删除它。这是不需要的。这将解决问题
z9gpfhce4#
您只需要将配置对象指向hdfs服务器。使用
conf.set("fs.defaultFS", "hdfs://urlhere:8020");