hadoop—使用AmazonS3作为输入、输出，并将中间结果存储在emr map reduce作业中

yxyvkwin 于 2021-06-03 发布在 Hadoop

关注(0)|答案(2)|浏览(335)

我正在尝试使用amazons3存储与emr。但是，当我当前运行我的代码时，会出现多个错误，比如

java.lang.IllegalArgumentException: This file system object (hdfs://10.254.37.109:9000)    does not support access to the request path 's3n://energydata/input/centers_200_10k_norm.csv' You possibly called FileSystem.get(conf) when you should have called FileSystem.get(uri, conf) to obtain a file system supporting your path.
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:384)
at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:129)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:154)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:429)
at edu.stanford.cs246.hw2.KMeans$CentroidMapper.setup(KMeans.java:112)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:375)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

在main中，我像这样设置输入和输出路径，并将s3n://energydata/input/centers\u 200\u 10k\u norm.csv放入配置文件中，我在mapper和reducer中检索到：

FileSystem fs = FileSystem.get(conf);
conf.set(CFILE, inPath); //inPath in this case is s3n://energydata/input/centers_200_10k_norm.csv
FileInputFormat.addInputPath(job, new Path(inputDir));
FileOutputFormat.setOutputPath(job, new Path(outputDir));

上面的错误出现在Map器和reducer中的具体示例，我尝试访问cfile（s3n://energydata/input/centers\u 200\u 10k\u norm.csv）。我就是这样试图找到路径的：

FileSystem fs = FileSystem.get(context.getConfiguration());
Path cFile = new Path(context.getConfiguration().get(CFILE));
DataInputStream d = new DataInputStream(fs.open(cFile));  ---->Error

s3n://energydata/input/centers\u 200\u 10k\u norm.csv是程序的输入参数之一，当我启动emr作业时，我将输入和输出目录指定为s3n://energydata/input和s3n://energydata/output
我试着按照hdfs文件路径中的建议去做，但还是得到了错误。任何帮助都将不胜感激。
谢谢！

hadoop mapreduce amazon-emr amazon-s3 amazon-web-services

来源：https://stackoverflow.com/questions/16219278/using-amazon-s3-as-input-output-and-to-store-intermediate-results-in-emr-map-red

2条答案

按热度按时间

8hhllhi21#

谢谢。实际上，我使用以下代码修复了它：

String uriStr =  "s3n://energydata/centroid/";
URI uri = URI.create(uriStr);
FileSystem fs = FileSystem.get(uri, context.getConfiguration());    
Path cFile = new Path(context.getConfiguration().get(CFILE));  
DataInputStream d = new DataInputStream(fs.open(cFile));

赞(0）回复(0）举报 2021-06-03

nukf8bse2#

请尝试：

Path cFile = new Path(context.getConfiguration().get(CFILE));
FileSystem fs = cFile.getFileSystem(context.getConfiguration());
DataInputStream d = new DataInputStream(fs.open(cFile));

赞(0）回复(0）举报 2021-06-03

我来回答

hadoop—使用AmazonS3作为输入、输出，并将中间结果存储在emr map reduce作业中

2条答案

相关问题

热门标签

最新问答