hadoop jar命令错误

hmtdttj4 于 2021-06-03 发布在 Hadoop

关注(0)|答案(4)|浏览(451)

在hdfs上执行jar file命令时，出现如下错误


# hadoop jar WordCountNew.jar WordCountNew /MRInput57/Input-Big.txt /MROutput57
15/11/06 19:46:32 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
15/11/06 19:46:32 INFO mapred.JobClient: Cleaning up the staging area hdfs://localhost:8020/var/lib/hadoop-0.20/cache/mapred/mapred/staging/root/.staging/job_201511061734_0003
15/11/06 19:46:32 ERROR security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /MRInput57/Input-Big.txt already exists
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /MRInput57/Input-Big.txt already exists
    at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:132)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:921)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:882)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:882)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:526)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:556)
    at MapReduce.WordCountNew.main(WordCountNew.java:114)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
My Driver class Program is as below
    public static void main(String[] args) throws IOException, Exception {
        // Configutation details w. r. t. Job, Jar file
        Configuration conf = new Configuration();
        Job job = new Job(conf, "WORDCOUNTJOB");
        // Setting Driver class
        job.setJarByClass(MapReduceWordCount.class);
        // Setting the Mapper class
        job.setMapperClass(TokenizerMapper.class);
        // Setting the Combiner class
        job.setCombinerClass(IntSumReducer.class);
        // Setting the Reducer class
        job.setReducerClass(IntSumReducer.class);
        // Setting the Output Key class
        job.setOutputKeyClass(Text.class);
        // Setting the Output value class
        job.setOutputValueClass(IntWritable.class);
        // Adding the Input path
        FileInputFormat.addInputPath(job, new Path(args[0]));
        // Setting the output path
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        // System exit strategy
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }

有人能纠正我代码中的问题吗？
向普拉纳夫问好

hadoop

来源：https://stackoverflow.com/questions/33579383/hadoop-jar-command-error

4条答案

按热度按时间

rsaldnfx1#

您需要检查输出目录是否已经存在，如果已经存在，请将其删除。mapreduce无法（或不会）将文件写入存在的目录。它需要创建目录才能确定。
添加以下内容：

Path outPath = new Path(args[1]);
FileSystem dfs = FileSystem.get(outPath.toUri(), conf);
if (dfs.exists(outPath)) {
    dfs.delete(outPath, true);
}

赞(0）回复(0）举报 2021-06-03

cfh9epnr2#

您试图创建以存储输出的输出目录已存在。因此请尝试删除以前同名的目录或更改输出目录的名称。

赞(0）回复(0）举报 2021-06-03

xkftehaa3#

正如其他人所注意到的，您得到这个错误是因为输出目录已经存在，很可能是因为您以前尝试过执行这个作业。
您可以在运行作业之前删除现有的输出目录，即：


# hadoop fs -rm -r /MROutput57 && \
hadoop jar WordCountNew.jar WordCountNew /MRInput57/Input-Big.txt /MROutput57

赞(0）回复(0）举报 2021-06-03

qyzbxkaa4#

在执行程序之前，输出目录不应存在。删除现有目录或提供新目录，或删除程序中的输出目录。
我更喜欢在命令提示符下执行程序之前从命令提示符下删除输出目录。
从命令提示符：

hdfs dfs -rm -r <your_output_directory_HDFS_URL>

来自java：

Chris Gerken code is good enough.

赞(0）回复(0）举报 2021-06-03

我来回答

hadoop jar命令错误

4条答案

相关问题

热门标签

最新问答