intellij:mapreduce错误:线程“main”0中出现异常:没有这样的文件或目录

voj3qocg  于 2021-05-27  发布在  Hadoop
关注(0)|答案(1)|浏览(403)

我一直在开发一个map reduce程序,它在hadoop虚拟机hdfs环境下运行良好。但是当我用intellij在windows中尝试同样的程序时,我得到了这个错误。
wordcount.class//将此用作测试其是否工作的示例程序。

  1. public class WordCount {
  2. public static class TokenizerMapper
  3. extends Mapper<Object, Text, Text, IntWritable> {
  4. private final static IntWritable one = new IntWritable(1);
  5. private Text word = new Text();
  6. public void map(Object key, Text value, Context context
  7. ) throws IOException, InterruptedException {
  8. StringTokenizer itr = new StringTokenizer(value.toString());
  9. while (itr.hasMoreTokens()) {
  10. word.set(itr.nextToken());
  11. context.write(word, one);
  12. }
  13. }
  14. }
  15. public static class IntSumReducer
  16. extends Reducer<Text, IntWritable, Text, IntWritable> {
  17. private IntWritable result = new IntWritable();
  18. public void reduce(Text key, Iterable<IntWritable> values,
  19. Context context
  20. ) throws IOException, InterruptedException {
  21. int sum = 0;
  22. for (IntWritable val : values) {
  23. sum += val.get();
  24. }
  25. result.set(sum);
  26. context.write(key, result);
  27. }
  28. }
  29. public static void main(String[] args) throws Exception {
  30. Configuration conf = new Configuration();
  31. Job job = Job.getInstance(conf, "word count");
  32. job.setJarByClass(WordCount.class);
  33. job.setMapperClass(TokenizerMapper.class);
  34. job.setCombinerClass(IntSumReducer.class);
  35. job.setReducerClass(IntSumReducer.class);
  36. job.setOutputKeyClass(Text.class);
  37. job.setOutputValueClass(IntWritable.class);
  38. FileInputFormat.addInputPath(job, new Path(args[0]));
  39. FileOutputFormat.setOutputPath(job, new Path(args[1]));
  40. System.exit(job.waitForCompletion(true) ? 0 : 1);
  41. }
  42. }

intellij错误日志

  1. 2019-12-12 21:42:04,139 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1181)) - session.id is deprecated. Instead, use dfs.metrics.session-id
  2. 2019-12-12 21:42:04,144 INFO [main] jvm.JvmMetrics (JvmMetrics.java:init(79)) - Initializing JVM Metrics with processName=JobTracker, sessionId=
  3. 2019-12-12 21:42:08,029 WARN [main] mapreduce.JobResourceUploader (JobResourceUploader.java:uploadFiles(64)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
  4. 2019-12-12 21:42:08,089 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(251)) - Cleaning up the staging area file:/tmp/hadoop/mapred/staging/Abhishek1224360463/.staging/job_local1224360463_0001
  5. Exception in thread "main" 0: No such file or directory
  6. at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:236)
  7. at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:767)
  8. at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:506)
  9. at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:487)
  10. at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:503)
  11. at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:619)
  12. at org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:94)
  13. at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:97)
  14. at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:192)
  15. at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
  16. at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
  17. at java.security.AccessController.doPrivileged(Native Method)
  18. at javax.security.auth.Subject.doAs(Subject.java:422)
  19. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
  20. at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
  21. at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1359)
  22. at WordCount.main(WordCount.java:59)

我通过将目录名作为参数发送到主类(即编辑运行配置并传递包含文本文件的目录名)来提供输入(输入参数:输入输出)我在项目根文件夹下有输入目录。

w7t8yxp5

w7t8yxp51#

在管理员模式下运行intellij就成功了。但这很奇怪。如果有人向我解释这件事,我将不胜感激。

相关问题