java找到接口org.apache.hadoop.mapreduce.taskattemptcontext，但应为类

2vuwiymt  于 2021-06-01  发布在  Hadoop
关注(0)|答案(1)|浏览(393)

我的hadoop应用程序在执行时遇到了一些错误，每当它开始执行map 0%reduce 0%时，就会出现某种错误
17/06/02 16:21:44 info mapreduce.job:任务id:尝试\u 14963966027749 \u 0015 \u m \u0000000，状态：失败错误：找到接口org.apache.hadoop.mapreduce.taskattemptcontext，但应为类
我被困在这里，任何能帮我的人。。

  1. hduser@master:/home/mnh/Desktop$ hadoop jar  13.jar /usr/local/hadoop/input/cars.mp4 /usr/local/hadoop/cars9
  2. 17/06/02 16:07:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  3. 17/06/02 16:07:37 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.137.52:8050
  4. 17/06/02 16:07:38 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
  5. 17/06/02 16:08:35 INFO input.FileInputFormat: Total input paths to process : 1
  6. 17/06/02 16:08:35 INFO mapreduce.JobSubmitter: number of splits:1
  7. 17/06/02 16:08:35 INFO Configuration.deprecation: mapred.task.timeout is deprecated. Instead, use mapreduce.task.timeout
  8. 17/06/02 16:08:35 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1496396027749_0012
  9. 17/06/02 16:08:36 INFO impl.YarnClientImpl: Submitted application application_1496396027749_0012
  10. 17/06/02 16:08:36 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1496396027749_0012/
  11. 17/06/02 16:08:36 INFO mapreduce.Job: Running job: job_1496396027749_0012
  12. 17/06/02 16:08:46 INFO mapreduce.Job: Job job_1496396027749_0012 running in uber mode : false
  13. 17/06/02 16:08:46 INFO mapreduce.Job:  map 0% reduce 0%
  14. 17/06/02 16:08:53 INFO mapreduce.Job: Task Id : attempt_1496396027749_0012_m_000000_0, Status : FAILED
  15. Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
  16. 17/06/02 16:09:00 INFO mapreduce.Job: Task Id : attempt_1496396027749_0012_m_000000_1, Status : FAILED
  17. Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
  18. 17/06/02 16:09:06 INFO mapreduce.Job: Task Id : attempt_1496396027749_0012_m_000000_2, Status : FAILED
  19. Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
  20. 17/06/02 16:09:14 INFO mapreduce.Job:  map 100% reduce 100%
  21. 17/06/02 16:09:15 INFO mapreduce.Job: Job job_1496396027749_0012 failed with state FAILED due to: Task failed task_1496396027749_0012_m_000000
  22. Job failed as tasks failed. failedMaps:1 failedReduces:0
  24. 17/06/02 16:09:15 INFO mapreduce.Job: Counters: 12
  25.     Job Counters 
  26.         Failed map tasks=4
  27.         Launched map tasks=4
  28.         Other local map tasks=3
  29.         Data-local map tasks=1
  30.         Total time spent by all maps in occupied slots (ms)=19779
  31.         Total time spent by all reduces in occupied slots (ms)=0
  32.         Total time spent by all map tasks (ms)=19779
  33.         Total vcore-seconds taken by all map tasks=19779
  34.         Total megabyte-seconds taken by all map tasks=20253696
  35.     Map-Reduce Framework
  36.         CPU time spent (ms)=0
  37.         Physical memory (bytes) snapshot=0
  38.         Virtual memory (bytes) snapshot=0

我的主要课程：

  1. package fypusinghadoop;
  3. import java.net.URI;
  5. import org.apache.commons.logging.Log;
  6. import org.apache.commons.logging.LogFactory;
  7. import org.apache.hadoop.fs.Path;
  8. import org.apache.hadoop.conf.*;
  9. import org.apache.hadoop.io.*;
  10. import org.apache.hadoop.mapreduce.Job;
  11. import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
  12. import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
  13. import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
  14. import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
  16. import output.VideoOutputFormat;
  17. import input.VideoInputFormat;
  19. public class FypUsingHadoop  {
  20.     private static final Log LOG = LogFactory.getLog(FypUsingHadoop.class);
  22.     public static void main(String[] args) throws Exception {
  24.         Configuration conf = new Configuration();
  25.         long milliSeconds = 1800000; 
  26.         conf.setLong("mapred.task.timeout", milliSeconds);
  27.         Job job = Job.getInstance(conf);
  28.         job.setJarByClass(FypUsingHadoop.class);
  29.         job.setOutputKeyClass(Text.class);
  30.         job.setOutputValueClass(VideoObject.class);
  31.         job.setMapperClass(VidMapper.class);    
  32.         job.setReducerClass(VidReducer.class);
  33.         job.setInputFormatClass(VideoInputFormat.class);
  34.         job.setOutputFormatClass(VideoOutputFormat.class);
  35.         FileInputFormat.addInputPath(job, new Path(args[0]));
  37.         FileOutputFormat.setOutputPath(job, new Path(args[1]));
  39.         job.waitForCompletion(true);        
  40.     }
  41. }

这是我的mapper类：

  1. package fypusinghadoop;
  3. import java.io.ByteArrayInputStream;
  4. import java.io.IOException;
  5. import java.lang.reflect.Field;
  7. import org.apache.commons.logging.Log;
  8. import org.apache.commons.logging.LogFactory;
  9. import org.apache.hadoop.fs.FSDataInputStream;
  10. import org.apache.hadoop.fs.FSDataOutputStream;
  11. import org.apache.hadoop.fs.FileSystem;
  12. import org.apache.hadoop.fs.LocalFileSystem;
  13. import org.apache.hadoop.fs.Path;
  14. import org.apache.hadoop.io.IntWritable;
  15. import org.apache.hadoop.io.LongWritable;
  16. import org.apache.hadoop.io.Text;
  17. import org.apache.hadoop.mapreduce.Mapper;
  18. import org.apache.hadoop.mapreduce.Mapper.Context;
  20. import static org.bytedeco.javacpp.helper.opencv_objdetect.cvHaarDetectObjects;
  22. import org.bytedeco.javacpp.Loader;
  23. import org.bytedeco.javacpp.opencv_core;
  24. import org.bytedeco.javacpp.opencv_core.CvMemStorage;
  25. import org.bytedeco.javacpp.opencv_core.CvRect;
  26. import org.bytedeco.javacpp.opencv_core.CvScalar;
  27. import org.bytedeco.javacpp.opencv_core.CvSeq;
  28. import org.bytedeco.javacpp.opencv_core.CvSize;
  29. import static org.bytedeco.javacpp.opencv_core.IPL_DEPTH_8U;
  30. import org.bytedeco.javacpp.opencv_core.IplImage;
  31. import static org.bytedeco.javacpp.opencv_core.cvClearMemStorage;
  32. import static org.bytedeco.javacpp.opencv_core.cvClearSeq;
  33. import static org.bytedeco.javacpp.opencv_core.cvCreateImage;
  34. import static org.bytedeco.javacpp.opencv_core.cvGetSeqElem;
  35. import static org.bytedeco.javacpp.opencv_core.cvLoad;
  36. import static org.bytedeco.javacpp.opencv_core.cvPoint;
  37. import static org.bytedeco.javacpp.opencv_imgproc.CV_AA;
  38. import static org.bytedeco.javacpp.opencv_imgproc.CV_BGR2GRAY;
  39. import static org.bytedeco.javacpp.opencv_imgproc.cvCvtColor;
  40. import static org.bytedeco.javacpp.opencv_imgproc.cvRectangle;
  41. import static org.bytedeco.javacpp.opencv_objdetect.CV_HAAR_DO_CANNY_PRUNING;
  42. import org.bytedeco.javacpp.opencv_objdetect.CvHaarClassifierCascade;
  43. import org.bytedeco.javacv.CanvasFrame;
  44. import org.bytedeco.javacv.FFmpegFrameGrabber;
  45. import org.bytedeco.javacv.Frame;
  46. import org.bytedeco.javacv.FrameGrabber;
  47. import org.bytedeco.javacv.FrameRecorder;
  48. import org.bytedeco.javacv.OpenCVFrameConverter;
  49. import org.bytedeco.javacv.OpenCVFrameGrabber;
  50. import org.bytedeco.javacv.OpenCVFrameRecorder;
  51. import org.opencv.core.Core;
  53. public class VidMapper extends Mapper<Text, VideoObject, Text, VideoObject> {
  55.     private static final Log LOG = LogFactory.getLog(VidMapper.class);
  56.     private static FrameGrabber grabber;
  57.     private static Frame currentFrame;
  59.     public void map(Text key, VideoObject value, Context context)
  60.             throws IOException, InterruptedException {
  61.         {
  62.             System.out.println("hamzaaj  : " + key);
  63.             ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(
  64.                     value.getVideoByteArray());
  65.             LOG.info("Log__VideoConverter__byteArray: "
  66.                     + byteArrayInputStream.available());
  68.             String fileName = key.toString();
  69.             int id = value.getId();
  70.             LocalFileSystem fs = FileSystem
  71.                     .getLocal(context.getConfiguration());
  72.             Path filePath = new Path("/usr/local/hadoop/ia3/newVideo", fileName);
  73.             Path resFile = new Path("/usr/local/hadoop/ia3/", "res_" + fileName);
  74.             System.out.println("File to Process :" + filePath.toString());
  75.             FSDataOutputStream out = fs.create(filePath, true);
  76.             out.write(value.getVideoByteArray());
  77.             out.close();
  78.             try {
  80.                 System.out.println("Setting Properties");
  81.                 System.setProperty("java.library.path",
  82.                         "/home/mnh/Documents/OpenCV/opencv-3.2.0/build/lib");
  83.                  System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
  84.                 System.load("/home/mnh/Documents/OpenCV/opencv-3.2.0/build/lib/libopencv_core.so");
  86.                 System.out.println("Loading classifier");
  87.                 CvHaarClassifierCascade classifier = new CvHaarClassifierCascade(
  88.                         cvLoad("/home/mnh/Desktop/haarcascade_frontalface_alt.xml"));
  89.                 if (classifier.isNull()) {
  90.                     System.err.println("Error loading classifier file");
  91.                 }
  92.                 grabber = new FFmpegFrameGrabber(
  93.                         "/usr/local/hadoop/input/cars.mp4");
  94.                 grabber.start();
  95.                 OpenCVFrameConverter.ToIplImage converter = new OpenCVFrameConverter.ToIplImage();
  97.                 IplImage grabbedImage = converter.convert(grabber.grab());
  98.                 int width = grabbedImage.width();
  99.                 int height = grabbedImage.height();
  100.                 IplImage grayImage = IplImage.create(width, height,
  101.                         IPL_DEPTH_8U, 1);
  102.                 IplImage rotatedImage = grabbedImage.clone();
  104.                 CvMemStorage storage = CvMemStorage.create();
  105.                 CvSize frameSize = new CvSize(grabber.getImageWidth(),
  106.                         grabber.getImageHeight());
  107.                 CvSeq faces = null;
  108.                 FrameRecorder recorder = FrameRecorder.createDefault(
  109.                         resFile.toString(), width, height);
  110.                 recorder.start();
  111.                 System.out.println("Video processing .........started");
  112.                 // CanvasFrame frame = new CanvasFrame("Some Title",
  113.                 // CanvasFrame.getDefaultGamma()/grabber.getGamma());
  114.                 CanvasFrame frame = new CanvasFrame("Some Title",
  115.                         CanvasFrame.getDefaultGamma() / grabber.getGamma());
  116.                 int i = 0;
  117.                 while (((grabbedImage = converter.convert(grabber.grab())) != null)) {
  118.                     i++;
  119.                     cvClearMemStorage(storage);
  120.                     // Let's try to detect some faces! but we need a grayscale
  121.                     // image...
  122.                     cvCvtColor(grabbedImage, grayImage, CV_BGR2GRAY);
  123.                     faces = cvHaarDetectObjects(grayImage, classifier, storage,
  124.                             1.1, 3, CV_HAAR_DO_CANNY_PRUNING);
  125.                     int total = faces.total();
  126.                     for (int j = 0; j < total; j++) {
  127.                         CvRect r = new CvRect(cvGetSeqElem(faces, j));
  128.                         int x = r.x(), y = r.y(), w = r.width(), h = r.height();
  129.                         cvRectangle(grabbedImage, cvPoint(x, y),
  130.                                 cvPoint(x + w, y + h), CvScalar.RED, 1, CV_AA,
  131.                                 0);
  132.                     }
  133.                     cvClearSeq(faces);
  134.                     Frame rotatedFrame = converter.convert(grabbedImage);
  135.                     recorder.record(rotatedFrame);
  136.                     System.out.println("Hello" + i);
  137.                 }
  138.                 grabber.stop();
  139.                 recorder.stop();
  140.                 System.out.println("Video processing .........Completed");
  142.             } catch (Exception e) {
  143.                 e.printStackTrace();
  144.             }
  145.             FSDataInputStream fin = fs.open(new Path(resFile.toString()));
  146.             byte[] b = new byte[fin.available()];
  147.             fin.readFully(b);
  148.             fin.close();
  149.             VideoObject vres = new VideoObject(b);
  150.             vres.setId(id);
  151.             context.write(key, vres);
  152.             // fs.delete(new Path(resFile.toString()),false);
  153.             fs.delete(filePath, false);
  154.         }
  156.     }
  157. }

这是我的课程：

  1. package fypusinghadoop;
  3. import java.io.IOException;
  4. import java.util.Iterator;
  5. import org.apache.hadoop.io.IntWritable;
  7. import org.apache.hadoop.io.Text;
  8. import org.apache.hadoop.mapreduce.Reducer;
  9. import org.apache.hadoop.mapreduce.OutputCommitter;
  11. public class VidReducer extends Reducer<Text, VideoObject, Text, VideoObject> {
  13.     public void reduce(Text key, Iterable<VideoObject> values, Context context)
  14.       throws IOException, InterruptedException {
  15.         Iterator<VideoObject> it = values.iterator();
  16.         while(it.hasNext()) {
  17.             System.out.println("Reducer"+"   "+it.next());
  18.         context.write(key, it.next());
  19.         }
  20.     }    
  21.  }

请给我一些正确的方法来创建完全可以运行的jar文件。

JavahadoopJarhadoop2executable-jar

1条答案

按热度按时间
dxxyhpgq

dxxyhpgq1#

您正在Hadoop2+示例中使用针对Hadoop1JAR编译的jar。
在java中编译hadoop代码时，将hadoop-core-1.x.y.jar（例如hadoop-core-1.2.1.jar）替换为以下两个jar：
hadoop-common-2.x.y.jar或hadoop-common-3.x.y.jar，例如hadoop-common-3.2.0.jar
hadoop-mapreduce-client-core-2.x.y.jar或hadoop-mapreduce-client-core-3.x.y.jar，例如hadoop-mapreduce-client-core-3.2.0.jar

赞(0）分享  回复(0）举报  2021-06-01
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com