在org.apache.hadoop.util.reflectionutils.setjobconf中配置对象时出错

yi0zb3m4  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(349)

关于这个错误，我问了很多问题，但找不到解决问题的办法。在这里，我使用hadoop实现了对twitter数据的情绪分析。
主要类别：

public class SentimentAnalysis extends Configured implements Tool{
private static File file;

public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();
    Classify classify = new Classify();

    /**
     * Mapper which reads Tweets text file Store 
     * as <"Positive",1> or <"Negative",1>
     */
    public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        String line = value.toString();//streaming each tweet from the text file
        if (line != null) {
            word.set(classify.classify(line)); //invoke classify class to get tweet group of each text
            output.collect(word, one);
        } else {
            word.set("Error");
            output.collect(word, one);//Key,value for Mapper
        }
    }
}
public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
    /**
     * Count the frequency of each classified text group
     */

    @Override
    public void reduce(Text key, Iterator<IntWritable> classifiedText,
            OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        int sum = 0;
        while (classifiedText.hasNext()) {
            sum += classifiedText.next().get(); //Sum the frequency
        }
        output.collect(key, new IntWritable(sum));
    }
}
public static class Classify {
    String[] categories;
    @SuppressWarnings("rawtypes")
    LMClassifier lmc;

    /**
     * Constructor loading serialized object created by Model class to local
     * LMClassifer of this class
     */
    @SuppressWarnings("rawtypes")
    public Classify() {
        try {

            lmc = (LMClassifier) AbstractExternalizable.readObject(file);
            categories = lmc.categories();
        } catch (ClassNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    /**
     * Classify whether the text is positive or negative based on Model object
     * 
     * @param text
     * @return classified group i.e either positive or negative
     */
    public String classify(String text) {
        ConditionalClassification classification = lmc.classify(text);
        return classification.bestCategory();
    }
}

public static void main(String[] args) throws Exception {
    int ret = ToolRunner.run(new SentimentAnalysis(), args);
    System.exit(ret);
}

@Override
public int run(String[] args) throws Exception {
    if(args.length < 2) {
        System.out.println("Invalid input and output directories");
        return -1;
    }
    JobConf conf = new JobConf(getConf(), SentimentAnalysis.class);
    conf.setJobName("sentimentanalysis");
    conf.setJarByClass(SentimentAnalysis.class);
    conf.setOutputKeyClass(Text.class);
    conf.setOutputValueClass(IntWritable.class);
    conf.setMapOutputKeyClass(Text.class);
    conf.setMapOutputValueClass(IntWritable.class);
    conf.setMapperClass(Map.class);
    //conf.setCombinerClass(Reduce.class);
    conf.setReducerClass(Reduce.class);
    conf.setInputFormat(TextInputFormat.class);
    conf.setOutputFormat(TextOutputFormat.class);
    FileInputFormat.setInputPaths(conf, new Path(args[0]));
    FileOutputFormat.setOutputPath(conf, new Path(args[1]));
    file = new File(args[2]);
    JobClient.runJob(conf);
    return 0;
}
}

错误：

[cloudera@localhost ~]$ hadoop jar Sentiment.jar SentimentAnalysis test.txt SentimentOutput classifier.txt

test.txt包含了一些需要分析其情绪的tweet。classifier.txt是一个编码的文本文件，它帮助classifier（lmclassifier）类分析test.txt中出现的tweet。

14/10/05 20:59:23 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/10/05 20:59:24 INFO mapred.FileInputFormat: Total input paths to process : 1
14/10/05 20:59:24 INFO mapred.JobClient: Running job: job_201410041909_0035
14/10/05 20:59:25 INFO mapred.JobClient:  map 0% reduce 0%
14/10/05 20:59:41 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000000_0, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 20:59:41 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000001_0, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 20:59:52 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000000_1, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 20:59:53 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000001_1, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 21:00:04 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000000_2, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 21:00:04 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000001_2, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 21:00:19 INFO mapred.JobClient: Job complete: job_201410041909_0035
14/10/05 21:00:19 INFO mapred.JobClient: Counters: 7
14/10/05 21:00:19 INFO mapred.JobClient:   Job Counters 
14/10/05 21:00:19 INFO mapred.JobClient:     Failed map tasks=1
14/10/05 21:00:19 INFO mapred.JobClient:     Launched map tasks=8
14/10/05 21:00:19 INFO mapred.JobClient:     Data-local map tasks=8
14/10/05 21:00:19 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=98236
14/10/05 21:00:19 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=0
14/10/05 21:00:19 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
14/10/05 21:00:19 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
14/10/05 21:00:19 INFO mapred.JobClient: Job Failed: NA
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1416)
at SentimentAnalysis.run(SentimentAnalysis.java:124)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at SentimentAnalysis.main(SentimentAnalysis.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
JavahadoopubuntumapreduceRuntimeException

1条答案

按热度按时间
jljoyd4f

jljoyd4f1#

我个人没有使用hadoop的经验，但是如果你“查看”堆栈跟踪，它似乎是org.apache.hadoop.util.reflectionutils.setjobconf中的运行时异常。。。

private static void setJobConf(Object theObject, Configuration conf) {
75     //If JobConf and JobConfigurable are in classpath, AND
76     //theObject is of type JobConfigurable AND
77     //conf is of type JobConf then
78     //invoke configure on theObject
79     try {
80       Class<?> jobConfClass = 
81         conf.getClassByName("org.apache.hadoop.mapred.JobConf");
82       Class<?> jobConfigurableClass = 
83         conf.getClassByName("org.apache.hadoop.mapred.JobConfigurable");
84        if (jobConfClass.isAssignableFrom(conf.getClass()) &&
85             jobConfigurableClass.isAssignableFrom(theObject.getClass())) {
86         Method configureMethod = 
87           jobConfigurableClass.getMethod("configure", jobConfClass);
88         configureMethod.invoke(theObject, conf);
89       }
90     } catch (ClassNotFoundException e) {
91       //JobConf/JobConfigurable not in classpath. no need to configure
92     } catch (Exception e) {
93       throw new RuntimeException("Error in configuring object", e);
94     }
95   }

显然，jobconf和jobconfiguble类都在类路径中（否则它将通过cnfe catch块），因此发生了另一个异常。。。似乎嵌套的异常是java.lang.reflect.invocationtargetexception，这表明上面第88行的“invoke”有问题。
因此，尝试在传入配置的目标作业示例上调用“configure”方法失败。
invocationtargetexception可能已经 Package 了实际的原因异常，因此您需要在顶层捕获runtimeexception，然后再捕获e.getcause（）.getcause（）.printstacktrace（），以找出调用configure方法失败的原因。

赞(0）分享  回复(0）举报  2021-06-03
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com