作业提交后classnotfoundexception

4dbbbstv  于 2021-06-03  发布在  Hadoop
关注(0)|答案(4)|浏览(498)

我正在尝试使用SpringData-hadoop从本地机器的ide在远程集群上执行mr代码
//hadoop 1.1.2、spring 3.2.4、spring data hadoop 1.0.0
尝试了以下版本：
hadoop 1.2.1、spring 4.0.1、spring data hadoop 2.0.2
applicationcontext.xml：

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <beans xmlns="http://www.springframework.org/schema/beans"
  3.     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:hdp="http://www.springframework.org/schema/hadoop"
  4.     xmlns:context="http://www.springframework.org/schema/context"
  5.     xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd 
  6.     http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd
  7.     http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.2.xsd">
  9.     <context:property-placeholder location="resources/hadoop.properties" />
  11.     <hdp:configuration file-system-uri="${hd.fs}" job-tracker-uri="${hd.jobtracker.uri}">
  13.     </hdp:configuration>
  15.     <hdp:job id="wc-job" mapper="com.hadoop.basics.WordCounter.WCMapper"
  16.         reducer="com.hadoop.basics.WordCounter.WCReducer" input-path="${wordcount.input.path}"
  17.         output-path="${wordcount.output.path}" user="bigdata">
  18.     </hdp:job>
  20.     <hdp:job-runner id="myjobs-runner" job-ref="wc-job"
  21.         run-at-startup="true" />
  23.     <hdp:resource-loader id="resourceLoader" uri="${hd.fs}"
  24.         user="bigdata" />   
  25. </beans>

wordcounter.java语言：

  1. package com.hadoop.basics;
  3. import java.io.IOException;
  4. import java.util.StringTokenizer;
  6. import org.apache.hadoop.conf.Configured;
  7. import org.apache.hadoop.io.IntWritable;
  8. import org.apache.hadoop.io.Text;
  9. import org.apache.hadoop.mapreduce.Mapper;
  10. import org.apache.hadoop.mapreduce.Reducer;
  11. import org.springframework.context.support.AbstractApplicationContext;
  12. import org.springframework.context.support.ClassPathXmlApplicationContext;
  14. public class WordCounter {
  16.     private static IntWritable one = new IntWritable(1);
  18.     public static class WCMapper extends Mapper<Text, Text, Text, IntWritable> {
  20.         @Override
  21.         protected void map(
  22.                 Text key,
  23.                 Text value,
  24.                 org.apache.hadoop.mapreduce.Mapper<Text, Text, Text, IntWritable>.Context context)
  25.                 throws IOException, InterruptedException {
  26.             // TODO Auto-generated method stub
  27.             StringTokenizer strTokenizer = new StringTokenizer(value.toString());
  28.             Text token = new Text();
  30.             while (strTokenizer.hasMoreTokens()) {
  31.                 token.set(strTokenizer.nextToken());
  32.                 context.write(token, one);
  33.             }
  34.         }
  35.     }
  37.     public static class WCReducer extends
  38.             Reducer<Text, IntWritable, Text, IntWritable> {
  39.         @Override
  40.         protected void reduce(
  41.                 Text key,
  42.                 Iterable<IntWritable> values,
  43.                 org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable>.Context context)
  44.                 throws IOException, InterruptedException {
  45.             // TODO Auto-generated method stub
  47.             int sum = 0;
  49.             for (IntWritable value : values) {
  50.                 sum += value.get();
  51.             }
  53.             context.write(key, new IntWritable(sum));
  54.         }
  55.     }
  57.     public static void main(String[] args) {
  58.         AbstractApplicationContext context = new ClassPathXmlApplicationContext(
  59.                 "applicationContext.xml", WordCounter.class);
  60.         System.out.println("Word Count Application Running");
  61.         context.registerShutdownHook();
  62.     }
  63. }

输出为：

  1. Aug 23, 2013 11:07:48 AM org.springframework.context.support.AbstractApplicationContext prepareRefresh
  2. INFO: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@1815338: startup date [Fri Aug 23 11:07:48 IST 2013]; root of context hierarchy
  3. Aug 23, 2013 11:07:48 AM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
  4. INFO: Loading XML bean definitions from class path resource [com/hadoop/basics/applicationContext.xml]
  5. Aug 23, 2013 11:07:48 AM org.springframework.core.io.support.PropertiesLoaderSupport loadProperties
  6. INFO: Loading properties file from class path resource [resources/hadoop.properties]
  7. Aug 23, 2013 11:07:48 AM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
  8. INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@7c197e: defining beans [org.springframework.context.support.PropertySourcesPlaceholderConfigurer#0,hadoopConfiguration,wc-job,myjobs-runner,resourceLoader]; root of factory hierarchy
  9. Aug 23, 2013 11:07:49 AM org.springframework.data.hadoop.mapreduce.JobExecutor$2 run
  10. INFO: Starting job [wc-job]
  11. Aug 23, 2013 11:07:49 AM org.apache.hadoop.mapred.JobClient copyAndConfigureFiles
  12. WARNING: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
  13. Aug 23, 2013 11:07:49 AM org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  14. INFO: Total input paths to process : 1
  15. Aug 23, 2013 11:07:50 AM org.apache.hadoop.util.NativeCodeLoader <clinit>
  16. WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  17. Aug 23, 2013 11:07:50 AM org.apache.hadoop.io.compress.snappy.LoadSnappy <clinit>
  18. WARNING: Snappy native library not loaded
  19. Aug 23, 2013 11:07:52 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  20. INFO: Running job: job_201308231532_0002
  21. Aug 23, 2013 11:07:53 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  22. INFO:  map 0% reduce 0%
  23. Aug 23, 2013 11:08:12 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  24. INFO: Task Id : attempt_201308231532_0002_m_000000_0, Status : FAILED
  25. java.lang.RuntimeException: java.lang.ClassNotFoundException: com.hadoop.basics.WordCounter$WCMapper
  26.     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:849)
  27.     at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
  28.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
  29.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
  30.     at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
  31.     at java.security.AccessController.doPrivileged(Native Method)
  32.     at javax.security.auth.Subject.doAs(Subject.java:415)
  33.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
  34.     at org.apache.hadoop.mapred.Child.main(Child.java:249)
  35. Caused by: java.lang.ClassNotFoundException: com.hadoop.basics.WordCounter$WCMapper
  36.     at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
  37.     at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
  38.     at java.security.AccessController.doPrivileged(Native Method)
  39.     at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
  40.     at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
  41.     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
  42.     at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
  43.     at java.lang.Class.forName0(Native Method)
  44.     at java.lang.Class.forName(Class.java:264)
  45.     at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:802)
  46.     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:847)
  47.     ... 8 more
  49. Aug 23, 2013 11:08:33 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  50. INFO: Task Id : attempt_201308231532_0002_m_000000_1, Status : FAILED
  51. java.lang.RuntimeException: java.lang.ClassNotFoundException: com.hadoop.basics.WordCounter$WCMapper
  52.     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:849)
  53.     at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
  54.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
  55.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
  56.     at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
  57.     at java.security.AccessController.doPrivileged(Native Method)
  58.     at javax.security.auth.Subject.doAs(Subject.java:415)
  59.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
  60.     at org.apache.hadoop.mapred.Child.main(Child.java:249)
  61. Caused by: java.lang.ClassNotFoundException: com.hadoop.basics.WordCounter$WCMapper
  62.     at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
  63.     at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
  64.     at java.security.AccessController.doPrivileged(Native Method)
  65.     at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
  66.     at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
  67.     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
  68.     at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
  69.     at java.lang.Class.forName0(Native Method)
  70.     at java.lang.Class.forName(Class.java:264)
  71.     at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:802)
  72.     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:847)
  73.     ... 8 more
  75. Aug 23, 2013 11:08:51 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  76. INFO: Task Id : attempt_201308231532_0002_m_000000_2, Status : FAILED
  77. java.lang.RuntimeException: java.lang.ClassNotFoundException: com.hadoop.basics.WordCounter$WCMapper
  78.     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:849)
  79.     at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
  80.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
  81.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
  82.     at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
  83.     at java.security.AccessController.doPrivileged(Native Method)
  84.     at javax.security.auth.Subject.doAs(Subject.java:415)
  85.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
  86.     at org.apache.hadoop.mapred.Child.main(Child.java:249)
  87. Caused by: java.lang.ClassNotFoundException: com.hadoop.basics.WordCounter$WCMapper
  88.     at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
  89.     at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
  90.     at java.security.AccessController.doPrivileged(Native Method)
  91.     at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
  92.     at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
  93.     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
  94.     at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
  95.     at java.lang.Class.forName0(Native Method)
  96.     at java.lang.Class.forName(Class.java:264)
  97.     at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:802)
  98.     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:847)
  99.     ... 8 more
  101. Aug 23, 2013 11:09:24 AM org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  102. INFO: Job complete: job_201308231532_0002
  103. Aug 23, 2013 11:09:24 AM org.apache.hadoop.mapred.Counters log
  104. INFO: Counters: 7
  105. Aug 23, 2013 11:09:24 AM org.apache.hadoop.mapred.Counters log
  106. INFO:   Job Counters 
  107. Aug 23, 2013 11:09:24 AM org.apache.hadoop.mapred.Counters log
  108. INFO:     SLOTS_MILLIS_MAPS=86688
  109. Aug 23, 2013 11:09:24 AM org.apache.hadoop.mapred.Counters log
  110. INFO:     Total time spent by all reduces waiting after reserving slots (ms)=0
  111. Aug 23, 2013 11:09:24 AM org.apache.hadoop.mapred.Counters log
  112. INFO:     Total time spent by all maps waiting after reserving slots (ms)=0
  113. Aug 23, 2013 11:09:24 AM org.apache.hadoop.mapred.Counters log
  114. INFO:     Launched map tasks=4
  115. Aug 23, 2013 11:09:24 AM org.apache.hadoop.mapred.Counters log
  116. INFO:     Data-local map tasks=4
  117. Aug 23, 2013 11:09:24 AM org.apache.hadoop.mapred.Counters log
  118. INFO:     SLOTS_MILLIS_REDUCES=0
  119. Aug 23, 2013 11:09:24 AM org.apache.hadoop.mapred.Counters log
  120. INFO:     Failed map tasks=1
  121. Aug 23, 2013 11:09:24 AM org.springframework.data.hadoop.mapreduce.JobExecutor$2 run
  122. INFO: Completed job [wc-job]
  123. Aug 23, 2013 11:09:24 AM org.springframework.beans.factory.support.DefaultSingletonBeanRegistry destroySingletons
  124. INFO: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@7c197e: defining beans [org.springframework.context.support.PropertySourcesPlaceholderConfigurer#0,hadoopConfiguration,wc-job,myjobs-runner,resourceLoader]; root of factory hierarchy
  125. Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'myjobs-runner': Invocation of init method failed; nested exception is java.lang.IllegalStateException: Job wc-job] failed to start; status=FAILED
  126.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1482)
  127.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
  128.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
  129.     at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295)
  130.     at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
  131.     at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292)
  132.     at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
  133.     at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628)
  134.     at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
  135.     at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
  136.     at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:197)
  137.     at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:172)
  138.     at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:158)
  139.     at com.hadoop.basics.WordCounter.main(WordCounter.java:58)
  140. Caused by: java.lang.IllegalStateException: Job wc-job] failed to start; status=FAILED
  141.     at org.springframework.data.hadoop.mapreduce.JobExecutor$2.run(JobExecutor.java:219)
  142.     at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:49)
  143.     at org.springframework.data.hadoop.mapreduce.JobExecutor.startJobs(JobExecutor.java:168)
  144.     at org.springframework.data.hadoop.mapreduce.JobExecutor.startJobs(JobExecutor.java:160)
  145.     at org.springframework.data.hadoop.mapreduce.JobRunner.call(JobRunner.java:52)
  146.     at org.springframework.data.hadoop.mapreduce.JobRunner.afterPropertiesSet(JobRunner.java:44)
  147.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1541)
  148.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1479)
  149.     ... 13 more

什么配置。我错过了吗？是否真的可以使用spring数据远程提交hadoop作业而不创建jar等？

hadoopspringspring-dataspring-data-hadoop

4条答案

按热度按时间
fafcakar

fafcakar1#

右键单击项目，build path->configure build path。选择libraries选项卡，单击addexternaljars并复制hadoop目录中所有必需的jar文件。我想，这会解决你的问题。

赞(0）分享  回复(0）举报  2021-06-03
xvw2m8pv

xvw2m8pv2#

我得到了同样的问题，然后分离了mapper和reducer类的工作，并在 applicationContext.xml :

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <beans xmlns="http://www.springframework.org/schema/beans"
  3.     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:util="http://www.springframework.org/schema/util"
  4.     xmlns:context="http://www.springframework.org/schema/context"
  5.     xmlns:hdp="http://www.springframework.org/schema/hadoop" xmlns:batch="http://www.springframework.org/schema/batch"
  6.     xsi:schemaLocation="
  7.     http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
  8.     http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd
  9.      http://www.springframework.org/schema/context  http://www.springframework.org/schema/context/spring-context.xsd
  10.      http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd
  11.     http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-4.2.xsd">
  13.     <context:property-placeholder location="classpath:application.properties" />
  14.     <hdp:configuration namenode-principal="hdfs://xx.yy.com" rm-manager-uri="xx.yy.com"
  15.         security-method="kerb" user-keytab="location" rm-manager-principal="username"
  16.         user-principal="username">
  17.         fs.default.name=${fs.default.name}
  18.         mapred.job.tracker=${mapred.job.tracker}
  19.     </hdp:configuration>
  21.     <hdp:job id="wordCountJobId" input-path="${input.path}"
  22.         output-path="${output.path}" jar-by-class="com.xx.poc.Application"
  23.         mapper="com.xx.poc.Map" reducer="com.xx.poc.Reduce" />
  25.     <hdp:job-runner id="wordCountJobRunner" job-ref="wordCountJobId"
  26.         run-at-startup="true" />
  27. </beans>
展开查看全部
赞(0）分享  回复(0）举报  2021-06-03
kh212irz

kh212irz3#

我在这里的第一个hadoop工作也遇到了同样的问题，解决方法是在eclipse中编译jar文件。
当您想在eclipse中将java项目导出为jar文件时，有两个选项可用：

  1. Extract required libraries into generated JAR
  2. Package required libraries into generated JAR

第一个选择解决了我的问题，也可能解决了你的问题。

赞(0）分享  回复(0）举报  2021-06-03
csbfibhn

csbfibhn4#

你可以试试 jar-by-class 属性来定位jar并让hadoop将jar上传到tasktracker。

  1. <hdp:job id="wc-job" mapper="com.hadoop.basics.WordCounter.WCMapper"
  2.     reducer="com.hadoop.basics.WordCounter.WCReducer"
  3.     input-path="${wordcount.input.path}"
  4.     jar-by-class="com.hadoop.basics.WordCounter"
  5.     output-path="${wordcount.output.path}" user="bigdata">
  6. </hdp:job>

最后，wcmapper和wcreducer应该是静态的，否则它们不能由hadoop创建。

赞(0）分享  回复(0）举报  2021-06-03
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com