hbase mapreduce作业编译但在运行时中断

我有一个项目，正在成功地实施mapreduce作业。但我现在正在尝试添加另一个从hbase提取数据的作业。这个项目编译得很好，但当我试图运行的工作，我得到了 java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.Job.addFileToClassPath(Lorg/apache/hadoop/fs/Path;)V 有关完整的堆栈跟踪，请参见底部的代码。我已经能够将它缩小到maven依赖，但是我找不到问题所在。我添加的唯一依赖项是

<dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-server</artifactId>
            <version>0.98.6-cdh5.3.2</version>
            <exclusions>
                <exclusion>
                    <artifactId>jasper-compiler</artifactId>
                    <groupId>tomcat</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jasper-runtime</artifactId>
                    <groupId>tomcat</groupId>
                </exclusion>
                <exclusion>
                    <groupId>org.mortbay.jetty</groupId>
                    <artifactId>jsp-2.1</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

完整的pom.xml可在下面找到。什么是冲突？我找不到 org.apache.hadoop.mapreduce.Job 取决于，但是每当我尝试添加依赖项时，它就会破坏编译，就像我有重复的依赖项一样。edit：我发现依赖关系是从另一个类继承的，这就是为什么会产生重复的依赖关系。看起来hadoopmapreduce客户机核心jar在运行时没有包含，但是它没有作用域，我已经确认它在buildpath中。

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>parent-project</artifactId>
        <groupId>com.project</groupId>
        <version>HEAD-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>this-project</artifactId>
    <packaging>jar</packaging>

    <properties>
        <org.springframework-version>4.1.5.RELEASE</org.springframework-version>
        <spring.batch.version>3.0.0.RELEASE</spring.batch.version>
    </properties>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>com.project</groupId>
            <artifactId>custom-library/artifactId>
            <version>HEAD-SNAPSHOT</version>
            <exclusions>
                <exclusion>
                    <artifactId>log4j</artifactId>
                    <groupId>log4j</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>slf4j-jcl</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>slf4j-log4j12</artifactId>
                    <groupId>org.slf4j</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>javax.servlet</artifactId>
                    <groupId>org.glassfish</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>juel-impl</artifactId>
                    <groupId>de.odysseus.juel</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>servlet-api</artifactId>
                    <groupId>javax.servlet</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.5.0-cdh5.3.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>2.5.0-cdh5.3.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-client</artifactId>
            <version>2.5.0-cdh5.3.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.5.0-cdh5.3.2</version>
        </dependency>
        <dependency>
            <groupId>com.hadoop.gplcompression</groupId>
            <artifactId>hadoop-lzo</artifactId>
            <version>0.4.15-gplextras5.0.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.maven.plugin-tools</groupId>
            <artifactId>maven-plugin-annotations</artifactId>
            <version>3.3</version>
        </dependency>
        <dependency>
            <groupId>cglib</groupId>
            <artifactId>cglib</artifactId>
            <version>2.2.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>0.98.6-cdh5.3.2</version>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.hadoop</groupId>
                    <artifactId>hadoop-core</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-protocol</artifactId>
            <version>0.98.6-cdh5.3.2</version>
            <exclusions>
                <exclusion>
                    <artifactId>jasper-compiler</artifactId>
                    <groupId>tomcat</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jasper-runtime</artifactId>
                    <groupId>tomcat</groupId>
                </exclusion>
                <exclusion>
                    <groupId>org.mortbay.jetty</groupId>
                    <artifactId>jsp-2.1</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-server</artifactId>
            <version>0.98.6-cdh5.3.2</version>
            <exclusions>
                <exclusion>
                    <artifactId>jasper-compiler</artifactId>
                    <groupId>tomcat</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jasper-runtime</artifactId>
                    <groupId>tomcat</groupId>
                </exclusion>
                <exclusion>
                    <groupId>org.mortbay.jetty</groupId>
                    <artifactId>jsp-2.1</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
            <version>3.4.5-cdh5.3.2</version>
            <exclusions>
                <exclusion>
                    <artifactId>jasper-compiler</artifactId>
                    <groupId>tomcat</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jasper-runtime</artifactId>
                    <groupId>tomcat</groupId>
                </exclusion>
                <exclusion>
                    <groupId>org.mortbay.jetty</groupId>
                    <artifactId>jsp-2.1</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-api</artifactId>
            <version>2.6.0</version>
            <exclusions>
                <exclusion>
                    <artifactId>jasper-compiler</artifactId>
                    <groupId>tomcat</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jasper-runtime</artifactId>
                    <groupId>tomcat</groupId>
                </exclusion>
                <exclusion>
                    <groupId>org.mortbay.jetty</groupId>
                    <artifactId>jsp-2.1</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>
</project>

尝试用 Postman 命中终结点时出错：

<html>
    <head>
        <title>Apache Tomcat/7.0.57 - Error report</title>
    </head>
    <body>
        <h1>HTTP Status 500 - Handler processing failed; nested exception is java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.Job.addFileToClassPath(Lorg/apache/hadoop/fs/Path;)V</h1>
        <HR size="1" noshade="noshade">
            <p>
                <b>type</b> Exception report
            </p>
            <p>
                <b>message</b>
                <u>Handler processing failed; nested exception is java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.Job.addFileToClassPath(Lorg/apache/hadoop/fs/Path;)V</u>
            </p>
            <p>
                <b>description</b>
                <u>The server encountered an internal error that prevented it from fulfilling this request.</u>
            </p>
            <p>
                <b>exception</b>
                <pre>org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.Job.addFileToClassPath(Lorg/apache/hadoop/fs/Path;)V
    org.springframework.web.servlet.DispatcherServlet.triggerAfterCompletionWithError(DispatcherServlet.java:1287)
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:961)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
    org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
</pre>
            </p>
            <p>
                <b>root cause</b>
                <pre>java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.Job.addFileToClassPath(Lorg/apache/hadoop/fs/Path;)V
    com.project.module.basic.util.MapredJobUtil.setExternalHdfsJobPaths(MapredJobUtil.java:107)
    com.project.module.basic.driver.MapredDriver.setExternalHdfsJobPaths(MapredDriver.java:89)
    com.project.module.basic.driver.MapredDriver.setStandardMapredJob(MapredDriver.java:79)
    com.project.module.basic.driver.BasicProcessDriver.setup(BasicProcessDriver.java:46)
    com.project.module.process.service.impl.processConfigService.setupProcess(ProcessConfigService.java:79)
    com.project.module.process.service.impl.processConfigService$$FastClassBySpringCGLIB$$7c1056d6.invoke(&lt;generated&gt;)
    org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:649)
    com.project.module.process.service.impl.processServiceImpl.setupprocess(processServiceImpl.java:102)
    com.project.module.process.service.impl.processServiceImpl.runprocess(processServiceImpl.java:56)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:606)
    org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    org.springframework.cache.interceptor.CacheInterceptor$1.invoke(CacheInterceptor.java:52)
    org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:317)
    org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:350)
    org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:299)
    org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:61)
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
    com.sun.proxy.$Proxy83.runprocess(Unknown Source)
    com.project.module.process.controller.impl.processControllerImpl.runprocess(processControllerImpl.java:28)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:606)
    org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
    org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
    org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:777)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:706)
    org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
    org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
</pre>
            </p>
            <p>
                <b>note</b>
                <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.57 logs.</u>
            </p>
            <HR size="1" noshade="noshade">
                <h3>Apache Tomcat/7.0.57</h3>
            </body>
        </html>

hbase mapreduce作业编译但在运行时中断

暂无答案！

相关问题

热门标签

最新问答