在oozie中计划从spark scala写入hbase时出现问题

k2arahey  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(371)

嗨,从我们的spark scala应用程序,我们正在连接到hbase和写作。当我们把jar穿过去时,它就像一个符咒。

<action name="spark-action">
        <spark xmlns="uri:oozie:spark-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <master>yarn-cluster</master>
            <mode>cluster</mode>
            <name>Hbase-Test</name>
              <class>org.sample.ConnectorTest</class>
            <jar>hdfs://nameservice1/app/MyhbaseConnector.jar</jar>
              <spark-opts>--jars ${sparkLib} --files ${files} --driver-class-path ${driverClassPath}
              </spark-opts>
              <arg>testValue</arg>
        </spark>
        <ok to="mail"/>
        <error to="Kill"/>
    </action>

但是,当同样的计划在oozie工作流中以spark动作进行时,我们得到了以下异常。我们还传递了一些Spark的行动选择。

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:803)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:312)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:312)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 13 more
rnmwe5a2

rnmwe5a21#

下载:
https://raw.githubusercontent.com/swordsmanliu/sparkstreaminghbase/master/lib/spark-core_2.11-1.5.2.logging.jar
并运行:

spark-submit --jars ./spark-core_2.11-1.5.2.logging.jar ...

那是因为 org.apache.spark.Logging 已在spark 1.6取消+

相关问题