spark-kafka流代码对livy不起作用

svdrlsy4  于 2021-06-06  发布在  Kafka
关注(0)|答案(0)|浏览(277)

我一直在用scala代码来运行spark流来采集Kafka的数据。一旦更新了所有依赖项并将新的依赖项添加到已编译的jar中，我就可以使用spark submit和代码运行它 spark-submit /codigos/codigos/adquisicion.jar . 但通过livy2运行时，在中创建流时出现异常：

  1. val stream                          = KafkaUtils.createDirectStream[String, String](
  2.         ssc,
  3.         PreferConsistent,
  4.         Subscribe[String, String](topics, kafkaParams)
  5.     )

通过livy2运行它的代码是：

  1. uri<-"http://192.168.0.28:8999/batches"
  3. jsonBody2 = paste0('{ "file": "/user/bigdata/adquisicion.jar",
  5.                      "numExecutors":1,
  7.                      "executorCores":3,
  9.                      "name": "Adquisición Datos",
  11.                      "args": [""],
  13.                      "className": "codigos.adquisicionDatos" }')
  15. request <- httr::POST(url = uri, body = jsonBody2, httr::add_headers(.headers = c('Content-Type' = 'application/json', 'X-Requested-by'='user')))

收到的异常如下：

  1. 18/11/06 08:01:00 ERROR ApplicationMaster: User class threw exception: java.lang.NoClassDefFoundError: org/apache/kafka/clients/consumer/Consumer
  2. java.lang.NoClassDefFoundError: org/apache/kafka/clients/consumer/Consumer
  3.     at org.apache.spark.streaming.kafka010.ConsumerStrategies$.Subscribe(ConsumerStrategy.scala:256)
  4.     at codigos.adquisicionDatos$.main(adquisicionDatos.scala:251)
  5.     at codigos.adquisicionDatos.main(adquisicionDatos.scala)
  6.     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  7.     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  8.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  9.     at java.lang.reflect.Method.invoke(Method.java:498)
  10.     at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:646)
  11. Caused by: java.lang.ClassNotFoundException: org.apache.kafka.clients.consumer.Consumer
  12.     at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  13.     at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  14.     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
  15.     at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  16.     ... 8 more

编辑：
pom文件：

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  3.          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  4.     <modelVersion>4.0.0</modelVersion>
  6.     <groupId>codigos</groupId>
  7.     <artifactId>adquisicionDatos</artifactId>
  8.     <version>1.2</version>
  9.     <packaging>jar</packaging>
  11.     <name>Adquisicion Datos Kafka</name>
  13.     <properties>
  14.         <app.main.class>adquisicionDatos.SparkKMeansApp</app.main.class>
  15.         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  17.         <spark.core.version>2.3.2</spark.core.version>
  18.         <!--hbase.version>0.98.7-hadoop1</hbase.version-->
  19.         <hbase.version>1.0.1</hbase.version>
  20.         <slf4j.version>1.7.5</slf4j.version>
  21.         <guava.version>13.0.1</guava.version>
  22.         <gson.version>2.2.4</gson.version>
  23.         <spark.version>2.3.2</spark.version>
  24.     </properties>
  26.     <repositories>
  27.         <repository>
  28.             <id>scalanlp.org</id>
  29.             <name>ScalaNLP Maven2 Repository</name>
  30.             <url>http://repo.scalanlp.org/repo</url>
  31.         </repository>
  32.         <repository>
  33.             <id>apache release</id>
  34.             <url>https://repository.apache.org/content/repositories/releases/</url>
  35.         </repository>
  36.         <repository>
  37.             <id>scala-tools.org</id>
  38.             <name>Scala-tools Maven2 Repository</name>
  39.             <url>http://scala-tools.org/repo-releases</url>
  40.         </repository>
  41.         <repository>
  42.             <id>cloudera.repo</id>
  43.             <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
  44.             <name>Cloudera Repositories</name>
  45.             <releases>
  46.                 <enabled>true</enabled>
  47.             </releases>
  48.             <snapshots>
  49.                 <enabled>false</enabled>
  50.             </snapshots>
  51.         </repository>
  52.     </repositories>
  54.     <pluginRepositories>
  55.         <pluginRepository>
  56.             <id>scala-tools.org</id>
  57.             <name>Scala-tools Maven2 Repository</name>
  58.             <url>http://scala-tools.org/repo-releases</url>
  59.         </pluginRepository>
  60.     </pluginRepositories>
  62.     <dependencies>
  63.         <!-- https://mvnrepository.com/artifact/org.sameersingh.scalaplot/scalaplot -->
  64.         <dependency>
  65.             <groupId>org.sameersingh.scalaplot</groupId>
  66.             <artifactId>scalaplot</artifactId>
  67.             <version>0.1</version>
  68.         </dependency>
  70.     <!-- https://mvnrepository.com/artifact/javax.mail/javax.mail-api -->
  71.         <dependency>
  72.             <groupId>javax.mail</groupId>
  73.             <artifactId>javax.mail-api</artifactId>
  74.             <version>1.5.1</version>
  75.         </dependency>
  77.         <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11 -->
  78.         <dependency>
  79.             <groupId>org.apache.spark</groupId>
  80.             <artifactId>spark-core_2.11</artifactId>
  81.             <version>${spark.version}</version>
  82.             <scope>provided</scope>
  83.         </dependency>
  85.         <dependency>
  86.             <groupId>org.apache.spark</groupId>
  87.             <artifactId>spark-streaming_2.11</artifactId>
  88.             <version>${spark.version}</version>
  89.         </dependency>
  90.         <!--
  91.         <dependency>
  92.             <groupId>org.apache.spark</groupId>
  93.             <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
  94.             <version>${spark.version}</version>
  95.         </dependency>
  96.         -->
  97.         <dependency>
  98.            <groupId>org.apache.spark</groupId>
  99.            <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
  100.            <version>${spark.version}</version>
  101.            <scope>compile</scope>
  102.            <exclusions>
  103.                <exclusion>
  104.                    <groupId>org.apache.kafka</groupId>
  105.                    <artifactId>kafka_2.11</artifactId>
  106.                </exclusion>
  107.           </exclusions>
  108.        </dependency>
  109.        <dependency>
  110.             <groupId>org.apache.kafka</groupId>
  111.             <artifactId>kafka_2.11</artifactId>
  112.             <version>0.10.2.1</version>
  113.        </dependency>
  114.         <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.11 -->
  115.         <dependency>
  116.             <groupId>org.apache.spark</groupId>
  117.             <artifactId>spark-sql_2.11</artifactId>
  118.             <version>${spark.version}</version>
  119.         </dependency>
  120.         <!-- https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector_2.11 -->
  121.         <dependency>
  122.             <groupId>com.datastax.spark</groupId>
  123.             <artifactId>spark-cassandra-connector_2.11</artifactId>
  124.             <version>${spark.version}</version>
  125.             <!--<version>2.0.5</version>-->
  126.         </dependency>
  127.         <!-- https://mvnrepository.com/artifact/org.postgresql/postgresql -->
  128.         <dependency>
  129.             <groupId>org.postgresql</groupId>
  130.             <artifactId>postgresql</artifactId>
  131.             <version>42.1.1</version>
  132.         </dependency>
  133.         <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
  134.         <dependency>
  135.             <groupId>org.apache.kafka</groupId>
  136.             <artifactId>kafka-clients</artifactId>
  137.             <version>0.10.2.1</version>
  138.         </dependency>
  140.     </dependencies>
  142.     <build>
  143.         <pluginManagement>
  144.             <plugins>
  145.                 <plugin>
  146.                     <groupId>org.scala-tools</groupId>
  147.                     <artifactId>maven-scala-plugin</artifactId>
  148.                     <version>2.15.2</version>
  149.                 </plugin>
  150.                 <plugin>
  151.                     <groupId>org.apache.maven.plugins</groupId>
  152.                     <artifactId>maven-compiler-plugin</artifactId>
  153.                     <version>3.1</version>
  154.                     <configuration>
  155.                         <source>1.6</source>
  156.                         <target>1.6</target>
  157.                     </configuration>
  158.                 </plugin>
  159.             </plugins>
  160.         </pluginManagement>
  162.         <plugins>
  163.             <plugin>
  164.                 <groupId>org.scala-tools</groupId>
  165.                 <artifactId>maven-scala-plugin</artifactId>
  166.                 <executions>
  167.                     <execution>
  168.                         <id>scala-compile-first</id>
  169.                         <phase>process-resources</phase>
  170.                         <goals>
  171.                             <goal>add-source</goal>
  172.                             <goal>compile</goal>
  173.                         </goals>
  174.                     </execution>
  175.                     <execution>
  176.                         <id>scala-test-compile</id>
  177.                         <phase>process-test-resources</phase>
  178.                         <goals>
  179.                             <goal>testCompile</goal>
  180.                         </goals>
  181.                     </execution>
  182.                 </executions>
  183.             </plugin>
  184.             <plugin>
  185.                 <groupId>org.apache.maven.plugins</groupId>
  186.                 <artifactId>maven-shade-plugin</artifactId>
  187.                 <version>2.3</version>
  188.                 <executions>
  189.                     <execution>
  190.                         <phase>package</phase>
  191.                         <goals>
  192.                             <goal>shade</goal>
  193.                         </goals>
  194.                     </execution>
  195.                 </executions>
  196.                 <configuration>
  197.                     <filters>
  198.                         <filter>
  199.                             <artifact>*:*</artifact>
  200.                             <excludes>
  201.                                 <exclude>META-INF/*.SF</exclude>
  202.                                 <exclude>META-INF/*.DSA</exclude>
  203.                                 <exclude>META-INF/*.RSA</exclude>
  204.                             </excludes>
  205.                         </filter>
  206.                     </filters>
  207.                     <finalName>${project.artifactId}-${project.version}-jar-with-dependencies</finalName>
  208.                 </configuration>
  209.             </plugin>
  210.             <plugin>
  211.                 <groupId>org.apache.maven.plugins</groupId>
  212.                 <artifactId>maven-assembly-plugin</artifactId>
  213.                 <version>2.4</version>
  214.                 <configuration>
  215.                     <descriptorRefs>
  216.                         <descriptorRef>jar-with-dependencies</descriptorRef>
  217.                     </descriptorRefs>
  218.                     <archive>
  219.                         <manifest>
  220.                             <mainClass>codigos.adquisicionDatos</mainClass>
  221.                         </manifest>
  222.                     </archive>
  223.                 </configuration>
  224.                 <executions>
  225.                     <execution>
  226.                         <phase>package</phase>
  227.                         <goals>
  228.                             <goal>single</goal>
  229.                         </goals>
  230.                     </execution>
  231.                 </executions>
  232.             </plugin>
  233.         </plugins>
  234.     </build>
  235. </project>

我在服务器上编译它，然后将它复制到docker（通过共享卷） cp ~/adquisicionDatos/target/adquisicionDatos1.2-jar-with-dependencies.jar /mnt/codigos/adquisicion.jar )并使用incrond脚本将其上载到hdfs，以便在更新时自动上载文件（工作测试）。
如果在Spark下运行 root 用户一切正常，但使用livy或running under livy 用户不工作并抛出上面的错误。

scalaapache-kafkaapache-sparklivy

暂无答案！

目前还没有任何答案，快来回答吧！

我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com