I want to run grades example proposed by apache beam go sdk using spark runner on a spark cluster with one master and two slaves (spark2.4.5 version ). I start the job runner using ./gradlew :runners:spark:2:job-server:runShadow -PsparkMasterUrl=spark://localhost:7077 and run it using the following command : grades -runner=spark -endpoint=localhost:8099 -job_name=gradetest
However I get the following error.
2021/10/08 11:42:34 (): org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.type.TypeBindings.emptyBindings()Lcom/fasterxml/jackson/databind/type/TypeBindings;
at org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:73)
at org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:104)
at org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:92)
at org.apache.beam.runners.spark.SparkPipelineRunner.run(SparkPipelineRunner.java:199)I assume that the problem can be from the spark job runner because i test another code using python and I get the same error. Any help would be gratefully appreciated
1条答案
按热度按时间k5ifujac1#
很抱歉，我回答得太晚了，我只是在调查相关的问题。
不幸的是，Spark 2的作业服务器在移植Beam中使用的Jackson版本时发生了故障。默认情况下，Spark首先从它自己的系统类路径加载类。而那个类路径包含了一个非常过时的Jackson版本，它不再与Beam兼容了。
虽然在向Spark提交Beam Java应用程序时有一些变通方法可以解决这个问题，但不幸的是，这些方法不适用于Spark作业服务器。
此外，Spark 2支持在Beam中已被弃用。
我建议运行Spark 3的作业服务器。