使用Akka HTTP的Spark即服务

uyto3xhc  于 2022-11-05  发布在  Spark
关注(0)|答案(2)|浏览(208)

我关注了一个使用Akka HTTP将Spark公开为WebService的博客。
当我独立运行时，代码运行良好，但当我试图将其连接到Spark Standalone集群时，代码失败。我做了一些分析，许多人建议正确设置主机条目和spark-env.sh。我做了所有这些，但仍然不工作。

注意：我的Spark版本和我的pom中的Spark版本也是相同的，我可以使用“spark-submit”将作业提交到同一集群。
主机文件

10.93.125.98 master
10.93.125.97 cluster-node-2
10.93.125.99 cluster-node-3

Spark从属文件

master
cluster-node-2
cluster-node-3

Spark环境文件

export SPARK_MASTER_HOST=master

Scala程式码

object SparkFactory {
  val spark: SparkSession = SparkSession.builder
    .master(AppConfig.sparkMaster)
    .appName(AppConfig.sparkAppName)
    .getOrCreate

  val sc = spark.sparkContext
  val sparkConf = sc.getConf
}

完整的代码可以在下面的GitHub repo中找到
https://github.com/spoddutur/spark-as-service-using-embedded-server
当我试图连接到spark集群时，得到下面的错误。任何指针都将非常感谢。

[service-user@10.93.125.98 cloud]$ java -jar spark-with-scala-0.0.1-SNAPSHOT-allinone.jar -m spark://master:7077
Press RETURN to stop...
[INFO] [07/10/2019 15:53:49.205] [WebServer-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(WebServer)] Server online at http://master:8001/
^C[service-user@10.93.125.98 cloud]$ java -jar spark-with-scala-0.0.1-SNAPSHOT-allinone.jar -m spark://master:7077
Press RETURN to stop...
[INFO] [07/10/2019 15:56:15.096] [WebServer-akka.actor.default-dispatcher-2] [akka.actor.ActorSystemImpl(WebServer)] Server online at http://master:8001/
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/07/10 15:56:27 INFO SparkContext: Running Spark version 2.4.3
19/07/10 15:56:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/07/10 15:56:28 INFO SparkContext: Submitted application: spark-with-scala
19/07/10 15:56:28 INFO SecurityManager: Changing view acls to: service-user
19/07/10 15:56:28 INFO SecurityManager: Changing modify acls to: service-user
19/07/10 15:56:28 INFO SecurityManager: Changing view acls groups to:
19/07/10 15:56:28 INFO SecurityManager: Changing modify acls groups to:
19/07/10 15:56:28 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(service-user); groups with view permissions: Set(); users  with modify permissions: Set(service-user); groups with modify permissions: Set()
19/07/10 15:56:29 INFO Utils: Successfully started service 'sparkDriver' on port 39006.
19/07/10 15:56:29 INFO SparkEnv: Registering MapOutputTracker
19/07/10 15:56:29 INFO SparkEnv: Registering BlockManagerMaster
19/07/10 15:56:29 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/07/10 15:56:29 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/07/10 15:56:29 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-f77fe9f4-dbb8-49db-bbf2-9fca39e18e11
19/07/10 15:56:30 INFO MemoryStore: MemoryStore started with capacity 127.2 MB
19/07/10 15:56:30 INFO SparkEnv: Registering OutputCommitCoordinator
19/07/10 15:56:30 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/07/10 15:56:30 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://master:4040
19/07/10 15:56:30 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://master:7077...
19/07/10 15:56:31 INFO TransportClientFactory: Successfully created connection to master/10.93.125.98:7077 after 79 ms (0 ms spent in bootstraps)
[INFO] [07/10/2019 15:56:47.256] [WebServer-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(WebServer)] Request timeout encountered for request [GET /count Empty]
19/07/10 15:56:50 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://master:7077...
19/07/10 15:57:10 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://master:7077...
19/07/10 15:57:30 ERROR StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
19/07/10 15:57:30 WARN StandaloneSchedulerBackend: Application ID is not initialized yet.
19/07/10 15:57:30 INFO SparkUI: Stopped Spark web UI at http://master:4040
19/07/10 15:57:30 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40701.
19/07/10 15:57:30 INFO NettyBlockTransferService: Server created on master:40701
19/07/10 15:57:31 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/07/10 15:57:31 INFO StandaloneSchedulerBackend: Shutting down all executors
19/07/10 15:57:31 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
19/07/10 15:57:31 WARN StandaloneAppClient$ClientEndpoint: Drop UnregisterApplication(null) because has not yet connected to master
19/07/10 15:57:31 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/07/10 15:57:31 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, master, 40701, None)
19/07/10 15:57:31 INFO MemoryStore: MemoryStore cleared
19/07/10 15:57:31 INFO BlockManager: BlockManager stopped
19/07/10 15:57:31 INFO BlockManagerMasterEndpoint: Registering block manager master:40701 with 127.2 MB RAM, BlockManagerId(driver, master, 40701, None)
19/07/10 15:57:31 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, master, 40701, None)
19/07/10 15:57:31 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, master, 40701, None)
19/07/10 15:57:31 INFO BlockManagerMaster: BlockManagerMaster stopped
19/07/10 15:57:31 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/07/10 15:57:31 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
        at scala.Predef$.require(Predef.scala:281)
        at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:516)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935)
        at org.apache.spark.sql.SparkSession$Builder$$Lambda$650.00000000641966D0.apply(Unknown Source)
        at scala.Option.getOrElse(Option.scala:138)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
        at com.learning.spark.SparkFactory$.<init>(SparkFactory.scala:19)
        at com.learning.spark.SparkFactory$.<clinit>(SparkFactory.scala)
        at com.learning.spark.web.HttpService$.<init>(HttpService.scala:13)
        at com.learning.spark.web.HttpService$.<clinit>(HttpService.scala)
        at com.learning.spark.web.WebServer$.$anonfun$routes$12(WebServer.scala:47)
        at com.learning.spark.web.WebServer$$$Lambda$579.000000006405B250.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.RouteDirectives.$anonfun$complete$1(RouteDirectives.scala:47)
        at akka.http.scaladsl.server.directives.RouteDirectives$$Lambda$580.000000006405B700.apply(Unknown Source)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:66)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$578.000000006405A3C0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:43)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$577.0000000064059D40.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$563.00000000640532B0.apply(Unknown Source)
        at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
        at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
        at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResultWith$2(BasicDirectives.scala:72)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$558.0000000064050AA0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.ExecutionDirectives.$anonfun$handleExceptions$2(ExecutionDirectives.scala:32)
        at akka.http.scaladsl.server.directives.ExecutionDirectives$$Lambda$553.000000006404E0B0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.Route$.$anonfun$asyncHandler$1(Route.scala:86)
        at akka.http.scaladsl.server.Route$$$Lambda$419.00000000ED29C4C0.apply(Unknown Source)
        at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1375)
        at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:523)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:409)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:606)
        at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:485)
        at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:581)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:749)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:764)
        at akka.actor.Actor.aroundReceive(Actor.scala:539)
        at akka.actor.Actor.aroundReceive$(Actor.scala:537)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:671)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:612)
        at akka.actor.ActorCell.invoke(ActorCell.scala:581)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
        at akka.dispatch.Mailbox.run(Mailbox.scala:229)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
19/07/10 15:57:31 INFO SparkContext: SparkContext already stopped.
19/07/10 15:57:31 INFO SparkContext: Successfully stopped SparkContext
Uncaught error from thread [WebServer-akka.actor.default-dispatcher-5]: null, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[WebServer]
java.lang.ExceptionInInitializerError
        at java.lang.J9VMInternals.ensureError(J9VMInternals.java:146)
        at java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:135)
        at com.learning.spark.web.HttpService$.<init>(HttpService.scala:13)
        at com.learning.spark.web.HttpService$.<clinit>(HttpService.scala)
        at com.learning.spark.web.WebServer$.$anonfun$routes$12(WebServer.scala:47)
        at com.learning.spark.web.WebServer$$$Lambda$579.000000006405B250.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.RouteDirectives.$anonfun$complete$1(RouteDirectives.scala:47)
        at akka.http.scaladsl.server.directives.RouteDirectives$$Lambda$580.000000006405B700.apply(Unknown Source)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:66)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$578.000000006405A3C0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:43)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$577.0000000064059D40.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$563.00000000640532B0.apply(Unknown Source)
        at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
        at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
        at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResultWith$2(BasicDirectives.scala:72)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$558.0000000064050AA0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.ExecutionDirectives.$anonfun$handleExceptions$2(ExecutionDirectives.scala:32)
        at akka.http.scaladsl.server.directives.ExecutionDirectives$$Lambda$553.000000006404E0B0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.Route$.$anonfun$asyncHandler$1(Route.scala:86)
        at akka.http.scaladsl.server.Route$$$Lambda$419.00000000ED29C4C0.apply(Unknown Source)
        at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1375)
        at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:523)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:409)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:606)
        at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:485)
        at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:581)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:749)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:764)
        at akka.actor.Actor.aroundReceive(Actor.scala:539)
        at akka.actor.Actor.aroundReceive$(Actor.scala:537)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:671)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:612)
        at akka.actor.ActorCell.invoke(ActorCell.scala:581)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
        at akka.dispatch.Mailbox.run(Mailbox.scala:229)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
        at scala.Predef$.require(Predef.scala:281)
        at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:516)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935)
        at org.apache.spark.sql.SparkSession$Builder$$Lambda$650.00000000641966D0.apply(Unknown Source)
        at scala.Option.getOrElse(Option.scala:138)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
        at com.learning.spark.SparkFactory$.<init>(SparkFactory.scala:19)
        at com.learning.spark.SparkFactory$.<clinit>(SparkFactory.scala)
        ... 57 more
19/07/10 15:57:31 INFO ShutdownHookManager: Shutdown hook called
[ERROR] [SECURITY][07/10/2019 15:57:31.461] [WebServer-akka.actor.default-dispatcher-5] [akka.actor.ActorSystemImpl(WebServer)] Uncaught error from thread [WebServer-akka.actor.default-dispatcher-5]: null, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[WebServer]
java.lang.ExceptionInInitializerError
        at java.lang.J9VMInternals.ensureError(J9VMInternals.java:146)
        at java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:135)
        at com.learning.spark.web.HttpService$.<init>(HttpService.scala:13)
        at com.learning.spark.web.HttpService$.<clinit>(HttpService.scala)
        at com.learning.spark.web.WebServer$.$anonfun$routes$12(WebServer.scala:47)
        at com.learning.spark.web.WebServer$$$Lambda$579.000000006405B250.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.RouteDirectives.$anonfun$complete$1(RouteDirectives.scala:47)
        at akka.http.scaladsl.server.directives.RouteDirectives$$Lambda$580.000000006405B700.apply(Unknown Source)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:66)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$578.000000006405A3C0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:43)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$577.0000000064059D40.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$563.00000000640532B0.apply(Unknown Source)
        at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
        at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
        at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResultWith$2(BasicDirectives.scala:72)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$558.0000000064050AA0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.ExecutionDirectives.$anonfun$handleExceptions$2(ExecutionDirectives.scala:32)
        at akka.http.scaladsl.server.directives.ExecutionDirectives$$Lambda$553.000000006404E0B0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.Route$.$anonfun$asyncHandler$1(Route.scala:86)
        at akka.http.scaladsl.server.Route$$$Lambda$419.00000000ED29C4C0.apply(Unknown Source)
        at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1375)
        at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:523)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:409)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:606)
        at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:485)
        at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:581)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:749)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:764)
        at akka.actor.Actor.aroundReceive(Actor.scala:539)
        at akka.actor.Actor.aroundReceive$(Actor.scala:537)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:671)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:612)
        at akka.actor.ActorCell.invoke(ActorCell.scala:581)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
        at akka.dispatch.Mailbox.run(Mailbox.scala:229)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
        at scala.Predef$.require(Predef.scala:281)
        at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:516)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935)
        at org.apache.spark.sql.SparkSession$Builder$$Lambda$650.00000000641966D0.apply(Unknown Source)
        at scala.Option.getOrElse(Option.scala:138)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
        at com.learning.spark.SparkFactory$.<init>(SparkFactory.scala:19)
        at com.learning.spark.SparkFactory$.<clinit>(SparkFactory.scala)
        ... 57 more
akka

2条答案

按热度按时间
lnlaulya

lnlaulya1#

我得到了上述问题的修复。分享的根本原因，供他人参考和保存时间。
根本原因是，我使用的是Spark 2.4.3，它内部使用的是Scala 2.11，而我使用的Scala项目使用的是Scala 2.12。这一直在制造所有的问题。
我已经在我的项目中降级了Scala版本以使其正常工作。

赞(0）分享  回复(0）举报  2022-11-05
2uluyalo

2uluyalo2#

您的问题缺少最小的可重复示例、堆栈跟踪的尾部，甚至缺少指向您所引用的文章的链接。
但是，据我所知，您没有在应用程序中示例化SparkContext的示例。

赞(0）分享  回复(0）举报  2022-11-05
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com