pyspark 使用增量湖的spark-sql/spark-submit导致空指针异常(位于org.apache.spark.storage.BlockManagerMasterEndpoint)

pvcm50d1  于 2023-01-01  发布在  Spark
关注(0)|答案(1)|浏览(210)

我正在使用delta lake on通过提交以下命令使用pyspark

  1. spark-sql --packages io.delta:delta-core_2.12:0.8.0 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"

系统规格：

  • Spark-3.0.3
  • 比例尺-2.12.10
  • Java语言1.8.0
  • Hadoop-2.7

我正在查看参考博客https://docs.delta.io/latest/quick-start.htmlhttps://www.confessionsofadataguy.com/introduction-to-delta-lake-on-apache-spark-for-data-engineers/
但是当我使用没有delta配置的spark时spark工作正常。
错误（截断堆栈跟踪）：

  1. 22/12/29 20:38:10 INFO Executor: Starting executor ID driver on host 192.168.0.100
  2. 22/12/29 20:38:10 INFO Executor: Fetching spark://192.168.0.100:50920/jars/org.antlr_antlr4-runtime-4.7.jar with timestamp 1672326488255
  3. 22/12/29 20:38:23 WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a result reporting of ProcessTree metrics is stopped
  4. 22/12/29 20:38:23 INFO Executor: Told to re-register on heartbeat
  5. 22/12/29 20:38:23 INFO BlockManager: BlockManager null re-registering with master
  6. 22/12/29 20:38:23 INFO BlockManagerMaster: Registering BlockManager null
  7. 22/12/29 20:38:23 ERROR Inbox: Ignoring error
  9.     java.lang.NullPointerException
  10.             at org.apache.spark.storage.BlockManagerMasterEndpoint.org$apache$spark$storage$BlockManagerMasterEndpoint$$register(BlockManagerMasterEndpoint.scala:404)
  11.             at org.apache.spark.storage.BlockManagerMasterEndpoint$$anonfun$receiveAndReply$1.applyOrElse(BlockManagerMasterEndpoint.scala:97)
  12.             at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:103)
  13.             at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
  14.             at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
  15.             at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
  16.             at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
  17.             at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  18.             at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  19.             at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  20.             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  21.             at java.lang.Thread.run(Thread.java:745)
  22.     22/12/29 20:38:23 WARN Executor: Issue communicating with driver in heartbeater
  23.     org.apache.spark.SparkException: Exception thrown in awaitResult:
  24.             at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:302)
  25.             at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
  26.             at org.apache.spark.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:103)
  27.             at org.apache.spark.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:87)
  28.             at org.apache.spark.storage.BlockManagerMaster.registerBlockManager(BlockManagerMaster.scala:66)
  29.             at org.apache.spark.storage.BlockManager.reregister(BlockManager.scala:567)
  30.             at org.apache.spark.executor.Executor.reportHeartBeat(Executor.scala:934)
  31.             at org.apache.spark.executor.Executor.$anonfun$heartbeater$1(Executor.scala:200)
  32.             at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
  33.             at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1934)
  34.             at org.apache.spark.Heartbeater$$anon$1.run(Heartbeater.scala:46)
  35.             at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  36.             at java.lang.Thread.run(Thread.java:745)
  37.     Caused by: java.lang.NullPointerException
  38.             at org.apache.spark.storage.BlockManagerMasterEndpoint.org$apache$spark$storage$BlockManagerMasterEndpoint$$register(BlockManagerMasterEndpoint.scala:404)
  39.             at org.apache.spark.storage.BlockManagerMasterEndpoint$$anonfun$receiveAndReply$1.applyOrElse(BlockManagerMasterEndpoint.scala:97)
  40.             at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:103)
  41.             at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
  42.             at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  43.             ... 3 more
  44.     22/12/29 20:38:31 ERROR Utils: Aborting task
  45.     java.io.IOException: Failed to connect to /192.168.0.100:50920
  46.             at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:253)
  47.             at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:195)
  48.             at org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(NettyRpcEnv.scala:392)
  49.             at org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$openChannel$4(NettyRpcEnv.scala:360)
  50.             at scala.Option.getOrElse(Option.scala:189)
  51.             at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:931)
  52.             at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:49)
  53.             at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.<init>(SparkSQLCLIDriver.scala:321)  
  54.             at java.lang.reflect.Method.invoke(Method.java:498)
  55.             at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
  56.     Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: /192.168.0.100:50920
  57.     Caused by: java.net.ConnectException: Connection timed out: no further information
  58.             at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
  59.             at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
  60.             at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
  61.             at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
  62.             at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
  63.             at java.lang.Thread.run(Thread.java:745)
  64.     22/12/29 20:38:31 ERROR SparkContext: Error initializing SparkContext.
  65.     java.io.IOException: Failed to connect to /192.168.0.100:50920
  66.             at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:253)
  67.             at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:195)
  68.             at org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(NettyRpcEnv.scala:392)
  69.             at org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$openChannel$4(NettyRpcEnv.scala:360)
  70.             at java.lang.reflect.Method.invoke(Method.java:498)
  71.             at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
  72.             at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
  73.             at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
  74.             at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  75.             at java.lang.Thread.run(Thread.java:745)
  76.     22/12/29 20:38:31 INFO SparkUI: Stopped Spark web UI at http://192.168.0.100:4041
  77.     22/12/29 20:38:31 ERROR Utils: Uncaught exception in thread main
  78.     java.lang.NullPointerException
  79.             at org.apache.spark.scheduler.local.LocalSchedulerBackend.org$apache$spark$scheduler$local$LocalSchedulerBackend$$stop(LocalSchedulerBackend.scala:168)
  80.             at org.apache.spark.scheduler.local.LocalSchedulerBackend.stop(LocalSchedulerBackend.scala:144)
  81.             at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:734)
  82.             at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:2171)
  83.     java.lang.NullPointerException
  84.             at org.apache.spark.executor.Executor.$anonfun$stop$3(Executor.scala:304)
  85.             at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
  86.             at org.apache.spark.util.Utils$.withContextClassLoader(Utils.scala:221)
  87.             at org.apache.spark.executor.Executor.stop(Executor.scala:304)
  88.             at org.apache.spark.executor.Executor.$anonfun$stopHookReference$1(Executor.scala:74)
  89.             at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:214)
  90.             at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
  91.             at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
  92.             at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
  93.     22/12/29 20:38:31 INFO ShutdownHookManager: Shutdown hook called

错过了什么？

pyspark

1条答案

按热度按时间
velaa5lx

velaa5lx1#

试着这样做：

  1. !pip3 install findspark --user
  2. import findspark
  3. findspark.init()
  1. import os
  2. os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages io.delta:delta- core_2.12:2.2.0  --driver-memory 2g pyspark-shell'
  1. spark = SparkSession.builder.appName("your application") \
  2. .config("spark.jars.packages", "io.delta:delta-core_2.12:2.2.0") \
  3. .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
  4. .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
  5. .getOrCreate()
赞(0）分享  回复(0）举报  2023-01-01
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com