livy spark互动会话

wmomyfyw  于 2021-05-27  发布在  Spark
关注(0)|答案(0)|浏览(527)

我正在尝试用livy创建spark交互式会话,并且我需要添加一个lib,比如我在hdfs中丢失的jar(请参阅我的代码)。但会话已停止,日志在下面。
代码:

client = LivyClient('http://sandbox.c4e.kyomei.fr:10500')
session = client.create_session(SessionKind.SPARK , jars = ['hdfs://sandbox-hdp.hortonworks.com:8020/tmp/tsa2-assembly-0.1.jar'] )

日志:

> 20/05/09 01:43:48 INFO LineBufferedStream: Exception in thread "main"
> scala.reflect.internal.FatalError: object Predef does not have a
> member classOf                    20/05/09 01:43:48 WARN RSCClient:
> Error stopping RPC.                                                   
> io.netty.util.concurrent.BlockingOperationException:
> DefaultChannelPromise@786d2cd8(uncancellable)                         
> 
>         at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:394)
> 
>         at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
> 
>         at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:230)
> ....

> Exception in thread "Thread-32" java.io.IOException: Stream closed    
> 
>         at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
> 
>         at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)       
> 
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:345)

> 20/05/09 01:43:48 WARN ContextLauncher: Child process exited with code
20/05/09 01:43:48 ERROR SparkProcApp: job was killed by user

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题