spark hbase连接器在并行模式下不工作?

wh6knrhe  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(287)

我正在尝试使用hortonworks hbase connector for spark 2.0来处理hbase(https://github.com/hortonworks-spark/shc/tree/v1.1.0-2.0)
通过上面链接中提供的示例,

  1. val spark = SparkSession
  2. .builder()
  3. .appName(getClass.toString)
  4. .getOrCreate()
  5. def withCatalog(cat: String, spark: SparkSession): DataFrame = {
  6. spark
  7. .read
  8. .options(Map(HBaseTableCatalog.tableCatalog->cat))
  9. .format("org.apache.spark.sql.execution.datasources.hbase")
  10. .load()
  11. }
  12. val df = withCatalog(cat, spark)
  13. df.printSchema()
  14. df.show(20, false)

架构:

  1. val cat =
  2. s"""{
  3. |"table":{"namespace":"test", "name":"test_src_data", "tableCoder":"PrimitiveType"},
  4. |"rowkey":"tfkod_description",
  5. |"columns":{
  6. |"col0":{"cf":"rowkey", "col":"tfkod_description", "type":"string"},
  7. |"src_stream_desc":{"cf":"src_data", "col":"src_desc", "type":"string"}
  8. |}
  9. |}""".stripMargin

完成spark2之后,提交作业运行并只打印模式。后来所有的执行者都存在并永远被卡住了。
日志中的最后一条消息:
现有执行人41已被移除(新的总数为1)
但我可以成功地以顺序方式使用hbase,即put或bulkput,而不是rdd或df(使用任何hbase连接器)的方式在spark中工作。
hbase/spark配置中是否有任何错误是由于哪个spark执行器不能并行工作造成的?或者工作节点中缺少什么?
来自工作进程的错误消息:

  1. 19/05/13 11:36:44 ERROR ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
  2. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
  3. at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
  4. at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
  5. at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:642)
  6. at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:166)
  7. at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:769)
  8. at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:766)
  9. at java.security.AccessController.doPrivileged(Native Method)
  10. at javax.security.auth.Subject.doAs(Subject.java:422)
  11. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
  12. at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:766)
  13. at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:920)
  14. at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:889)
  15. at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1222)
  16. at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
  17. at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
  18. at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
  19. at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:201)
  20. at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:180)
  21. at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
  22. at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:346)
  23. at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:320)
  24. at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
  25. at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
  26. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  27. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  28. at java.lang.Thread.run(Thread.java:745)
  29. Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
  30. at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
  31. at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
  32. at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
  33. at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
  34. at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
  35. at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
  36. at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
  37. ... 25 more

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题