我想使用hortonworks lib从spark连接到hbase(https://mvnrepository.com/artifact/com.hortonworks/shc/1.1.1-2.1-s_2.11). 我将配置文件添加到spark(hbase site.xml)中,该文件的属性与我的hbase完全相同。
以下是完整命令:
spark-shell --jars /home/me/Downloads/spark-hbase-connector_2.10-1.0.3.jar,/home/me/Downloads/hbase-spark-2.0.2.3.1.0.6-1.jar,/home/me/Downloads/spark-sql_2.12-2.4.7.jar,/home/me/Downloads/shc-core-1.1.1-2.1-s_2.11.jar,/home/me/Downloads/audience-annotations-0.13.0.jar --packages org.apache.hbase:hbase:2.4.1 --files /usr/local/spark/conf/hbase-site.xml
我尝试按照hortonworks insert方法的示例:https://github.com/sparkbyexamples/spark-hbase-hortonworks-examples/blob/master/src/main/scala/com/sparkbyexamples/spark/dataframe/hbase/hortonworks/hbasesparkinsert.scala
我得到以下错误:
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/NamespaceNotFoundException
at org.apache.spark.sql.execution.datasources.hbase.DefaultSource.createRelation(HBaseRelation.scala:59)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:83)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:81)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:677)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:677)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:677)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:286)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:272)
... 51 elided
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.NamespaceNotFoundException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
示例类中指定的命名空间是“default”。在我的hbase中,我还得到:
NAMESPACE
default
hbase
如果错误是namespacenotfoundexception,但是我有这个名称空间,那么是什么问题导致了错误?
暂无答案!
目前还没有任何答案,快来回答吧!