尝试运行作业以通过spark从hbase表提取数据时出现以下错误。非常感谢您的帮助。
错误:
org.apache.hadoop.hbase.donotretryioexception:java.lang.illegalaccesserror:尝试访问com.google.common.base.stopwatch.()v类org.apache.hadoop.hbase.zookeeper.metatablelocator
Hadoop Version: 3.1.1
HBase Version: 2.1.6
SPARK Version: 2.3.2
spark-submit --master yar --conf spark.executor.extraClassPath=/etc/hbase/conf --conf spark.driver.extraClassPath=/etc/hbase/conf --class CLASS_NAME --packages com.hortonworks:shc-core:1.1.1-2.1-s_2.11 --repositories http://repo.hortonworks.com/content/groups/public/ JAR_FILE_NAME
pom相关性:
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.3.2</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.11.11</version>
</dependency>
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.3.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.25</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.25</version>
</dependency>
<dependency>
<groupId>it.nerdammer.bigdata</groupId>
<artifactId>spark-hbase-connector_2.10</artifactId>
<version>1.0.3</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-common -->
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-common</artifactId>
<version>2.1.6</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-client -->
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>2.1.6</version>
</dependency>
2条答案
按热度按时间9udxz4iz1#
您需要运行hbase和hadoop的兼容版本。
由于hadoop版本是3.1.1,请尝试使用
HBase-2.2.x
或者HBase-2.3.x
hbase版本而不是2.1.x
.您可以在文档中找到hbase hadoop版本支持列表。
f5emj3cl2#
我有个突破口。
太简单了。只需在spark submit命令中添加一个语句——jars/usr/hdp/current/hbase client/lib/shaded clients/hbase-shaded-mapreduce-2.1.6.3.1.5.0-152.jar。
谢谢!