设置hive.exec.pre.hooks时发生classnotfoundexception

ergxz8rk  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(595)

我遵循此文档进行配置单元挂钩:
http://dharmeshkakadia.github.io/hive-hook/
但我在 show tables ```
2018-08-12 09:57:38,122 ERROR org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-315]: hive.exec.pre.hooks Class not found: HiveExampleHook
2018-08-12 09:57:38,122 ERROR org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-315]: FAILED: Hive Internal Error: java.lang.ClassNotFoundException(HiveExampleHook)
java.lang.ClassNotFoundException: HiveExampleHook
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.hadoop.hive.ql.hooks.HooksLoader.getHooks(HooksLoader.java:100)
at org.apache.hadoop.hive.ql.hooks.HooksLoader.getHooks(HooksLoader.java:64)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1674)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

2018-08-12 09:57:38,122 INFO org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-315]: </PERFLOG method=Driver.execute start=1534067858120 end=1534067858122 duration=2 from=org.apache.hadoop.hive.ql.Driver>
2018-08-12 09:57:38,122 INFO org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-315]: Completed executing command(queryId=hive_20180812095757_e6516d83-ddc9-4f82-8151-def7e7f1eb37); Time taken: 0.002 seconds
2018-08-12 09:57:38,122 INFO org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-315]:
2018-08-12 09:57:38,122 INFO org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-315]: </PERFLOG method=releaseLocks start=1534067858122 end=1534067858122 duration=0 from=org.apache.hadoop.hive.ql.Driver>
2018-08-12 09:57:38,130 ERROR org.apache.hive.service.cli.operation.Operation: [HiveServer2-Background-Pool: Thread-315]: Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Hive Internal Error: java.lang.ClassNotFoundException(HiveExampleHook)
at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238)
at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: HiveExampleHook
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.hadoop.hive.ql.hooks.HooksLoader.getHooks(HooksLoader.java:100)
at org.apache.hadoop.hive.ql.hooks.HooksLoader.getHooks(HooksLoader.java:64)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1674)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
... 11 more

我相信最后一步 `add jar target/Hive-hook-example-1.0.jar;` 这是错误的。
我尝试了以下方法:
我将jar文件放入hdfs/user/hive/:
add jar hdfs:///user/hive/hive-hook-example-1.0.jar;
我还在hiveserver2节点中将“hive auxiliary jars directory”设置为/home/centos/hiveexamplehook/target/hive-hook-example-1.0.jar,然后重新启动hive plus beeline。
将jar文件复制到 `/opt/cloudera/parcels/CDH/jars/` 将jar文件复制到 `/opt/cloudera/parcels/CDH/lib/hive/lib/` 什么都帮不了。
你知道吗?
更新1:
如果我这样做了 `LIST JARS;` 这会显示

+----------------------------------------------------+--+
| resource |
+----------------------------------------------------+--+
| /tmp/3fe67bb1-5cfd-427f-8faa-cab6524afeb3_resources/Hive-hook-example-1.0.jar |
+----------------------------------------------------+--+

我试过两种方法 `CREATE FUNCTION` 也是:

CREATE TEMPORARY FUNCTION test1 AS 'HiveExampleHook';
INFO : Compiling command(queryId=hive_20180812153838_47589f9d-eaeb-410d-80b0-9cf414ae557f): CREATE TEMPORARY FUNCTION test1 AS 'HiveExampleHook'
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling command(queryId=hive_20180812153838_47589f9d-eaeb-410d-80b0-9cf414ae557f); Time taken: 0.002 seconds
INFO : Executing command(queryId=hive_20180812153838_47589f9d-eaeb-410d-80b0-9cf414ae557f): CREATE TEMPORARY FUNCTION test1 AS 'HiveExampleHook'
INFO : Starting task [Stage-0:FUNC] in serial mode
ERROR : FAILED: Class HiveExampleHook not found
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask
INFO : Completed executing command(queryId=hive_20180812153838_47589f9d-eaeb-410d-80b0-9cf414ae557f); Time taken: 0.003 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask (state=08S01,code=1)

还有。。。

CREATE TEMPORARY FUNCTION test1 AS 'HiveExampleHook' USING JAR 'hdfs:///user/hive/Hive-hook-example-1.0.jar';
INFO : Compiling command(queryId=hive_20180812153939_cf1f31c9-0361-47dc-8903-78221bd12401): CREATE TEMPORARY FUNCTION test1 AS 'HiveExampleHook' USING JAR 'hdfs:///user/hive/Hive-hook-example-1.0.jar'
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling command(queryId=hive_20180812153939_cf1f31c9-0361-47dc-8903-78221bd12401); Time taken: 0.004 seconds
INFO : Executing command(queryId=hive_20180812153939_cf1f31c9-0361-47dc-8903-78221bd12401): CREATE TEMPORARY FUNCTION test1 AS 'HiveExampleHook' USING JAR 'hdfs:///user/hive/Hive-hook-example-1.0.jar'
INFO : Starting task [Stage-0:FUNC] in serial mode
INFO : converting to local hdfs:///user/hive/Hive-hook-example-1.0.jar
INFO : Added [/tmp/3fe67bb1-5cfd-427f-8faa-cab6524afeb3_resources/Hive-hook-example-1.0.jar] to class path
INFO : Added resources: [hdfs:///user/hive/Hive-hook-example-1.0.jar]
ERROR : FAILED: Class HiveExampleHook not found
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask
INFO : Completed executing command(queryId=hive_20180812153939_cf1f31c9-0361-47dc-8903-78221bd12401); Time taken: 0.03 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask (state=08S01,code=1)

很明显,它可以找到jar,但不能找到类名。我说得对吗?
更新2:
我试过这个:

[Hive-hook-example]# java -cp pwd/target/Hive-hook-example-1.0.jar HiveExampleHook

但还是有这个:

Error: Could not find or load main class HiveExampleHook

我相信这是我犯的愚蠢的错误。
更新3:
好吧,我知道了。您必须使用hivecli而不是beeline才能工作。

hive> add jar hdfs:///user/hive/Hive-hook-example-1.0.jar;
add jar hdfs:///user/hive/Hive-hook-example-1.0.jar
converting to local hdfs:///user/hive/Hive-hook-example-1.0.jar
Added [/tmp/0a90132d-70cd-4ef0-b4cd-e75dc823e5ca_resources/Hive-hook-example-1.0.jar] to class path
Added resources: [hdfs:///user/hive/Hive-hook-example-1.0.jar]
hive> set hive.exec.pre.hooks=HiveExampleHook;
set hive.exec.pre.hooks=HiveExampleHook
hive> show tables;
show tables
Hello from the hook !!
OK
test1
Time taken: 0.023 seconds, Fetched: 5 row(s)

所以问题是如何在直线上运行?因为配置单元cli已弃用。
更新4:
我决定这样做:
![](https://i.stack.imgur.com/P0nAc.png)
跑 `beeline` 看到这个:

2018-08-12 16:39:13,286 INFO org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-60]:
2018-08-12 16:39:13,286 INFO org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-60]: </PERFLOG method=PreHook.HiveExampleHook start=1534091953286 end=1534091953286 duration=0 from=org.apache.hadoop.hive.ql.Driver>
2

这是一些进步,虽然我不知道这意味着什么,以及是否运行类。因为我看不到任何输出。
ruyhziif

ruyhziif1#

对于beeline,在添加jar时必须使用hdfs路径。记住beeline只是一个jdbc cli,所以当您使用addjar with local path时,它会引用您的本地路径,这对于集群上运行的配置单元会话是不可访问的。
(谢谢你的邀请https://twitter.com/quanghoc/status/1028671393376874496 寻求帮助。我是你提到的博客的作者。)

相关问题