hive udf在select中抛出类未找到异常

kt06eoxx  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(486)

我被困在使用自定义框。我需要在我的udf中解析简单的useragent。我找到了一个流行的useragent解析器http://www.bitwalker.eu/software/user-agent-utils 包括在我的项目中。在项目中，我使用maven。我添加了所有依赖项，实现了所有东西并对其进行了测试。它在我的本地机器上工作得很好。接下来，我将在maven中进行干净的安装，以构建jar。我通过addjar{myjarname}在配置单元中使用这个jar，然后创建一个函数：create temporary function{functionname}as{pathtoudfclass}，得到了如下异常。

  1. Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"bidid":"8326c0ec49e5746f1af03400f37e5797","tstamp":20131022185001163,"logtype":1
  2. ,"ipinyouid":"D89E8S5bwWz","useragent":"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; InfoPath.2)","ip":"61.138.253.*","regionid":374,"cityid":375,"adexchange":1
  3. ,"domain":"449a7568331085d43d5867de26ce1ee1","url":"5ecba5b62bafd3428cdc1398b40cf88f","anonymousurl":"null","adslotid":null,"adslotwidth":300,"adslotheight":250,"adslotvisibility":"Na","adslotformat":"Na","adslo
  4. tfloorprice":0,"creativeid":"10722","biddingprice":294,"payingprice":135,"landingpageurl":"null","advertiserid":2821,"userprofileids":[10006,10110,10063]}                                                         
  5.         at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)                                                                                                                  
  6.         at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)                                                                                                                  
  7.         at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:325)                                                                                                                  
  8.         at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)                                                                                                        
  9.         ... 14 more                                                                                                                                                                                                
  10. Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"bidid":"8326c0ec49e5746f1af03400f37e5797","tstamp":20131022185001163,"logtype":1,"ipinyouid":"D89E8S5bwWz","
  11. useragent":"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; InfoPath.2)","ip":"61.138.253.*","regionid":374,"cityid":375,"adexchange":1,"domain":"449a7568331085d43
  12. d5867de26ce1ee1","url":"5ecba5b62bafd3428cdc1398b40cf88f","anonymousurl":"null","adslotid":null,"adslotwidth":300,"adslotheight":250,"adslotvisibility":"Na","adslotformat":"Na","adslotfloorprice":0,"creativeid":
  13. "10722","biddingprice":294,"payingprice":135,"landingpageurl":"null","advertiserid":2821,"userprofileids":[10006,10110,10063]}                                                                                     
  14.         at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:565)                                                                                                                                
  15.         at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)                                                                                                                  
  16.         ... 17 more                                                                                                                                                                                                
  17. Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public org.apache.hadoop.io.Text hive.homework3.UserAgentDetector.evaluate(org.apache.hadoop.io.Text)  on object hive.homewor
  18. k3.UserAgentDetector@1b340ab of class hive.homework3.UserAgentDetector with arguments {Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; InfoPath.2):org.apache.hadoo
  19. p.io.Text} of size 1                                                                                                                                                                                               
  20.         at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:1019)                                                                                                                      
  21.         at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:182)                                                                                                              
  22.         at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)                                                                                            
  23.         at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)                                                                                                                    
  24.         at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)                                                                                                                    
  25.         at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:81)                                                                                                                           
  26.         at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:841)                                                                                                                                      
  27.         at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:133)                                                                                                                    
  28.         at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:170)                                                                                                                       
  29.         at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:555)                                                                                                                                
  30.         ... 18 more                                                                                                                                                                                                
  31. Caused by: java.lang.reflect.InvocationTargetException                                                                                                                                                             
  32.         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)                                                                                                                                             
  33.         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)                                                                                                                           
  34.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)                                                                                                                   
  35.         at java.lang.reflect.Method.invoke(Method.java:498)                                                                                                                                                        
  36.         at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:995)                                                                                                                       
  37.         ... 27 more                                                                                                                                                                                                
  38. Caused by: java.lang.NoClassDefFoundError: eu/bitwalker/useragentutils/UserAgent                                                                                                                                   
  39.         at hive.homework3.UserAgentDetector.formatter(UserAgentDetector.java:30)                                                                                                                                   
  40.         at hive.homework3.UserAgentDetector.evaluate(UserAgentDetector.java:22)                                                                                                                                    
  41.         ... 32 more                                                                                                                                                                                                
  42. Caused by: java.lang.ClassNotFoundException: eu.bitwalker.useragentutils.UserAgent                                                                                                                                 
  43.         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)                                                                                                                                              
  44.         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)                                                                                                                                                   
  45.         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)                                                                                                                                           
  46.         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)                                                                                                                                                   
  47.         ... 34 more                                                                                                                                                                                                
  48. ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:21, Vertex vertex_1501829365845_0009_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. fa
  49. iledVertices:1 killedVertices:0

从这些我了解到最重要的是：

  1. Caused by: java.lang.ClassNotFoundException: eu.bitwalker.useragentutils.UserAgent                                                                                                                                 
  2.         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)                                                                                                                                              
  3.         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)                                                                                                                                                   
  4.         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)                                                                                                                                           
  5.         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)                                                                                                                                                   
  6.         ... 34 more

我在maven项目中使用的这个外部库。
这是自定义项创建，顺便说一句，在当地的作品一切罚款和测试都通过了。但在 hive 里它不起作用。我想我使用的这个lib有问题，但是如果我在本地工作的话有可能吗？

  1. import eu.bitwalker.useragentutils.UserAgent;
  2. import org.apache.hadoop.hive.ql.exec.Description;
  3. import org.apache.hadoop.hive.ql.exec.UDF;
  4. import org.apache.hadoop.io.Text;
  6. @Description(
  7.         name = "agentdetector",
  8.         value = "_FUNC_(str) - detects a user-agent of user",
  9.         extended = "Example:\n" +
  10.                 " > SELECT agent(line) FROM test ipy; \n"
  11. )
  12. public class UserAgentDetector extends UDF {
  14.     public Text evaluate(Text text) {
  15.         Text value = new Text("");
  16.         if (text != null) {
  17.             value.set(formatter(text));
  18.             return value;
  19.         } else {
  20.             return null;
  21.         }
  22.     }
  24.     private Text formatter(Text text) {
  25.         UserAgent userAgent = UserAgent.parseUserAgentString(text.toString());
  26.         StringBuilder builder = new StringBuilder();
  27.         builder.append("Browser : ").append(userAgent.getBrowser().getName()).append("\n");
  28.         text.set(builder.toString());
  29.         return text;
  30.     }

maven的依赖关系是：

  1. <dependency>
  2.             <groupId>eu.bitwalker</groupId>
  3.             <artifactId>UserAgentUtils</artifactId>
  4.             <version>1.20</version>
  5.   </dependency>
JavahadoopHiveuser-defined-functionsJar

1条答案

按热度按时间
6gpjuf90

6gpjuf901#

要解决这个问题，只需在pom.xml中添加插件

  1. <plugin>
  2.                 <artifactId>maven-assembly-plugin</artifactId>
  3.                 <configuration>
  4.                     <archive>
  5.                         <manifest>
  6.                             <mainClass>{pathToMainClass}</mainClass>
  7.                         </manifest>
  8.                     </archive>
  9.                     <descriptorRefs>
  10.                         <descriptorRef>jar-with-dependencies</descriptorRef>
  11.                     </descriptorRefs>
  12.                 </configuration>
  13.             </plugin>

然后跑assembly:assembly in maven的插件。

赞(0）分享  回复(0）举报  2021-05-29
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com