我正在尝试使用elephant bird 4.1将协议缓冲区中编码的数据处理为pig表示。相关清管器代码:
REGISTER '/home/ubuntu/elephant-bird/core/target/elephant-bird-core-4.1-SNAPSHOT.jar';
REGISTER '/home/ubuntu/elephant-bird/pig/target/elephant-bird-pig-4.1-SNAPSHOT.jar';
REGISTER '/home/ubuntu/elephant-bird/hadoop-compat/target/elephant-bird-hadoop-compat-4.1-SNAPSHOT.jar';
REGISTER '/home/ubuntu/libraryContainingProto.jar';
DEFINE ProtobufBytesToTuple com.twitter.elephantbird.pig.piggybank.ProtobufBytesToTuple('com.my.converted.proto.class');
raw_protobuffer = LOAD 'hbase://table' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:cl') AS (downloader:chararray);
parsed = FOREACH raw_protobuffer GENERATE ProtobufBytesToTuple($0);
每当我尝试进行转换时,它都会失败,出现以下堆栈跟踪:
Caused by: java.lang.RuntimeException: could not instantiate 'com.twitter.elephantbird.pig.piggybank.ProtobufBytesToTuple' with arguments '[com.my.converted.proto.class]'
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:618)
at org.apache.pig.newplan.logical.expression.UserFuncExpression.getFieldSchema(UserFuncExpression.java:193)
at org.apache.pig.newplan.logical.optimizer.FieldSchemaResetter.execute(SchemaResetter.java:264)
at org.apache.pig.newplan.logical.expression.AllSameExpressionVisitor.visit(AllSameExpressionVisitor.java:143)
at org.apache.pig.newplan.logical.expression.UserFuncExpression.accept(UserFuncExpression.java:88)
at org.apache.pig.newplan.ReverseDependencyOrderWalker.walk(ReverseDependencyOrderWalker.java:70)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visitAll(SchemaResetter.java:67)
at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:122)
at org.apache.pig.newplan.logical.relational.LOGenerate.accept(LOGenerate.java:246)
at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:114)
at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:76)
at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at org.apache.pig.parser.LogicalPlanBuilder.expandAndResetVisitor(LogicalPlanBuilder.java:392)
at org.apache.pig.parser.LogicalPlanBuilder.buildForeachOp(LogicalPlanBuilder.java:924)
at org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:14195)
"~/pig_1379411496557.log" [readonly] 396L, 30186C 13,1-8 Top
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:618)
at org.apache.pig.newplan.logical.expression.UserFuncExpression.getFieldSchema(UserFuncExpression.java:193)
at org.apache.pig.newplan.logical.optimizer.FieldSchemaResetter.execute(SchemaResetter.java:264)
at org.apache.pig.newplan.logical.expression.AllSameExpressionVisitor.visit(AllSameExpressionVisitor.java:143)
at org.apache.pig.newplan.logical.expression.UserFuncExpression.accept(UserFuncExpression.java:88)
at org.apache.pig.newplan.ReverseDependencyOrderWalker.walk(ReverseDependencyOrderWalker.java:70)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visitAll(SchemaResetter.java:67)
at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:122)
at org.apache.pig.newplan.logical.relational.LOGenerate.accept(LOGenerate.java:246)
at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:114)
at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:76)
at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at org.apache.pig.parser.LogicalPlanBuilder.expandAndResetVisitor(LogicalPlanBuilder.java:392)
at org.apache.pig.parser.LogicalPlanBuilder.buildForeachOp(LogicalPlanBuilder.java:924)
at org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:14195)
at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1623)
at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:799)
at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:517)
at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:392)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:184)
... 15 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:586)
... 37 more
Caused by: java.lang.NoClassDefFoundError: com/twitter/elephantbird/util/TypeRef
at com.twitter.elephantbird.pig.piggybank.ProtobufBytesToTuple.<init>(ProtobufBytesToTuple.java:37)
... 42 more
Caused by: java.lang.ClassNotFoundException: com.twitter.elephantbird.util.TypeRef
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 43 more
我怀疑这可能与包含适当的jar有关,从我所看到的情况来看,这些依赖应该在象鸟核心中。
2条答案
按热度按时间vlju58qv1#
当我排除了“象鸟核心”库时,我能够重现同样的错误。所以我怀疑jar有问题。
您可以通过以下命令检查jar是否具有typeref.class
3bygqnnd2#
最后,它归结为这样一个事实:仅仅使用grunt中的register并不能为集群上的计算提供库。
为了让它们正常工作,我不得不(在我的例子中)将elephant bird pig、elephant bird core和elephant bird hadoop compat放到/usr/lib/hadoop目录下。