扁平Pig内袋

hfyxw5xn  于 2021-06-21  发布在  Pig
关注(0)|答案(1)|浏览(418)

我有这个:

(1,{(1,2,3)})

(4,{(4,2,1),(4,3,3)})

(8,{(8,3,4)})

我想要这个:

(1,1,2,3)

(4,4,2,1)

(4,4,3,3)

(8,8,3,4)

我发现这篇文章很有用,但没能让它发挥作用:如何在pig中将一个组扁平化为一个元组?这看起来很容易,但我对Pig还不熟悉。提前谢谢!
编辑以回答reo的问题(感谢reo):
在这里发布之前,我试着把我的问题放平,结果得到了以下错误(这里的例子中d是c):

ERROR 2117: Unexpected error when launching map reduce job.

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias d
        at org.apache.pig.PigServer.openIterator(PigServer.java:857)
        at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:682)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
        at org.apache.pig.Main.run(Main.java:490)
        at org.apache.pig.Main.main(Main.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias d
        at org.apache.pig.PigServer.storeEx(PigServer.java:956)
        at org.apache.pig.PigServer.store(PigServer.java:919)
        at org.apache.pig.PigServer.openIterator(PigServer.java:832)
        ... 12 more
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2117: Unexpected error when launching map reduce job.
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:322)
        at org.apache.pig.PigServer.launchPlan(PigServer.java:1270)
        at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1255)
        at org.apache.pig.PigServer.storeEx(PigServer.java:952)
        ... 14 more
Caused by: java.lang.RuntimeException: Could not resolve error that occured when launching map reduce job: java.lang.NullPointerException
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecsHelper(PigOutputFormat.java:193)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:187)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:937)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1117)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:896)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:870)
        at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
        at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
        at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
        at java.lang.Thread.run(Thread.java:662)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)

        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$JobControlThreadExceptionHandler.uncaughtException(MapReduceLauncher.java:631)
        at java.lang.Thread.dispatchUncaughtException(Thread.java:1874)

那是我最初的问题。但现在我在这里使用的可复制的例子上尝试了同样的方法,它是有效的。

grunt> A = load 'my_location_on_HDFS' using PigStorage('|') as (a,b,c);
grunt> B = group A by a;
grunt> describe B;
B: {group: bytearray,A: {(a: bytearray,b: bytearray,c: bytearray)}}
grunt> dump B;
(1,{(1,2,3)})
(4,{(4,2,1),(4,3,3)})
(8,{(8,3,4)})
grunt> C = foreach B generate group, flatten(A);
grunt> dump C;
(1,1,2,3)
(4,4,2,1)
(4,4,3,3)
(8,8,3,4)

我想我得去仔细看看我的原始数据/情况。

50pmv0ei

50pmv0ei1#

如果你把袋子压平,那么你会得到想要的输出让我们假设你上面的输入是b,你通过分组a的输入得到这个
就做一个描述b;

Describe B;    
B: {group: bytearray,A: {()}}

这是一个包,如果你可以试试

C= foreach B generate group, FLATTEN(A);

c将有你想要的值,扁平一个袋子将导致单独的行

相关问题