我想用pig中的udf对一列和另一列进行分类。我有数据
Id,name,age1,jhon,312,adi,153,sam,254,lina,28
Id,name,age
1,jhon,31
2,adi,15
3,sam,25
4,lina,28
预期产量
1,jhon,31,30-352,adi,15,10-153,sam,25,20-254,lina,28,25-30
1,jhon,31,30-35
2,adi,15,10-15
3,sam,25,20-25
4,lina,28,25-30
请建议
dpiehjr41#
你不需要自定义项就可以做到这一点。假设您已将数据加载到关系a。
B = FOREACH A GENERATE A.Id,A.name,A.age,(A.age%5 == 0 ? A.age-5 : (A.age/5)*5) as lower_age,(A.age%5 == 0 ? A.age : ((A.age/5)*5) + 5) as upper_age;C = FOREACH B GENERATE B.Id,B.name,B.age,CONCAT(CONCAT((chararray)lower_age,'-'),(chararray)upper_age);DUMP C;
B = FOREACH A GENERATE A.Id,A.name,A.age,(A.age%5 == 0 ? A.age-5 : (A.age/5)*5) as lower_age,(A.age%5 == 0 ? A.age : ((A.age/5)*5) + 5) as upper_age;
C = FOREACH B GENERATE B.Id,B.name,B.age,CONCAT(CONCAT((chararray)lower_age,'-'),(chararray)upper_age);
DUMP C;
eaf3rand2#
您可以在eclipse中创建pig udf在eclipse中使用pig jars创建一个项目,并尝试以下代码
package com;import java.io.IOException;import org.apache.pig.EvalFunc;import org.apache.pig.backend.executionengine.ExecException;import org.apache.pig.data.Tuple;public class Age extends EvalFunc<String>{ @Override public String exec(Tuple a) throws IOException { // TODO Auto-generated method stub if(a == null || a.size() == 0){ return null; } try{ Object object = a.get(0); if(object == null){ return null; } int i = (Integer) object; if(i >= 10 && i <= 20 ){ return "10-20"; } else if (i >= 21 && i <= 30){ return "20-30"; } else return ">30"; } catch (ExecException e){ throw new IOException(e); } }}
package com;
import java.io.IOException;
import org.apache.pig.EvalFunc;
import org.apache.pig.backend.executionengine.ExecException;
import org.apache.pig.data.Tuple;
public class Age extends EvalFunc<String>{
@Override
public String exec(Tuple a) throws IOException {
// TODO Auto-generated method stub
if(a == null || a.size() == 0){
return null;
}
try{
Object object = a.get(0);
if(object == null){
int i = (Integer) object;
if(i >= 10 && i <= 20 ){
return "10-20";
else if (i >= 21 && i <= 30){
return "20-30";
else
return ">30";
} catch (ExecException e){
throw new IOException(e);
现在将项目导出为jar并在pig shell中注册
REGISTER <path of your .jar file>
用包和类来定义它。
DEFINE U com.Age();a = LOAD '<input path>' using PigStorage(',') as (id:int,name:chararray,age:int);b = FOREACH a GENERATE id,name,age,U(age);
DEFINE U com.Age();
a = LOAD '<input path>' using PigStorage(',') as (id:int,name:chararray,age:int);
b = FOREACH a GENERATE id,name,age,U(age);
2条答案
按热度按时间dpiehjr41#
你不需要自定义项就可以做到这一点。假设您已将数据加载到关系a。
eaf3rand2#
您可以在eclipse中创建pig udf
在eclipse中使用pig jars创建一个项目,并尝试以下代码
现在将项目导出为jar并在pig shell中注册
用包和类来定义它。