pig group-无法获取多个字段

7y4bm7vi 于 2021-05-29 发布在 Hadoop

关注(0)|答案(3)|浏览(375)

我想知道每个部门拿最高工资的人是谁-我拿到了每个部门的最高工资，但没能把名字写进去。。。。用pig脚本附加文件

EmpData = LOAD '/data/EmpDet3.csv' using PigStorage(',') as 
(fname:chararray,lname:chararray,position:chararray,dept:chararray, salary:chararray);
Grp_Dept = GROUP EmpData by dept;
EmpDataC = FOREACH EmpData GENERATE fname,lname,position,dept,(int)SUBSTRING(salary,1,10) as sal;
Group_Pos = GROUP EmpDataC by position;
Max_Sal = FOREACH Group_Pos GENERATE group,MAX(EmpDataC.sal);
dump Max_Sal;

我越来越

(FIRE ENGINEER,103331)
(POLICE OFFICER,90778)
(POLICE OFFICER2,86520)
(WATER RATE TAKER,88968)
(CIVIL ENGINEER IV,104736)
(ELECTRICAL MECHANIC,91520)
(ASST TO THE ALDERMAN,70764)
(GENERAL LABORER - DSS,40560)
(CHIEF CONTRACT EXPEDITER,84780)

我需要的细节以及人的名字

(FIRE ENGINEER,Dudolfi,103331)
(POLICE OFFICER,AARON,90778)
(POLICE OFFICER2,ABBATE,86520)
(WATER RATE TAKER,AARON,88968)
(CIVIL ENGINEER IV,ABAD JR,104736)
(ELECTRICAL MECHANIC,ABBATACOLA,91520)
(ASST TO THE ALDERMAN,ABARCA,70764)
(GENERAL LABORER - DSS,ABARCA,40560)
(CHIEF CONTRACT EXPEDITER,AARON,84780)

我的输入文件

"AARON,  ELVIA J",WATER RATE TAKER,WATER MGMNT,"$88,968.00 "
"AARON,  JEFFERY M",POLICE OFFICER,POLICE,"$80,778.00 "
"AARON,  KARINA",POLICE OFFICER,POLICE,"$90,778.00 "
"AARON,  KIMBERLEI R",CHIEF CONTRACT EXPEDITER,GENERAL SERVICES,"$84,780.00 "
"ABAD JR,  VICENTE M",CIVIL ENGINEER IV,WATER MGMNT,"$104,736.00 "
"ABARCA,  ANABEL",ASST TO THE ALDERMAN,CITY COUNCIL,"$70,764.00 "
"ABARCA,  EMMANUEL",GENERAL LABORER - DSS,STREETS & SAN,"$40,560.00 "
"ABBATACOLA,  ROBERT J",ELECTRICAL MECHANIC,AVIATION,"$91,520.00 "
"ABBATEMARCO,  JAMES J",FIRE ENGINEER,FIRE,"$90,456.00 "
"ABBATE,  TERRY M",POLICE OFFICER2,POLICE,"$86,520.00 "
"XXRON,  KINA",POLICE OFFICER2,POLICE,"$50,778.00 "
"Dudolfi, Cris",FIRE ENGINEER,FIRE,"$103,331.00 "

hadoop apache-pig

来源：https://stackoverflow.com/questions/31238687/pig-group-not-able-to-get-multiple-fields

3条答案

按热度按时间

irtuqstp1#

好啊。。这会有用的。但也要记住，同一职位下的两个人可以有相同的工资，如果工资是最高的，那么下面的脚本也会生成他们的记录

Emp_Data = LOAD '/data/EmpDet3.csv' using PigStorage(',') as 
(fname:chararray,lname:chararray,position:chararray,dept:chararray, salary:chararray);
Grp_Dept = GROUP Emp_Data by dept;
Emp_DataC = FOREACH Emp_Data GENERATE fname,lname,position,dept,(int)SUBSTRING(salary,1,10) as sal;
Group_Pos = GROUP Emp_DataC by position;
Pos_max_sal = FOREACH Group_Pos GENERATE group as pos,MAX(EmpDataC.sal) as highest_sal;
Emp_max_sal = join  Emp_Data by (position,salary),Pos_max_sal by (pos,highest_sal)
final_set = foreach Emp_max_sal generate position, CONCAT(fname,lname) as emp_name, salary

赞(0）回复(0）举报 2021-05-30

2exbekwf2#

--谢谢nihal bhagchandani，bellow你的美尼顿工作了

EmpData = LOAD '/sachin/emp' using PigStorage(',') as (fname:chararray,lname:chararray,position:chararray,dept:chararray, salary:int);
Grp_Dept = GROUP EmpData by dept; 
EmpDataC = FOREACH EmpData GENERATE fname,lname,position,dept,salary as sal;
Group_Pos = GROUP EmpDataC by position; 
Max_Sal = FOREACH Group_Pos GENERATE group,MAX(EmpDataC.sal) as SalMax; 
filterMainData = JOIN EmpData BY salary, Max_Sal by SalMax; 
filterData = FOREACH filterMainData GENERATE EmpData::position as position, EmpData::fname as fname, EmpData::salary as salary; 
orderedData = ORDER filterData BY salary DESC; 
dump orderedData;

赞(0）回复(0）举报 2021-05-30

mzaanser3#

最好将这两个字段（department和position）分组并转换为元组。
像这样：

Emp_DataC = FOREACH Emp_Data GENERATE fname,lname,position,dept,(int)SUBSTRING(salary,1,10) as sal;
group_data = GROUP Emp_DataC by (dept,position);
tuple_data = foreach group_data generate group as tuple_name:TUBLE(dept as dept:chararray,position as position:chararray),MAX(EmpDataC.sal) as highest_sal;
data = foreach tuple_data generate tuple_name.dept as dept,tuple_name.position as position,highest_sal;

赞(0）回复(0）举报 2021-05-30

我来回答

pig group-无法获取多个字段

3条答案

相关问题

热门标签

最新问答