mapreduce:hive data 到hfile

bq9c1y66  于 2021-07-15  发布在  Hadoop
关注(0)|答案(0)|浏览(306)

我正在设置mapreduce作业配置,如下所示:

  1. Configuration conf = HBaseConfiguration.create();
  2. conf.set("hbase.zookeeper.quorum", "............");
  3. conf.addResource(new Path("/etc/hbase/hbase-conf/hbase-site.xml"));
  4. conf.addResource(new Path("/etc/hbase/hbase-conf/hdfs-site.xml"));
  5. conf.addResource(new Path("/etc/hbase/hbase-conf/core-site.xml"));
  6. conf.set("mapreduce.input.fileinputformat.input.dir.recursive", "true");
  7. conf.set(TableOutputFormat.OUTPUT_TABLE, hbaseTable);
  8. Kerberos.authKrb5("spade", conf);
  9. Job job = Job.getInstance(conf);
  10. Connection conn = ConnectionFactory.createConnection(conf);
  11. Table table = conn.getTable(TableName.valueOf(hbaseTable));
  12. job.setJarByClass(BulkLoadDriver.class);
  13. job.setInputFormatClass(TextInputFormat.class);
  14. job.setMapperClass(BulkLoadMapper.class);
  15. job.setMapOutputKeyClass(ImmutableBytesWritable.class);
  16. job.setMapOutputValueClass(Put.class);
  17. job.setNumReduceTasks(0);
  18. job.setOutputFormatClass(HFileOutputFormat2.class);
  19. HFileOutputFormat2.configureIncrementalLoad(job, table, conn.getRegionLocator(TableName.valueOf(hbaseTable)));
  20. FileInputFormat.setInputPaths(job, new Path(srcPath));
  21. HFileOutputFormat2.setOutputPath(job, new Path(descPath));
  22. System.exit(job.waitForCompletion(true) ? 0 : 1);

作业失败,出现以下异常:

  1. 21/02/03 19:46:26 INFO impl.YarnClientImpl: Submitted application application_1600770106038_2663097
  2. 21/02/03 19:46:26 INFO mapreduce.Job: The url to track the job: http://hb21-bd-master-130-21:8088/proxy/application_1600770106038_2663097/
  3. 21/02/03 19:46:26 INFO mapreduce.Job: Running job: job_1600770106038_2663097
  4. 21/02/03 19:46:33 INFO mapreduce.Job: Job job_1600770106038_2663097 running in uber mode : false
  5. 21/02/03 19:46:33 INFO mapreduce.Job: map 0% reduce 0%
  6. 21/02/03 19:46:45 INFO mapreduce.Job: map 100% reduce 0%
  7. 21/02/03 19:46:52 INFO mapreduce.Job: Task Id : attempt_1600770106038_2663097_r_000000_0, Status : FAILED
  8. Error: java.lang.IllegalArgumentException: Configuration parameter hbase.mapreduce.hfileoutputformat.table.name cannot be empty
  9. at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.createRecordWriter(HFileOutputFormat2.java:205)
  10. at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.getRecordWriter(HFileOutputFormat2.java:188)
  11. at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:540)
  12. at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614)
  13. at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
  14. at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
  15. at java.security.AccessController.doPrivileged(Native Method)
  16. at javax.security.auth.Subject.doAs(Subject.java:422)
  17. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
  18. at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

如您所见,jobs的map阶段执行正确,但reduce失败。我试过设置 conf.set(TableOutputFormat.Output_Table, HBaseTable) 但似乎没有帮助。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题