无法使用hbase和hadoop在map函数中执行“put”

mbjcgjjk  于 2021-05-30  发布在  Hadoop
关注(0)|答案(1)|浏览(389)

各位。我正在用mr处理一些日志文件,文件在hdfs上。我想从文件中检索一些信息并将它们存储到hbase。
所以我开始工作

  1. HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar crm_hbase-1.0.jar /datastream/music/useraction/2014-11-30/music_useraction_20141130-230003072+0800.24576015364769354.00018022.lzo

如果我只是以“hadoop jar x”运行作业,它会显示“not find hbaseconfiguration”
我的代码很简单,

  1. public int run(String[] strings) throws Exception {
  2. Configuration config = HBaseConfiguration.create();
  3. String kerbConfPrincipal = "ndir@HADOOP.HZ.NETEASE.COM";
  4. String kerbKeytab = "/srv/zwj/ndir.keytab";
  5. UserGroupInformation.loginUserFromKeytab(kerbConfPrincipal, kerbKeytab);
  6. UserGroupInformation ugi = UserGroupInformation.getLoginUser();
  7. System.out.println(" auth: " + ugi.getAuthenticationMethod());
  8. System.out.println(" name: " + ugi.getUserName());
  9. System.out.println(" using keytab:" + ugi.isFromKeytab());
  10. HBaseAdmin.checkHBaseAvailable(config);
  11. //set job name
  12. Job job = new Job(config, "Import from file ");
  13. job.setJarByClass(LogRun.class);
  14. //set map class
  15. job.setMapperClass(LogMapper.class);
  16. //set output format and output table name
  17. job.setOutputFormatClass(TableOutputFormat.class);
  18. job.getConfiguration().set(TableOutputFormat.OUTPUT_TABLE, "crm_data");
  19. job.setOutputKeyClass(ImmutableBytesWritable.class);
  20. job.setOutputValueClass(Put.class);
  21. job.setNumReduceTasks(0);
  22. TableMapReduceUtil.addDependencyJars(job);

但是当我试图运行这个mr时,我无法执行“context.write(null,put)”,似乎“map”在这一行停止了。我认为这与“kerbkeytab”有关,是不是意味着我在运行“map”过程时需要“login”

sgtfey8w

sgtfey8w1#

添加tablemapreduceutil后,它就可以工作了

  1. Job job = new Job(config, "Import from file ");
  2. job.setJarByClass(LogRun.class);
  3. //set map class
  4. job.setMapperClass(LogMapper.class);
  5. TableMapReduceUtil.initTableReducerJob(table, null, job);
  6. job.setNumReduceTasks(0);
  7. TableMapReduceUtil.addDependencyJars(job);
  8. FileInputFormat.setInputPaths(job,input);
  9. //FileInputFormat.addInputPath(job, new Path(strings[0]));
  10. int ret = job.waitForCompletion(true) ? 0 : 1;

相关问题