map reduce over hbase出错,为什么?

kmpatx3s  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(370)

我有下面的工作,还有很多其他喜欢的,就是这样展开的

String jobName = MR.class
            .getSimpleName();
    Configuration config = HBaseConfiguration.create();

    config.set(
            "hbase.master", HBASE_MASTER);
    config.set(
            "hbase.zookeeper.quorum", HBASE_ZOOKEEPERS);
    config.set(
            "hbase.zookeeper.property.clientPort", ZK_CLIENT_PORT);
    Job job = Job.getInstance(config, jobName);

    job.setJarByClass(MR.class);
    Scan scan = new Scan();

    //gets the raw band data cells
    scan.addColumn("gc".getBytes(), "s".getBytes());
    System.out.println("Job=" + jobName);
    System.out.println("\tHBASE_MASTER=" + HBASE_MASTER);
    System.out.println(
            "\tHBASE_ZOOKEEPERS=" + HBASE_ZOOKEEPERS);
    System.out.println(
            "\tINPUT_TABLE=" + INPUT_TABLE);
    System.out.println(
            "\tOUTPUT_TABLE= " + OUTPUT_TABLE);
    System.out.println(
            "\tCACHING_LEVEL= " + CACHING_LEVEL);
    // TODO: make caching size configurable
    scan.setCaching(CACHING_LEVEL);

    scan.setCacheBlocks(
            false);

    // null for output key/value since we're not sending anything to reduce
    TableMapReduceUtil.initTableMapperJob(INPUT_TABLE, scan,
            MR.MapClass.class,
            null,
            null,
            job);

    TableMapReduceUtil.initTableReducerJob(
            "PIXELS", // output table
            null, // reducer class
            job);
    job.setNumReduceTasks(0);
    // at least one, adjust as required

    boolean b = job.waitForCompletion(true);

但是我不断地得到这个错误,甚至不能把数据读入Map器

Error: java.io.BufferedReader.lines()Ljava/util/stream/Stream;
14/09/22 22:11:13 INFO mapreduce.Job: Task Id : attempt_1410880772411_0045_m_000009_2, Status : FAILED
Error: java.io.BufferedReader.lines()Ljava/util/stream/Stream;
14/09/22 22:11:13 INFO mapreduce.Job: Task Id : attempt_1410880772411_0045_m_000018_2, Status : FAILED
Error: java.io.BufferedReader.lines()Ljava/util/stream/Stream;
14/09/22 22:11:13 INFO mapreduce.Job: Task Id : attempt_1410880772411_0045_m_000002_2, Status : FAILED

以前从未见过,使用hbase有一段时间了。。。。以下是我尝试过的:

I can scan the table via the hbase shell no problem in the same way this job does
I can use the java api to scan in the same way no problem
I built and rebuilt my jar thinking some files were corrupt... not the problem
I have at least 5 other jobs setup the same way
I disabled, enabled, compacted, rebuilt everything you can imagine with the table
I am using maven shade to uber jar
I am running HBase 0.98.1-cdh5.1.0

非常感谢任何帮助或想法。

fykwrbwg

fykwrbwg1#

看来我应该等一下再问这个问题。问题是我实际上是在获取数据,但是我的一个外部jar是用java8构建的,而且.lines()在java7的bufferedreader中不存在,我正在运行java7,因此出现了错误。与hbase或map reduce无关。

相关问题