我有下面的工作,还有很多其他喜欢的,就是这样展开的
String jobName = MR.class
.getSimpleName();
Configuration config = HBaseConfiguration.create();
config.set(
"hbase.master", HBASE_MASTER);
config.set(
"hbase.zookeeper.quorum", HBASE_ZOOKEEPERS);
config.set(
"hbase.zookeeper.property.clientPort", ZK_CLIENT_PORT);
Job job = Job.getInstance(config, jobName);
job.setJarByClass(MR.class);
Scan scan = new Scan();
//gets the raw band data cells
scan.addColumn("gc".getBytes(), "s".getBytes());
System.out.println("Job=" + jobName);
System.out.println("\tHBASE_MASTER=" + HBASE_MASTER);
System.out.println(
"\tHBASE_ZOOKEEPERS=" + HBASE_ZOOKEEPERS);
System.out.println(
"\tINPUT_TABLE=" + INPUT_TABLE);
System.out.println(
"\tOUTPUT_TABLE= " + OUTPUT_TABLE);
System.out.println(
"\tCACHING_LEVEL= " + CACHING_LEVEL);
// TODO: make caching size configurable
scan.setCaching(CACHING_LEVEL);
scan.setCacheBlocks(
false);
// null for output key/value since we're not sending anything to reduce
TableMapReduceUtil.initTableMapperJob(INPUT_TABLE, scan,
MR.MapClass.class,
null,
null,
job);
TableMapReduceUtil.initTableReducerJob(
"PIXELS", // output table
null, // reducer class
job);
job.setNumReduceTasks(0);
// at least one, adjust as required
boolean b = job.waitForCompletion(true);
但是我不断地得到这个错误,甚至不能把数据读入Map器
Error: java.io.BufferedReader.lines()Ljava/util/stream/Stream;
14/09/22 22:11:13 INFO mapreduce.Job: Task Id : attempt_1410880772411_0045_m_000009_2, Status : FAILED
Error: java.io.BufferedReader.lines()Ljava/util/stream/Stream;
14/09/22 22:11:13 INFO mapreduce.Job: Task Id : attempt_1410880772411_0045_m_000018_2, Status : FAILED
Error: java.io.BufferedReader.lines()Ljava/util/stream/Stream;
14/09/22 22:11:13 INFO mapreduce.Job: Task Id : attempt_1410880772411_0045_m_000002_2, Status : FAILED
以前从未见过,使用hbase有一段时间了。。。。以下是我尝试过的:
I can scan the table via the hbase shell no problem in the same way this job does
I can use the java api to scan in the same way no problem
I built and rebuilt my jar thinking some files were corrupt... not the problem
I have at least 5 other jobs setup the same way
I disabled, enabled, compacted, rebuilt everything you can imagine with the table
I am using maven shade to uber jar
I am running HBase 0.98.1-cdh5.1.0
非常感谢任何帮助或想法。
1条答案
按热度按时间fykwrbwg1#
看来我应该等一下再问这个问题。问题是我实际上是在获取数据,但是我的一个外部jar是用java8构建的,而且.lines()在java7的bufferedreader中不存在,我正在运行java7,因此出现了错误。与hbase或map reduce无关。