org.apache.hadoop.mapred.Mapper.configure()方法的使用及代码示例

x33g5p2x  于2022-01-25 转载在 其他  
字(3.7k)|赞(0)|评价(0)|浏览(241)

本文整理了Java中org.apache.hadoop.mapred.Mapper.configure()方法的一些代码示例,展示了Mapper.configure()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。Mapper.configure()方法的具体详情如下:
包路径:org.apache.hadoop.mapred.Mapper
类名称:Mapper
方法名:configure

Mapper.configure介绍

暂无

代码示例

代码示例来源:origin: apache/flink

  1. @Override
  2. public void open(Configuration parameters) throws Exception {
  3. super.open(parameters);
  4. this.mapper.configure(jobConf);
  5. this.reporter = new HadoopDummyReporter();
  6. this.outputCollector = new HadoopOutputCollector<KEYOUT, VALUEOUT>();
  7. }

代码示例来源:origin: org.apache.crunch/crunch-core

  1. @Override
  2. public void initialize() {
  3. if (instance == null) {
  4. this.instance = ReflectionUtils.newInstance(mapperClass, getConfiguration());
  5. }
  6. instance.configure(new JobConf(getConfiguration()));
  7. outputCollector = new OutputCollectorImpl<K2, V2>();
  8. }

代码示例来源:origin: org.apache.flink/flink-hadoop-compatibility

  1. @Override
  2. public void open(Configuration parameters) throws Exception {
  3. super.open(parameters);
  4. this.mapper.configure(jobConf);
  5. this.reporter = new HadoopDummyReporter();
  6. this.outputCollector = new HadoopOutputCollector<KEYOUT, VALUEOUT>();
  7. }

代码示例来源:origin: org.apache.flink/flink-hadoop-compatibility_2.11

  1. @Override
  2. public void open(Configuration parameters) throws Exception {
  3. super.open(parameters);
  4. this.mapper.configure(jobConf);
  5. this.reporter = new HadoopDummyReporter();
  6. this.outputCollector = new HadoopOutputCollector<KEYOUT, VALUEOUT>();
  7. }

代码示例来源:origin: com.alibaba.blink/flink-hadoop-compatibility

  1. @Override
  2. public void open(Configuration parameters) throws Exception {
  3. super.open(parameters);
  4. this.mapper.configure(jobConf);
  5. this.reporter = new HadoopDummyReporter();
  6. this.outputCollector = new HadoopOutputCollector<KEYOUT, VALUEOUT>();
  7. }

代码示例来源:origin: apache/apex-malhar

  1. mapObject.configure(jobConf);

代码示例来源:origin: apache/chukwa

  1. public void testSetDefaultMapProcessor() throws IOException {
  2. Mapper<ChukwaArchiveKey, ChunkImpl, ChukwaRecordKey, ChukwaRecord> mapper =
  3. new Demux.MapClass();
  4. JobConf conf = new JobConf();
  5. conf.set("chukwa.demux.mapper.default.processor",
  6. "org.apache.hadoop.chukwa.extraction.demux.processor.mapper.MockMapProcessor,");
  7. mapper.configure(conf);
  8. ChunkBuilder cb = new ChunkBuilder();
  9. cb.addRecord(SAMPLE_RECORD_DATA.getBytes());
  10. ChunkImpl chunk = (ChunkImpl)cb.getChunk();
  11. ChukwaTestOutputCollector<ChukwaRecordKey, ChukwaRecord> output =
  12. new ChukwaTestOutputCollector<ChukwaRecordKey, ChukwaRecord>();
  13. mapper.map(new ChukwaArchiveKey(), chunk, output, Reporter.NULL);
  14. ChukwaRecordKey recordKey = new ChukwaRecordKey("someReduceType", SAMPLE_RECORD_DATA);
  15. assertEquals("MockMapProcessor never invoked - no records found", 1, output.data.size());
  16. assertNotNull("MockMapProcessor never invoked", output.data.get(recordKey));
  17. }

代码示例来源:origin: apache/chukwa

  1. public void testSetCustomeMapProcessor() throws IOException {
  2. Mapper<ChukwaArchiveKey, ChunkImpl, ChukwaRecordKey, ChukwaRecord> mapper =
  3. new Demux.MapClass();
  4. String custom_DataType = "cus_dt";
  5. JobConf conf = new JobConf();
  6. conf.set(custom_DataType,
  7. "org.apache.hadoop.chukwa.extraction.demux.processor.mapper.MockMapProcessor,");
  8. mapper.configure(conf);
  9. ChunkBuilder cb = new ChunkBuilder();
  10. cb.addRecord(SAMPLE_RECORD_DATA.getBytes());
  11. ChunkImpl chunk = (ChunkImpl)cb.getChunk();
  12. chunk.setDataType(custom_DataType);
  13. ChukwaTestOutputCollector<ChukwaRecordKey, ChukwaRecord> output =
  14. new ChukwaTestOutputCollector<ChukwaRecordKey, ChukwaRecord>();
  15. mapper.map(new ChukwaArchiveKey(), chunk, output, Reporter.NULL);
  16. ChukwaRecordKey recordKey = new ChukwaRecordKey("someReduceType", SAMPLE_RECORD_DATA);
  17. assertEquals("MockMapProcessor never invoked - no records found", 1, output.data.size());
  18. assertNotNull("MockMapProcessor never invoked", output.data.get(recordKey));
  19. }

相关文章