org.apache.avro.file.FileReader.sync()方法的使用及代码示例

x33g5p2x  于2022-01-19 转载在 其他  
字(7.3k)|赞(0)|评价(0)|浏览(113)

本文整理了Java中org.apache.avro.file.FileReader.sync()方法的一些代码示例，展示了FileReader.sync()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。FileReader.sync()方法的具体详情如下：
包路径：org.apache.avro.file.FileReader
类名称：FileReader
方法名：sync

FileReader.sync介绍

[英]Move to the next synchronization point after a position. To process a range of file entires, call this with the starting position, then check #pastSync(long) with the end point before each call to #next().
[中]移动到位置后的下一个同步点。要处理一系列文件实体，请使用起始位置调用它，然后在每次调用#next（）之前使用端点选中#pastSync（long）。

代码示例

代码示例来源：origin: apache/avro

protected AvroRecordReader(FileReader<T> reader, FileSplit split)
 throws IOException {
 this.reader = reader;
 reader.sync(split.getStart());                    // sync to start
 this.start = reader.tell();
 this.end = split.getStart() + split.getLength();
}

代码示例来源：origin: apache/avro

protected AvroAsTextRecordReader(FileReader<T> reader, FileSplit split)
 throws IOException {
 this.reader = reader;
 reader.sync(split.getStart());                    // sync to start
 this.start = reader.tell();
 this.end = split.getStart() + split.getLength();
}

代码示例来源：origin: apache/hive

public AvroGenericRecordReader(JobConf job, FileSplit split, Reporter reporter) throws IOException {
 this.jobConf = job;
 Schema latest;
 try {
  latest = getSchema(job, split);
 } catch (AvroSerdeException e) {
  throw new IOException(e);
 }
 GenericDatumReader<GenericRecord> gdr = new GenericDatumReader<GenericRecord>();
 if(latest != null) {
  gdr.setExpected(latest);
 }
 if (split.getLength() == 0) {
  this.isEmptyInput = true;
  this.start = 0;
  this.reader = null;
 }
 else {
  this.isEmptyInput = false;
  this.reader = new DataFileReader<GenericRecord>(new FsInput(split.getPath(), job), gdr);
  this.reader.sync(split.getStart());
  this.start = reader.tell();
 }
 this.stop = split.getStart() + split.getLength();
 this.recordReaderID = new UID();
}

代码示例来源：origin: Netflix/iceberg

@Override
public void sync(long position) throws IOException {
 reader.sync(position);
}

代码示例来源：origin: Netflix/iceberg

AvroRangeIterator(FileReader<D> reader, long start, long end) {
 this.reader = reader;
 this.end = end;
 try {
  reader.sync(start);
 } catch (IOException e) {
  throw new RuntimeIOException(e, "Failed to find sync past position %d", start);
 }
}

代码示例来源：origin: stratosphere/stratosphere

@Override
public void open(FileInputSplit split) throws IOException {
  super.open(split);
  DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>();
  SeekableInput in = new FSDataInputStreamWrapper(stream, (int) split.getLength());
  LOG.info("Opening split " + split);
  dataFileReader = DataFileReader.openReader(in, datumReader);
  dataFileReader.sync(split.getStart());
}

代码示例来源：origin: org.apache.pig/pig

@Override
public void initialize(final InputSplit isplit, final TaskAttemptContext tc)
  throws IOException, InterruptedException {
 FileSplit fsplit = (FileSplit) isplit;
 start  = fsplit.getStart();
 end    = fsplit.getStart() + fsplit.getLength();
 DatumReader<GenericData.Array<Object>> datumReader
  = new GenericDatumReader<GenericData.Array<Object>>(schema);
 reader = DataFileReader.openReader(
   new FsInput(fsplit.getPath(), tc.getConfiguration()),
   datumReader);
 reader.sync(start);
}

代码示例来源：origin: com.datasalt.pangool/pangool-core

protected void init(FileReader<T> reader, FileSplit split)
  throws IOException {
  this.reader = reader;
  reader.sync(split.getStart());                    // sync to start
  this.start = reader.tell();
  this.end = split.getStart() + split.getLength();
 }

代码示例来源：origin: org.apache.pig/pig

@Override
public void initialize(final InputSplit isplit, final TaskAttemptContext tc)
  throws IOException, InterruptedException {
 FileSplit fsplit = (FileSplit) isplit;
 start  = fsplit.getStart();
 end    = fsplit.getStart() + fsplit.getLength();
 DatumReader<GenericData.Record> datumReader
  = new GenericDatumReader<GenericData.Record>(schema);
 reader = DataFileReader.openReader(
   new FsInput(fsplit.getPath(), tc.getConfiguration()),
   datumReader);
 reader.sync(start);
}

代码示例来源：origin: org.apache.avro/avro-mapred

protected AvroRecordReader(FileReader<T> reader, FileSplit split)
 throws IOException {
 this.reader = reader;
 reader.sync(split.getStart());                    // sync to start
 this.start = reader.tell();
 this.end = split.getStart() + split.getLength();
}

代码示例来源：origin: org.apache.avro/avro-mapred

protected AvroAsTextRecordReader(FileReader<T> reader, FileSplit split)
 throws IOException {
 this.reader = reader;
 reader.sync(split.getStart());                    // sync to start
 this.start = reader.tell();
 this.end = split.getStart() + split.getLength();
}

代码示例来源：origin: jwills/avro-json

protected AvroAsJSONRecordReader(FileReader<GenericRecord> reader, FileSplit split)
  throws IOException {
 this.reader = reader;
 reader.sync(split.getStart());
 this.start = reader.tell();
 this.end = split.getStart() + split.getLength();
}

代码示例来源：origin: com.datasalt.pangool/pangool-core

protected void init(FileReader<T> reader, FileSplit split)
  throws IOException {
  this.reader = reader;
  reader.sync(split.getStart());                    // sync to start
  this.start = reader.tell();
  this.end = split.getStart() + split.getLength();
 }

代码示例来源：origin: datasalt/pangool

protected void init(FileReader<T> reader, FileSplit split)
  throws IOException {
  this.reader = reader;
  reader.sync(split.getStart());                    // sync to start
  this.start = reader.tell();
  this.end = split.getStart() + split.getLength();
 }

代码示例来源：origin: datasalt/pangool

protected void init(FileReader<T> reader, FileSplit split)
  throws IOException {
  this.reader = reader;
  reader.sync(split.getStart());                    // sync to start
  this.start = reader.tell();
  this.end = split.getStart() + split.getLength();
 }

代码示例来源：origin: tomslabs/avro-utils

protected AvroTextRecordReader(FileReader<T> reader, FileSplit split) throws IOException {
  this.reader = reader;
  reader.sync(split.getStart()); // sync to start
  this.start = reader.tell();
  this.end = split.getStart() + split.getLength();
}

代码示例来源：origin: tomslabs/avro-utils

protected AvroTypedBytesRecordReader(FileReader<T> reader, FileSplit split) throws IOException {
  this.reader = reader;
  reader.sync(split.getStart()); // sync to start
  this.start = reader.tell();
  this.end = split.getStart() + split.getLength();
}

代码示例来源：origin: stratosphere/stratosphere

@Override
public void open(FileInputSplit split) throws IOException {
  super.open(split);
  DatumReader<E> datumReader;
  if (org.apache.avro.specific.SpecificRecordBase.class.isAssignableFrom(avroValueType)) {
    datumReader = new SpecificDatumReader<E>(avroValueType);
  } else {
    datumReader = new ReflectDatumReader<E>(avroValueType);
  }
  
  LOG.info("Opening split " + split);
  
  SeekableInput in = new FSDataInputStreamWrapper(stream, (int) split.getLength());
  
  dataFileReader = DataFileReader.openReader(in, datumReader);
  dataFileReader.sync(split.getStart());
}

代码示例来源：origin: org.apache.crunch/crunch-core

@Override
public void initialize(InputSplit genericSplit, TaskAttemptContext context) throws IOException, InterruptedException {
 FileSplit split = (FileSplit) genericSplit;
 Configuration conf = context.getConfiguration();
 SeekableInput in = new FsInput(split.getPath(), conf);
 DatumReader<T> datumReader = AvroMode
   .fromConfiguration(context.getConfiguration())
   .getReader(schema);
 this.reader = DataFileReader.openReader(in, datumReader);
 reader.sync(split.getStart()); // sync to start
 this.start = reader.tell();
 this.end = split.getStart() + split.getLength();
}

代码示例来源：origin: apache/sqoop

private boolean checkAvroFileForLine(FileSystem fs, Path p, List<Integer> record)
  throws IOException {
 SeekableInput in = new FsInput(p, new Configuration());
 DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>();
 FileReader<GenericRecord> reader = DataFileReader.openReader(in, datumReader);
 reader.sync(0);
 while (reader.hasNext()) {
  if (valueMatches(reader.next(), record)) {
   return true;
  }
 }
 return false;
}

内容来源于网络，如有侵权，请联系作者删除！

FileReader

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com