org.apache.avro.file.FileReader.pastSync()方法的使用及代码示例

x33g5p2x  于2022-01-19 转载在 其他  
字(6.2k)|赞(0)|评价(0)|浏览(132)

本文整理了Java中org.apache.avro.file.FileReader.pastSync()方法的一些代码示例,展示了FileReader.pastSync()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。FileReader.pastSync()方法的具体详情如下:
包路径:org.apache.avro.file.FileReader
类名称:FileReader
方法名:pastSync

FileReader.pastSync介绍

[英]Return true if past the next synchronization point after a position.
[中]如果超过位置后的下一个同步点,则返回true。

代码示例

代码示例来源:origin: apache/avro

public boolean next(Text key, Text ignore) throws IOException {
 if (!reader.hasNext() || reader.pastSync(end))
  return false;
 datum = reader.next(datum);
 if (datum instanceof ByteBuffer) {
  ByteBuffer b = (ByteBuffer) datum;
  if (b.hasArray()) {
   int offset = b.arrayOffset();
   int start = b.position();
   int length = b.remaining();
   key.set(b.array(), offset + start, offset + start + length);
  } else {
   byte[] bytes = new byte[b.remaining()];
   b.duplicate().get(bytes);
   key.set(bytes);
  }
 } else {
  key.set(GenericData.get().toString(datum));
 }
 return true;
}

代码示例来源:origin: apache/avro

public boolean next(AvroWrapper<T> wrapper, NullWritable ignore)
 throws IOException {
 if (!reader.hasNext() || reader.pastSync(end))
  return false;
 wrapper.datum(reader.next(wrapper.datum()));
 return true;
}

代码示例来源:origin: apache/hive

@Override
public boolean next(NullWritable nullWritable, AvroGenericRecordWritable record) throws IOException {
 if(isEmptyInput || !reader.hasNext() || reader.pastSync(stop)) {
  return false;
 }
 GenericData.Record r = (GenericData.Record)reader.next();
 record.setRecord(r);
 record.setRecordReaderID(recordReaderID);
 record.setFileSchema(reader.getSchema());
 return true;
}

代码示例来源:origin: Netflix/iceberg

@Override
public boolean pastSync(long position) throws IOException {
 return reader.pastSync(position);
}

代码示例来源:origin: org.apache.pig/pig

@Override
public boolean nextKeyValue() throws IOException, InterruptedException {
 if (reader.pastSync(end)) {
  return false;
 }
 try {
  currentArray = reader.next();
 } catch (NoSuchElementException e) {
  return false;
 }
 return true;
}

代码示例来源:origin: jwills/avro-json

@Override
 public boolean next(Text key, Text value) throws IOException {
  if (!reader.hasNext() || reader.pastSync(end)) {
   return false;
  }
  datum = reader.next(datum);
  key.set(new String(datum.toString().getBytes("UTF-8"), "UTF-8"));
  return true;
 }
}

代码示例来源:origin: tomslabs/avro-utils

public boolean next(Text key, Text value) throws IOException {
  if (!reader.hasNext() || reader.pastSync(end)) {
    return false;
  }
  StringBuilder buf = new StringBuilder();
  JSONUtils.writeJSON(reader.next(), buf);
  key.set(buf.toString());
  return true;
}

代码示例来源:origin: Netflix/iceberg

@Override
public boolean hasNext() {
 try {
  return (reader.hasNext() && !reader.pastSync(end));
 } catch (IOException e) {
  throw new RuntimeIOException(e, "Failed to check range end: %d", end);
 }
}

代码示例来源:origin: com.google.cloud.bigdataoss/bigquery-connector

@Override
public boolean nextKeyValue() throws IOException, InterruptedException {
 Preconditions.checkState(currentRecord != null);
 // Stop reading as soon as we hit a sync point out of our split:
 if (dataFileReader.hasNext() && !dataFileReader.pastSync(splitStart + splitLength)) {
  currentKey.set(currentKey.get() + 1);
  dataFileReader.next(currentRecord);
  return true;
 } else {
  return false;
 }
}

代码示例来源:origin: org.apache.pig/pig

@Override
public boolean nextKeyValue() throws IOException, InterruptedException {
 if (reader.pastSync(end)) {
  return false;
 }
 try {
  currentRecord = reader.next(new GenericData.Record(schema));
 } catch (NoSuchElementException e) {
  return false;
 } catch (IOException ioe) {
  reader.sync(reader.tell()+1);
  throw ioe;
 }
 return true;
}

代码示例来源:origin: org.apache.hadoop.hive/hive-exec

@Override
public boolean next(NullWritable nullWritable, AvroGenericRecordWritable record) throws IOException {
 if(!reader.hasNext() || reader.pastSync(stop)) {
  return false;
 }
 GenericData.Record r = (GenericData.Record)reader.next();
 record.setRecord(r);
 return true;
}

代码示例来源:origin: jghoman/haivvreo

@Override
public boolean next(NullWritable nullWritable, AvroGenericRecordWritable record) throws IOException {
 if(!reader.hasNext() || reader.pastSync(stop)) return false;
 GenericData.Record r = (GenericData.Record)reader.next();
 record.setRecord(r);
 return true;
}

代码示例来源:origin: tomslabs/avro-utils

public boolean next(TypedBytesWritable key, TypedBytesWritable value) throws IOException {
  if (!reader.hasNext() || reader.pastSync(end)) {
    return false;
  }
  key.setValue("");
  // until https://github.com/apache/avro/pull/2/ is fixed, we can not rely
  // on toString() to provide a correct JSON string with unicode.
  //value.setValue(reader.next().toString());
  StringBuilder buf = new StringBuilder();
  JSONUtils.writeJSON(reader.next(), buf);
  value.setValue(buf.toString());
  return true;
}

代码示例来源:origin: com.datasalt.pangool/pangool-core

@Override
public boolean nextKeyValue() throws IOException, InterruptedException {
  if (!reader.hasNext() || reader.pastSync(end))
 return false;
wrapper.datum(reader.next(wrapper.datum()));
return true;
}

代码示例来源:origin: org.apache.avro/avro-mapred

public boolean next(AvroWrapper<T> wrapper, NullWritable ignore)
 throws IOException {
 if (!reader.hasNext() || reader.pastSync(end))
  return false;
 wrapper.datum(reader.next(wrapper.datum()));
 return true;
}

代码示例来源:origin: com.datasalt.pangool/pangool-core

@Override
public boolean nextKeyValue() throws IOException, InterruptedException {
  if (!reader.hasNext() || reader.pastSync(end)){
 return false;
  }
  
wrapper.datum(reader.next(wrapper.datum()));
return true;
}

代码示例来源:origin: datasalt/pangool

@Override
public boolean nextKeyValue() throws IOException, InterruptedException {
  if (!reader.hasNext() || reader.pastSync(end))
 return false;
wrapper.datum(reader.next(wrapper.datum()));
return true;
}

代码示例来源:origin: datasalt/pangool

@Override
public boolean nextKeyValue() throws IOException, InterruptedException {
  if (!reader.hasNext() || reader.pastSync(end)){
 return false;
  }
  
wrapper.datum(reader.next(wrapper.datum()));
return true;
}

代码示例来源:origin: org.apache.crunch/crunch-core

@Override
public boolean nextKeyValue() throws IOException, InterruptedException {
 if (!reader.hasNext() || reader.pastSync(end)) {
  key = null;
  value = null;
  return false;
 }
 if (key == null) {
  key = new AvroWrapper<T>();
 }
 if (value == null) {
  value = NullWritable.get();
 }
 key.datum(reader.next(key.datum()));
 return true;
}

代码示例来源:origin: com.facebook.presto.hive/hive-apache

@Override
public boolean next(NullWritable nullWritable, AvroGenericRecordWritable record) throws IOException {
 if(!reader.hasNext() || reader.pastSync(stop)) {
  return false;
 }
 GenericData.Record r = (GenericData.Record)reader.next();
 record.setRecord(r);
 record.setRecordReaderID(recordReaderID);
 record.setFileSchema(reader.getSchema());
 return true;
}

相关文章