parquet.io.api.Binary类的使用及代码示例

x33g5p2x  于2022-01-17 转载在 其他  
字(6.9k)|赞(0)|评价(0)|浏览(258)

本文整理了Java中parquet.io.api.Binary类的一些代码示例，展示了Binary类的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Binary类的具体详情如下：
包路径：parquet.io.api.Binary
类名称：Binary

Binary介绍

暂无

代码示例

代码示例来源：origin: prestodb/presto

  1. case STRING:
  2.   String v = ((StringObjectInspector) inspector).getPrimitiveJavaObject(value);
  3.   recordConsumer.addBinary(Binary.fromString(v));
  4.   break;
  5. case CHAR:
  6.   String vChar = ((HiveCharObjectInspector) inspector).getPrimitiveJavaObject(value).getStrippedValue();
  7.   recordConsumer.addBinary(Binary.fromString(vChar));
  8.   break;
  9. case VARCHAR:
  10.   String vVarchar = ((HiveVarcharObjectInspector) inspector).getPrimitiveJavaObject(value).getValue();
  11.   recordConsumer.addBinary(Binary.fromString(vVarchar));
  12.   break;
  13. case BINARY:
  14.   byte[] vBinary = ((BinaryObjectInspector) inspector).getPrimitiveJavaObject(value);
  15.   recordConsumer.addBinary(Binary.fromByteArray(vBinary));
  16.   break;
  17. case TIMESTAMP:

代码示例来源：origin: prestodb/presto

  1. /**
  2.  * Returns GMT timestamp from binary encoded parquet timestamp (12 bytes - julian date + time of day nanos).
  3.  *
  4.  * @param timestampBinary INT96 parquet timestamp
  5.  * @return timestamp in millis, GMT timezone
  6.  */
  7. public static long getTimestampMillis(Binary timestampBinary)
  8. {
  9.   if (timestampBinary.length() != 12) {
  10.     throw new PrestoException(NOT_SUPPORTED, "Parquet timestamp must be 12 bytes, actual " + timestampBinary.length());
  11.   }
  12.   byte[] bytes = timestampBinary.getBytes();
  13.   // little endian encoding - need to invert byte order
  14.   long timeOfDayNanos = Longs.fromBytes(bytes[7], bytes[6], bytes[5], bytes[4], bytes[3], bytes[2], bytes[1], bytes[0]);
  15.   int julianDay = Ints.fromBytes(bytes[11], bytes[10], bytes[9], bytes[8]);
  16.   return julianDayToMillis(julianDay) + (timeOfDayNanos / NANOS_PER_MILLISECOND);
  17. }

代码示例来源：origin: prestodb/presto

  1. public BinaryDictionary(DictionaryPage dictionaryPage, Integer length)
  2.     throws IOException
  3. {
  4.   super(dictionaryPage.getEncoding());
  5.   byte[] dictionaryBytes = dictionaryPage.getSlice().getBytes();
  6.   content = new Binary[dictionaryPage.getDictionarySize()];
  7.   int offset = 0;
  8.   if (length == null) {
  9.     for (int i = 0; i < content.length; i++) {
  10.       int len = readIntLittleEndian(dictionaryBytes, offset);
  11.       offset += 4;
  12.       content[i] = Binary.fromByteArray(dictionaryBytes, offset, len);
  13.       offset += len;
  14.     }
  15.   }
  16.   else {
  17.     checkArgument(length > 0, "Invalid byte array length: %s", length);
  18.     for (int i = 0; i < content.length; i++) {
  19.       content[i] = Binary.fromByteArray(dictionaryBytes, offset, length);
  20.       offset += length;
  21.     }
  22.   }
  23. }

代码示例来源：origin: apache/incubator-gobblin

  1. @Override
  2. BinaryValue convertField(JsonElement value) {
  3.  return new BinaryValue(Binary.fromString(value.getAsString()));
  4. }

代码示例来源：origin: prestodb/presto

  1. @Override
  2. protected void readValue(BlockBuilder blockBuilder, Type type)
  3. {
  4.   if (definitionLevel == columnDescriptor.getMaxDefinitionLevel()) {
  5.     Binary value = valuesReader.readBytes();
  6.     type.writeSlice(blockBuilder, Decimals.encodeUnscaledValue(new BigInteger(value.getBytes())));
  7.   }
  8.   else if (isValueNull()) {
  9.     blockBuilder.appendNull();
  10.   }
  11. }

代码示例来源：origin: com.facebook.presto.hive/hive-apache

  1. protected static Binary copy(Binary binary) {
  2.   return Binary.fromByteArray(
  3.     Arrays.copyOf(binary.getBytes(), binary.length()));
  4.  }
  5. }

代码示例来源：origin: com.facebook.presto.hive/hive-apache

  1. @Override
  2. public String getString() {
  3.  return binary.toStringUsingUTF8();
  4. }

代码示例来源：origin: com.facebook.presto.hive/hive-apache

  1. @Override
  2.  public void writeBytes(Binary v) {
  3.   int i = 0;
  4.   byte[] vb = v.getBytes();
  5.   int length = previous.length < vb.length ? previous.length : vb.length;
  6.   for(i = 0; (i < length) && (previous[i] == vb[i]); i++);
  7.   prefixLengthWriter.writeInteger(i);
  8.   suffixWriter.writeBytes(Binary.fromByteArray(vb, i, vb.length - i));
  9.   previous = vb;
  10.  }
  11. }

代码示例来源：origin: com.twitter/parquet-tools

  1. public static String binaryToString(Binary value) {
  2.   byte[] data = value.getBytes();
  3.   if (data == null) return null;
  4.   try {
  5.     CharBuffer buffer = UTF8_DECODER.decode(value.toByteBuffer());
  6.     return buffer.toString();
  7.   } catch (Throwable th) {
  8.   }
  9.   return "<bytes...>";
  10. }

代码示例来源：origin: com.facebook.presto.hive/hive-apache

  1. public static NanoTime fromBinary(Binary bytes) {
  2.  Preconditions.checkArgument(bytes.length() == 12, "Must be 12 bytes");
  3.  ByteBuffer buf = bytes.toByteBuffer();
  4.  buf.order(ByteOrder.LITTLE_ENDIAN);
  5.  long timeOfDayNanos = buf.getLong();
  6.  int julianDay = buf.getInt();
  7.  return new NanoTime(julianDay, timeOfDayNanos);
  8. }

代码示例来源：origin: com.facebook.presto.hive/hive-apache

  1. public void writeBytes(Binary v) {
  2.  //for rawdata, length(4 bytes int) is stored, followed by the binary content itself
  3.  rawDataByteSize += v.length() + 4;
  4.  currentWriter.writeBytes(v);
  5.  checkFallback();
  6. }

代码示例来源：origin: com.facebook.presto.hive/hive-apache

  1. @Override
  2. public final void writeBytes(Binary v) {
  3.  if (v.length() != length) {
  4.   throw new IllegalArgumentException("Fixed Binary size " + v.length() +
  5.     " does not match field type length " + length);
  6.  }
  7.  try {
  8.   v.writeTo(out);
  9.  } catch (IOException e) {
  10.   throw new ParquetEncodingException("could not write fixed bytes", e);
  11.  }
  12. }

代码示例来源：origin: asakusafw/asakusafw

  1. @Override
  2. public void addBinary(Binary value) {
  3.   ByteBuffer bytes = value.toByteBuffer().order(ByteOrder.LITTLE_ENDIAN);
  4.   long time = bytes.getLong();
  5.   int day = bytes.getInt();
  6.   addNanoTime(day, time);
  7. }

代码示例来源：origin: prestodb/presto

  1. @Test
  2. public void testInvalidBinaryLength()
  3. {
  4.   try {
  5.     byte[] invalidLengthBinaryTimestamp = new byte[8];
  6.     getTimestampMillis(Binary.fromByteArray(invalidLengthBinaryTimestamp));
  7.   }
  8.   catch (PrestoException e) {
  9.     assertEquals(e.getErrorCode(), NOT_SUPPORTED.toErrorCode());
  10.     assertEquals(e.getMessage(), "Parquet timestamp must be 12 bytes, actual 8");
  11.   }
  12. }

代码示例来源：origin: prestodb/presto

  1. private static BinaryStatistics stringColumnStats(String minimum, String maximum)
  2. {
  3.   BinaryStatistics statistics = new BinaryStatistics();
  4.   statistics.setMinMax(Binary.fromString(minimum), Binary.fromString(maximum));
  5.   return statistics;
  6. }

代码示例来源：origin: prestodb/presto

  1. List<Domain> domains = new ArrayList<>();
  2. for (int i = 0; i < dictionarySize; i++) {
  3.   domains.add(Domain.singleValue(type, Slices.wrappedBuffer(dictionary.decodeToBinary(i).getBytes())));

代码示例来源：origin: com.facebook.presto.hive/hive-apache

  1. @Override
  2.  public Binary readBytes() {
  3.   int prefixLength = prefixLengthReader.readInteger();
  4.   // This does not copy bytes
  5.   Binary suffix = suffixReader.readBytes();
  6.   int length = prefixLength + suffix.length();
  7.   
  8.   // We have to do this to materialize the output
  9.   if(prefixLength != 0) {
  10.    byte[] out = new byte[length];
  11.    System.arraycopy(previous.getBytes(), 0, out, 0, prefixLength);
  12.    System.arraycopy(suffix.getBytes(), 0, out, prefixLength, suffix.length());
  13.    previous =  Binary.fromByteArray(out);
  14.   } else {
  15.    previous = suffix;
  16.   }
  17.   return previous;
  18.  }
  19. }

代码示例来源：origin: com.twitter/parquet-pig

  1. @Override
  2. final public void addBinary(Binary value) {
  3.  currentKey = value.toStringUsingUTF8();
  4. }

代码示例来源：origin: com.facebook.presto.hive/hive-apache

  1. public static NanoTime fromBinary(Binary bytes) {
  2.  Preconditions.checkArgument(bytes.length() == 12, "Must be 12 bytes");
  3.  ByteBuffer buf = bytes.toByteBuffer();
  4.  buf.order(ByteOrder.LITTLE_ENDIAN);
  5.  long timeOfDayNanos = buf.getLong();
  6.  int julianDay = buf.getInt();
  7.  return new NanoTime(julianDay, timeOfDayNanos);
  8. }

代码示例来源：origin: com.facebook.presto.hive/hive-apache

  1. @Override
  2. public void writeBytes(Binary v) {
  3.  int id = binaryDictionaryContent.getInt(v);
  4.  if (id == -1) {
  5.   id = binaryDictionaryContent.size();
  6.   binaryDictionaryContent.put(copy(v), id);
  7.   // length as int (4 bytes) + actual bytes
  8.   dictionaryByteSize += 4 + v.length();
  9.  }
  10.  encodedValues.add(id);
  11. }

内容来源于网络，如有侵权，请联系作者删除！

Binary

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com