org.apache.gobblin.writer.WriterOutputFormat类的使用及代码示例

x33g5p2x  于2022-02-03 转载在 其他  
字(8.7k)|赞(0)|评价(0)|浏览(121)

本文整理了Java中org.apache.gobblin.writer.WriterOutputFormat类的一些代码示例，展示了WriterOutputFormat类的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。WriterOutputFormat类的具体详情如下：
包路径：org.apache.gobblin.writer.WriterOutputFormat
类名称：WriterOutputFormat

WriterOutputFormat介绍

[英]An enumeration of writer output formats.
[中]写入程序输出格式的枚举。

代码示例

代码示例来源：origin: apache/incubator-gobblin

/**
 * Get the output format of the writer of type {@link WriterOutputFormat}.
 *
 * @param branches number of forked branches
 * @param index branch index
 * @return output format of the writer
 */
public WriterOutputFormat getWriterOutputFormat(int branches, int index) {
 String writerOutputFormatValue = this.taskState.getProp(
   ForkOperatorUtils.getPropertyNameForBranch(ConfigurationKeys.WRITER_OUTPUT_FORMAT_KEY, branches, index),
   WriterOutputFormat.OTHER.name());
 log.debug("Found writer output format value = {}", writerOutputFormatValue);
 WriterOutputFormat wof = Enums.getIfPresent(WriterOutputFormat.class, writerOutputFormatValue.toUpperCase())
   .or(WriterOutputFormat.OTHER);
 log.debug("Returning writer output format = {}", wof);
 return wof;
}

代码示例来源：origin: apache/incubator-gobblin

this.setConfiguration(ConfigurationKeys.DATA_PUBLISHER_FINAL_DIR, workDir + "/event_data");
this.setConfiguration(ConfigurationKeys.DATA_PUBLISHER_METADATA_OUTPUT_DIR, workDir + "/metadata");
this.setConfiguration(ConfigurationKeys.WRITER_OUTPUT_FORMAT_KEY, WriterOutputFormat.PARQUET.toString());

代码示例来源：origin: apache/incubator-gobblin

/**
 * Get the file name to be used by the writer. If a {@link org.apache.gobblin.writer.partitioner.WriterPartioner} is used,
 * the partition will be added as part of the file name.
 */
public String getFileName(State properties) {
 String extension =
   this.format.equals(WriterOutputFormat.OTHER) ? getExtension(properties) : this.format.getExtension();
 String fileName = WriterUtils.getWriterFileName(properties, this.branches, this.branch, this.writerId, extension);
 if (this.partition.isPresent()) {
  fileName = getPartitionedFileName(properties, fileName);
 }
 List<StreamCodec> encoders = getEncoders();
 if (!encoders.isEmpty()) {
  StringBuilder filenameBuilder = new StringBuilder(fileName);
  for (StreamCodec codec : encoders) {
   filenameBuilder.append('.');
   filenameBuilder.append(codec.getTag());
  }
  fileName = filenameBuilder.toString();
 }
 return fileName;
}

代码示例来源：origin: apache/incubator-gobblin

/**
 * Write output to avro files at the given input location.
 */
@CliObjectOption(description = "Write output to Avro files. Specify the output directory as argument.")
public EmbeddedWikipediaExample avroOutput(String outputPath) {
 this.setConfiguration(ConfigurationKeys.WRITER_BUILDER_CLASS, AvroDataWriterBuilder.class.getName());
 this.setConfiguration(ConfigurationKeys.WRITER_DESTINATION_TYPE_KEY, Destination.DestinationType.HDFS.name());
 this.setConfiguration(ConfigurationKeys.WRITER_OUTPUT_FORMAT_KEY, WriterOutputFormat.AVRO.name());
 this.setConfiguration(ConfigurationKeys.WRITER_PARTITIONER_CLASS, WikipediaPartitioner.class.getName());
 this.setConfiguration(ConfigurationKeys.JOB_DATA_PUBLISHER_TYPE, BaseDataPublisher.class.getName());
 this.setConfiguration(ConfigurationKeys.CONVERTER_CLASSES_KEY, WikipediaConverter.class.getName());
 this.setConfiguration(ConfigurationKeys.DATA_PUBLISHER_FINAL_DIR, outputPath);
 return this;
}

代码示例来源：origin: org.apache.gobblin/gobblin-core

/**
 * Get the file name to be used by the writer. If a {@link org.apache.gobblin.writer.partitioner.WriterPartioner} is used,
 * the partition will be added as part of the file name.
 */
public String getFileName(State properties) {
 String extension =
   this.format.equals(WriterOutputFormat.OTHER) ? getExtension(properties) : this.format.getExtension();
 String fileName = WriterUtils.getWriterFileName(properties, this.branches, this.branch, this.writerId, extension);
 if (this.partition.isPresent()) {
  fileName = getPartitionedFileName(properties, fileName);
 }
 List<StreamCodec> encoders = getEncoders();
 if (!encoders.isEmpty()) {
  StringBuilder filenameBuilder = new StringBuilder(fileName);
  for (StreamCodec codec : encoders) {
   filenameBuilder.append('.');
   filenameBuilder.append(codec.getTag());
  }
  fileName = filenameBuilder.toString();
 }
 return fileName;
}

代码示例来源：origin: apache/incubator-gobblin

private void prepareWorkUnit(WorkUnit workUnit) {
  workUnit.setProp(ConfigurationKeys.TASK_ID_KEY, TestHelper.TEST_TASK_ID);
  workUnit.setProp(ConfigurationKeys.TASK_KEY_KEY, Long.toString(Id.parse(TestHelper.TEST_TASK_ID).getSequence()));
  workUnit.setProp(ConfigurationKeys.SOURCE_CLASS_KEY, SimpleJsonSource.class.getName());
  workUnit.setProp(ConfigurationKeys.CONVERTER_CLASSES_KEY, SimpleJsonConverter.class.getName());
  workUnit.setProp(ConfigurationKeys.WRITER_OUTPUT_FORMAT_KEY, WriterOutputFormat.AVRO.toString());
  workUnit.setProp(ConfigurationKeys.WRITER_DESTINATION_TYPE_KEY, Destination.DestinationType.HDFS.toString());
  workUnit.setProp(ConfigurationKeys.WRITER_STAGING_DIR, this.appWorkDir.toString() + Path.SEPARATOR + "staging");
  workUnit.setProp(ConfigurationKeys.WRITER_OUTPUT_DIR, this.taskOutputDir.toString());
  workUnit.setProp(ConfigurationKeys.WRITER_FILE_NAME, TestHelper.WRITER_FILE_NAME);
  workUnit.setProp(ConfigurationKeys.WRITER_FILE_PATH, TestHelper.REL_WRITER_FILE_PATH);
  workUnit.setProp(ConfigurationKeys.WRITER_BUILDER_CLASS, AvroDataWriterBuilder.class.getName());
  workUnit.setProp(ConfigurationKeys.SOURCE_SCHEMA, TestHelper.SOURCE_SCHEMA);
 }
}

代码示例来源：origin: apache/incubator-gobblin

jobProps.setProperty(ConfigurationKeys.TASK_LEVEL_POLICY_LIST_TYPE + ".0", "OPTIONAL,OPTIONAL");
jobProps.setProperty(ConfigurationKeys.TASK_LEVEL_POLICY_LIST_TYPE + ".1", "OPTIONAL,OPTIONAL");
jobProps.setProperty(ConfigurationKeys.WRITER_OUTPUT_FORMAT_KEY + ".0", WriterOutputFormat.AVRO.name());
jobProps.setProperty(ConfigurationKeys.WRITER_OUTPUT_FORMAT_KEY + ".1", WriterOutputFormat.AVRO.name());
jobProps.setProperty(ConfigurationKeys.WRITER_DESTINATION_TYPE_KEY + ".0", Destination.DestinationType.HDFS.name());
jobProps.setProperty(ConfigurationKeys.WRITER_DESTINATION_TYPE_KEY + ".1", Destination.DestinationType.HDFS.name());

代码示例来源：origin: org.apache.gobblin/gobblin-example

this.setConfiguration(ConfigurationKeys.DATA_PUBLISHER_FINAL_DIR, workDir + "/event_data");
this.setConfiguration(ConfigurationKeys.DATA_PUBLISHER_METADATA_OUTPUT_DIR, workDir + "/metadata");
this.setConfiguration(ConfigurationKeys.WRITER_OUTPUT_FORMAT_KEY, WriterOutputFormat.PARQUET.toString());

代码示例来源：origin: apache/incubator-gobblin

jobProps.setProperty(ConfigurationKeys.TASK_LEVEL_POLICY_LIST_TYPE + ".0", "OPTIONAL,OPTIONAL");
jobProps.setProperty(ConfigurationKeys.TASK_LEVEL_POLICY_LIST_TYPE + ".1", "OPTIONAL,OPTIONAL");
jobProps.setProperty(ConfigurationKeys.WRITER_OUTPUT_FORMAT_KEY + ".0", WriterOutputFormat.AVRO.name());
jobProps.setProperty(ConfigurationKeys.WRITER_OUTPUT_FORMAT_KEY + ".1", WriterOutputFormat.AVRO.name());
jobProps.setProperty(ConfigurationKeys.WRITER_DESTINATION_TYPE_KEY + ".0", Destination.DestinationType.HDFS.name());
jobProps.setProperty(ConfigurationKeys.WRITER_DESTINATION_TYPE_KEY + ".1", Destination.DestinationType.HDFS.name());

代码示例来源：origin: org.apache.gobblin/gobblin-runtime

/**
 * Get the output format of the writer of type {@link WriterOutputFormat}.
 *
 * @param branches number of forked branches
 * @param index branch index
 * @return output format of the writer
 */
public WriterOutputFormat getWriterOutputFormat(int branches, int index) {
 String writerOutputFormatValue = this.taskState.getProp(
   ForkOperatorUtils.getPropertyNameForBranch(ConfigurationKeys.WRITER_OUTPUT_FORMAT_KEY, branches, index),
   WriterOutputFormat.OTHER.name());
 log.debug("Found writer output format value = {}", writerOutputFormatValue);
 WriterOutputFormat wof = Enums.getIfPresent(WriterOutputFormat.class, writerOutputFormatValue.toUpperCase())
   .or(WriterOutputFormat.OTHER);
 log.debug("Returning writer output format = {}", wof);
 return wof;
}

代码示例来源：origin: org.apache.gobblin/gobblin-example

/**
 * Write output to avro files at the given input location.
 */
@CliObjectOption(description = "Write output to Avro files. Specify the output directory as argument.")
public EmbeddedWikipediaExample avroOutput(String outputPath) {
 this.setConfiguration(ConfigurationKeys.WRITER_BUILDER_CLASS, AvroDataWriterBuilder.class.getName());
 this.setConfiguration(ConfigurationKeys.WRITER_DESTINATION_TYPE_KEY, Destination.DestinationType.HDFS.name());
 this.setConfiguration(ConfigurationKeys.WRITER_OUTPUT_FORMAT_KEY, WriterOutputFormat.AVRO.name());
 this.setConfiguration(ConfigurationKeys.WRITER_PARTITIONER_CLASS, WikipediaPartitioner.class.getName());
 this.setConfiguration(ConfigurationKeys.JOB_DATA_PUBLISHER_TYPE, BaseDataPublisher.class.getName());
 this.setConfiguration(ConfigurationKeys.CONVERTER_CLASSES_KEY, WikipediaConverter.class.getName());
 this.setConfiguration(ConfigurationKeys.DATA_PUBLISHER_FINAL_DIR, outputPath);
 return this;
}

内容来源于网络，如有侵权，请联系作者删除！

WriterOutputFormat

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com