org.apache.gobblin.source.workunit.Extract.<init>()方法的使用及代码示例

x33g5p2x  于2022-01-19 转载在 其他  
字(11.9k)|赞(0)|评价(0)|浏览(168)

本文整理了Java中org.apache.gobblin.source.workunit.Extract.<init>()方法的一些代码示例，展示了Extract.<init>()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Extract.<init>()方法的具体详情如下：
包路径：org.apache.gobblin.source.workunit.Extract
类名称：Extract
方法名：<init>

Extract.<init>介绍

[英]Constructor.
[中]构造器。

代码示例

代码示例来源：origin: apache/incubator-gobblin

  1. /**
  2.  * Get the {@link org.apache.gobblin.source.workunit.Extract} associated with the {@link WorkUnit}.
  3.  *
  4.  * @return {@link org.apache.gobblin.source.workunit.Extract} associated with the {@link WorkUnit}
  5.  */
  6. public Extract getExtract() {
  7.  return new Extract(this.workUnit.getExtract());
  8. }

代码示例来源：origin: apache/incubator-gobblin

  1. @Override
  2. public List<WorkUnit> getWorkunits(SourceState state) {
  3.  int numWorkUnits = state.getPropAsInt(NUM_WORK_UNITS_KEY, DEFAULT_NUM_WORK_UNITS);
  4.  Extract extract = new Extract(TableType.APPEND_ONLY,
  5.    StressTestingSource.class.getPackage().getName(),
  6.    StressTestingSource.class.getSimpleName());
  7.  List<WorkUnit> wus = new ArrayList<>(numWorkUnits);
  8.  for (int i = 1; i <= numWorkUnits; ++i) {
  9.   WorkUnit wu = new WorkUnit(extract);
  10.   wus.add(wu);
  11.  }
  12.  return wus;
  13. }

代码示例来源：origin: apache/incubator-gobblin

  1. /**
  2.  * Constructor.
  3.  *
  4.  * @param state a {@link SourceState} the properties of which will be copied into this {@link WorkUnit} instance
  5.  * @param extract an {@link Extract}
  6.  *
  7.  * @deprecated Properties in {@link SourceState} should not be added to a {@link WorkUnit}. Having each
  8.  * {@link WorkUnit} contain a copy of {@link SourceState} is a waste of memory. Use {@link #create(Extract)}.
  9.  */
  10. @Deprecated
  11. public WorkUnit(SourceState state, Extract extract) {
  12.  // Values should only be null for deserialization
  13.  if (state != null) {
  14.   super.addAll(state);
  15.  }
  16.  if (extract != null) {
  17.   this.extract = extract;
  18.  } else {
  19.   this.extract = new Extract(null, null, null, null);
  20.  }
  21. }

代码示例来源：origin: apache/incubator-gobblin

  1. private Extract getExtractForFile(PartitionAwareFileRetriever.FileInfo file,
  2.   String topicName,
  3.   String namespace,
  4.   Map<Long, Extract> extractMap) {
  5.  Extract extract = extractMap.get(file.getWatermarkMsSinceEpoch());
  6.  if (extract == null) {
  7.   // Create an extract object for the dayPath
  8.   extract = new Extract(this.tableType, namespace, topicName);
  9.   LOG.info("Created extract: " + extract.getExtractId() + " for path " + topicName);
  10.   extractMap.put(file.getWatermarkMsSinceEpoch(), extract);
  11.  }
  12.  return extract;
  13. }

代码示例来源：origin: apache/incubator-gobblin

  1. @Override
  2. public List<WorkUnit> getWorkunits(SourceState state) {
  3.  Config rootCfg = ConfigUtils.propertiesToConfig(state.getProperties());
  4.  Config cfg = rootCfg.hasPath(CONFIG_NAMESPACE) ? rootCfg.getConfig(CONFIG_NAMESPACE) :
  5.     ConfigFactory.empty();
  6.  int numHellos = cfg.hasPath(NUM_HELLOS_KEY) ? cfg.getInt(NUM_HELLOS_KEY) : DEFAULT_NUM_HELLOS;
  7.  Extract extract = new Extract(TableType.APPEND_ONLY,
  8.     HelloWorldSource.class.getPackage().getName(),
  9.     HelloWorldSource.class.getSimpleName());
  10.  List<WorkUnit> wus = new ArrayList<>(numHellos);
  11.  for (int i = 1; i <= numHellos; ++i) {
  12.   WorkUnit wu = new WorkUnit(extract);
  13.   wu.setProp(HELLO_ID_FULL_KEY, i);
  14.   wus.add(wu);
  15.  }
  16.  return wus;
  17. }

代码示例来源：origin: apache/incubator-gobblin

  1. @Override
  2. public List<WorkUnit> getWorkunits(SourceState state) {
  3.  List<WorkUnit> workUnits = Lists.newArrayList();
  4.  if (!state.contains(ConfigurationKeys.SOURCE_FILEBASED_FILES_TO_PULL)) {
  5.   return workUnits;
  6.  }
  7.  // Create a single snapshot-type extract for all files
  8.  Extract extract = new Extract(Extract.TableType.SNAPSHOT_ONLY,
  9.    state.getProp(ConfigurationKeys.EXTRACT_NAMESPACE_NAME_KEY, "ExampleNamespace"), "ExampleTable");
  10.  String filesToPull = state.getProp(ConfigurationKeys.SOURCE_FILEBASED_FILES_TO_PULL);
  11.  for (String file : Splitter.on(',').omitEmptyStrings().split(filesToPull)) {
  12.   // Create one work unit for each file to pull
  13.   WorkUnit workUnit = WorkUnit.create(extract);
  14.   workUnit.setProp(SOURCE_FILE_KEY, file);
  15.   workUnits.add(workUnit);
  16.  }
  17.  return workUnits;
  18. }

代码示例来源：origin: apache/incubator-gobblin

  1. /**
  2.   * Returns a unique {@link Extract} instance.
  3.   * Any two calls of this method from the same {@link ExtractFactory} instance guarantees to
  4.   * return {@link Extract}s with different IDs.
  5.   *
  6.   * @param type {@link TableType}
  7.   * @param namespace dot separated namespace path
  8.   * @param table table name
  9.   * @return a unique {@link Extract} instance
  10.   */
  11.  public synchronized Extract getUniqueExtract(TableType type, String namespace, String table) {
  12.   Extract newExtract = new Extract(type, namespace, table);
  13.   while (this.createdInstances.contains(newExtract)) {
  14.    if (Strings.isNullOrEmpty(newExtract.getExtractId())) {
  15.     newExtract.setExtractId(this.dtf.print(new DateTime()));
  16.    } else {
  17.     DateTime extractDateTime = this.dtf.parseDateTime(newExtract.getExtractId());
  18.     newExtract.setExtractId(this.dtf.print(extractDateTime.plusSeconds(1)));
  19.    }
  20.   }
  21.   this.createdInstances.add(newExtract);
  22.   return newExtract;
  23.  }
  24. }

代码示例来源：origin: apache/incubator-gobblin

  1. public WorkUnit getWorkUnitWithWeight(long weight) {
  2.  WorkUnit workUnit = new WorkUnit(new Extract(Extract.TableType.APPEND_ONLY, "", ""));
  3.  workUnit.setProp(WEIGHT, Long.toString(weight));
  4.  return workUnit;
  5. }

代码示例来源：origin: apache/incubator-gobblin

  1. /**
  2.  * Create a new properly populated {@link Extract} instance.
  3.  *
  4.  * <p>
  5.  *   This method should always return a new unique {@link Extract} instance.
  6.  * </p>
  7.  *
  8.  * @param type {@link org.apache.gobblin.source.workunit.Extract.TableType}
  9.  * @param namespace namespace of the table this extract belongs to
  10.  * @param table name of the table this extract belongs to
  11.  * @return a new unique {@link Extract} instance
  12.  *
  13.  * @Deprecated Use {@link org.apache.gobblin.source.extractor.extract.AbstractSource#createExtract(
  14.  *org.apache.gobblin.source.workunit.Extract.TableType, String, String)}
  15.  */
  16. @Deprecated
  17. public synchronized Extract createExtract(Extract.TableType type, String namespace, String table) {
  18.  Extract extract = new Extract(this, type, namespace, table);
  19.  while (EXTRACT_SET.contains(extract)) {
  20.   if (Strings.isNullOrEmpty(extract.getExtractId())) {
  21.    extract.setExtractId(DTF.print(new DateTime()));
  22.   } else {
  23.    DateTime extractDateTime = DTF.parseDateTime(extract.getExtractId());
  24.    extract.setExtractId(DTF.print(extractDateTime.plusSeconds(1)));
  25.   }
  26.  }
  27.  EXTRACT_SET.add(extract);
  28.  return extract;
  29. }

代码示例来源：origin: apache/incubator-gobblin

  1. @BeforeClass
  2. public static void setUp() {
  3.  WorkUnit workUnit = new WorkUnit(new SourceState(),
  4.    new Extract(new SourceState(), Extract.TableType.SNAPSHOT_ONLY, "namespace", "dummy_table"));
  5.  state = new WorkUnitState(workUnit);
  6.  Type listType = new TypeToken<JsonObject>() {
  7.  }.getType();
  8.  Gson gson = new Gson();
  9.  testData = gson.fromJson(new InputStreamReader(
  10.      JsonElementConversionFactoryTest.class.getResourceAsStream("/converter/JsonElementConversionFactoryTest.json")),
  11.    listType);
  12. }

代码示例来源：origin: apache/incubator-gobblin

  1. private TaskState getStreamingTaskState() {
  2.  WorkUnitState workUnitState = new WorkUnitState(WorkUnit.create(
  3.    new Extract(Extract.TableType.SNAPSHOT_ONLY, this.getClass().getName(), this.getClass().getSimpleName())));
  4.  workUnitState.setProp(ConfigurationKeys.TASK_KEY_KEY, "1234");
  5.  TaskState taskState = new TaskState(workUnitState);
  6.  taskState.setProp(ConfigurationKeys.METRICS_ENABLED_KEY, Boolean.toString(false));
  7.  taskState.setProp(TaskConfigurationKeys.TASK_EXECUTION_MODE, ExecutionModel.STREAMING.name());
  8.  taskState.setJobId("1234");
  9.  taskState.setTaskId("testContinuousTaskId");
  10.  return taskState;
  11. }

代码示例来源：origin: apache/incubator-gobblin

  1. private JsonObject initResources(String resourceFilePath) {
  2.  Type listType = new TypeToken<JsonObject>() {
  3.  }.getType();
  4.  Gson gson = new Gson();
  5.  JsonObject testData =
  6.    gson.fromJson(new InputStreamReader(this.getClass().getResourceAsStream(resourceFilePath)), listType);
  7.  jsonRecord = testData.get("record").getAsJsonObject();
  8.  jsonSchema = testData.get("schema").getAsJsonArray();
  9.  WorkUnit workUnit = new WorkUnit(new SourceState(),
  10.    new Extract(new SourceState(), Extract.TableType.SNAPSHOT_ONLY, "namespace", "dummy_table"));
  11.  state = new WorkUnitState(workUnit);
  12.  state.setProp(ConfigurationKeys.CONVERTER_AVRO_TIME_FORMAT, "HH:mm:ss");
  13.  state.setProp(ConfigurationKeys.CONVERTER_AVRO_DATE_TIMEZONE, "PST");
  14.  return testData;
  15. }

代码示例来源：origin: apache/incubator-gobblin

  1. @Test
  2. public void testGetDefaultWriterFilePath() {
  3.  String namespace = "gobblin.test";
  4.  String tableName = "test-table";
  5.  SourceState sourceState = new SourceState();
  6.  WorkUnit state = WorkUnit.create(new Extract(sourceState, TableType.APPEND_ONLY, namespace, tableName));
  7.  Assert.assertEquals(WriterUtils.getWriterFilePath(state, 0, 0), new Path(state.getExtract().getOutputFilePath()));
  8.  Assert.assertEquals(WriterUtils.getWriterFilePath(state, 2, 0), new Path(state.getExtract().getOutputFilePath(),
  9.    ConfigurationKeys.DEFAULT_FORK_BRANCH_NAME + "0"));
  10. }

代码示例来源：origin: apache/incubator-gobblin

  1. TaskState getEmptyTestTaskState(String taskId) {
  2.  // Create a TaskState
  3.  WorkUnit workUnit = WorkUnit.create(
  4.    new Extract(Extract.TableType.SNAPSHOT_ONLY, this.getClass().getName(), this.getClass().getSimpleName()));
  5.  workUnit.setProp(ConfigurationKeys.TASK_KEY_KEY, "taskKey");
  6.  TaskState taskState = new TaskState(new WorkUnitState(workUnit));
  7.  taskState.setProp(ConfigurationKeys.METRICS_ENABLED_KEY, Boolean.toString(false));
  8.  taskState.setTaskId(taskId);
  9.  taskState.setJobId("1234");
  10.  return taskState;
  11. }

代码示例来源：origin: apache/incubator-gobblin

  1. TaskState getEmptyTestTaskState(String taskId) {
  2.  // Create a TaskState
  3.  WorkUnit workUnit = WorkUnit.create(
  4.      new Extract(Extract.TableType.SNAPSHOT_ONLY, this.getClass().getName(), this.getClass().getSimpleName()));
  5.  workUnit.setProp(ConfigurationKeys.TASK_KEY_KEY, "taskKey");
  6.  TaskState taskState = new TaskState(new WorkUnitState(workUnit));
  7.  taskState.setProp(ConfigurationKeys.METRICS_ENABLED_KEY, Boolean.toString(false));
  8.  taskState.setTaskId(taskId);
  9.  taskState.setJobId("1234");
  10.  return taskState;
  11. }

代码示例来源：origin: apache/incubator-gobblin

  1. @Test
  2. public void testGetDefaultWriterFilePathWithWorkUnitState() {
  3.  String namespace = "gobblin.test";
  4.  String tableName = "test-table";
  5.  SourceState sourceState = new SourceState();
  6.  WorkUnit workUnit = WorkUnit.create(new Extract(sourceState, TableType.APPEND_ONLY, namespace, tableName));
  7.  WorkUnitState workUnitState = new WorkUnitState(workUnit);
  8.  Assert.assertEquals(WriterUtils.getWriterFilePath(workUnitState, 0, 0), new Path(workUnitState.getExtract()
  9.    .getOutputFilePath()));
  10.  Assert.assertEquals(WriterUtils.getWriterFilePath(workUnitState, 2, 0), new Path(workUnitState.getExtract()
  11.    .getOutputFilePath(), ConfigurationKeys.DEFAULT_FORK_BRANCH_NAME + "0"));
  12. }

代码示例来源：origin: apache/incubator-gobblin

  1. @Test
  2. public void testGetWriterFilePath() {
  3.  Extract extract = new Extract(TableType.SNAPSHOT_ONLY, "org.apache.gobblin.dbNamespace", "tableName");
  4.  WorkUnit state = WorkUnit.create(extract);
  5.  state.setProp(ConfigurationKeys.WRITER_FILE_PATH, TEST_WRITER_FILE_PATH);
  6.  Assert.assertEquals(WriterUtils.getWriterFilePath(state, 0, 0), TEST_WRITER_FILE_PATH);
  7.  state.setProp(ConfigurationKeys.WRITER_FILE_PATH + ".0", TEST_WRITER_FILE_PATH);
  8.  Assert.assertEquals(WriterUtils.getWriterFilePath(state, 1, 1), TEST_WRITER_FILE_PATH);
  9.  state.removeProp(ConfigurationKeys.WRITER_FILE_PATH);
  10.  state.setProp(ConfigurationKeys.WRITER_FILE_PATH_TYPE, "tablename");
  11.  Assert.assertEquals(WriterUtils.getWriterFilePath(state, 0, 0), new Path("tableName"));
  12.  state.setProp(ConfigurationKeys.WRITER_FILE_PATH_TYPE, "namespace_table");
  13.  Assert.assertEquals(WriterUtils.getWriterFilePath(state, 0, 0),
  14.    new Path("org/apache/gobblin/dbNamespace/tableName"));
  15. }

代码示例来源：origin: apache/incubator-gobblin

  1. public static WorkUnitState getWorkUnitState1() {
  2.   WorkUnit wu = new WorkUnit(new Extract(Extract.TableType.APPEND_ONLY, "namespace", "table"));
  3.   wu.setWatermarkInterval(
  4.     new WatermarkInterval(new LongWatermark(20160101235959L), new LongWatermark(20160102235959L)));
  5.   State js = new State();
  6.   return new WorkUnitState(wu, js);
  7.  }
  8. }

代码示例来源：origin: apache/incubator-gobblin

  1. String extractId = fileSet.getName().replace(':', '_');
  2. Extract extract = new Extract(Extract.TableType.SNAPSHOT_ONLY, CopyConfiguration.COPY_PREFIX, extractId);
  3. List<WorkUnit> workUnitsForPartition = Lists.newArrayList();
  4. for (CopyEntity copyEntity : fileSet.getFiles()) {

代码示例来源：origin: apache/incubator-gobblin

  1. SourceEntity sourceEntity = SourceEntity.fromSourceEntityName(sourceEntityName);
  2. sourceEntities[i] = sourceEntity;
  3. extracts[i] = new Extract(TableType.APPEND_ONLY, "", sourceEntity.getDestTableName());
  4. for (int j = 0; j < 3; ++j) {
  5.  WorkUnit wu = new WorkUnit(extracts[i]);

内容来源于网络，如有侵权，请联系作者删除！

Extract

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com