org.apache.gobblin.source.workunit.Extract.getOutputFilePath()方法的使用及代码示例

x33g5p2x  于2022-01-19 转载在 其他  
字(3.9k)|赞(0)|评价(0)|浏览(146)

本文整理了Java中org.apache.gobblin.source.workunit.Extract.getOutputFilePath()方法的一些代码示例,展示了Extract.getOutputFilePath()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。Extract.getOutputFilePath()方法的具体详情如下:
包路径:org.apache.gobblin.source.workunit.Extract
类名称:Extract
方法名:getOutputFilePath

Extract.getOutputFilePath介绍

[英]Get the writer output file path corresponding to this Extract.
[中]获取与此提取对应的写入器输出文件路径。

代码示例

代码示例来源:origin: apache/incubator-gobblin

@Test
public void testGetDefaultWriterFilePath() {
 String namespace = "gobblin.test";
 String tableName = "test-table";
 SourceState sourceState = new SourceState();
 WorkUnit state = WorkUnit.create(new Extract(sourceState, TableType.APPEND_ONLY, namespace, tableName));
 Assert.assertEquals(WriterUtils.getWriterFilePath(state, 0, 0), new Path(state.getExtract().getOutputFilePath()));
 Assert.assertEquals(WriterUtils.getWriterFilePath(state, 2, 0), new Path(state.getExtract().getOutputFilePath(),
   ConfigurationKeys.DEFAULT_FORK_BRANCH_NAME + "0"));
}

代码示例来源:origin: apache/incubator-gobblin

@Test
public void testGetDefaultWriterFilePathWithWorkUnitState() {
 String namespace = "gobblin.test";
 String tableName = "test-table";
 SourceState sourceState = new SourceState();
 WorkUnit workUnit = WorkUnit.create(new Extract(sourceState, TableType.APPEND_ONLY, namespace, tableName));
 WorkUnitState workUnitState = new WorkUnitState(workUnit);
 Assert.assertEquals(WriterUtils.getWriterFilePath(workUnitState, 0, 0), new Path(workUnitState.getExtract()
   .getOutputFilePath()));
 Assert.assertEquals(WriterUtils.getWriterFilePath(workUnitState, 2, 0), new Path(workUnitState.getExtract()
   .getOutputFilePath(), ConfigurationKeys.DEFAULT_FORK_BRANCH_NAME + "0"));
}

代码示例来源:origin: apache/incubator-gobblin

public void runTestWithFork(Properties jobProps) throws Exception {
 String jobName = jobProps.getProperty(ConfigurationKeys.JOB_NAME_KEY);
 String jobId = JobLauncherUtils.newJobId(jobName).toString();
 jobProps.setProperty(ConfigurationKeys.JOB_ID_KEY, jobId);
 try (JobLauncher jobLauncher = JobLauncherFactory.newJobLauncher(this.launcherProps, jobProps)) {
  jobLauncher.launchJob(null);
 }
 List<JobState.DatasetState> datasetStateList = this.datasetStateStore.getAll(jobName, sanitizeJobNameForDatasetStore(jobId) + ".jst");
 DatasetState datasetState = datasetStateList.get(0);
 Assert.assertEquals(datasetState.getState(), JobState.RunningState.COMMITTED);
 Assert.assertEquals(datasetState.getCompletedTasks(), 4);
 Assert.assertEquals(datasetState.getJobFailures(), 0);
 FileSystem lfs = FileSystem.getLocal(new Configuration());
 for (TaskState taskState : datasetState.getTaskStates()) {
  Assert.assertEquals(taskState.getWorkingState(), WorkUnitState.WorkingState.COMMITTED);
  Path path = new Path(this.launcherProps.getProperty(ConfigurationKeys.DATA_PUBLISHER_FINAL_DIR),
    new Path(taskState.getExtract().getOutputFilePath(), "fork_0"));
  Assert.assertTrue(lfs.exists(path));
  Assert.assertEquals(lfs.listStatus(path).length, 2);
  Assert.assertEquals(taskState.getPropAsLong(ConfigurationKeys.WRITER_RECORDS_WRITTEN + ".0"),
    TestExtractor.TOTAL_RECORDS);
  path = new Path(this.launcherProps.getProperty(ConfigurationKeys.DATA_PUBLISHER_FINAL_DIR),
    new Path(taskState.getExtract().getOutputFilePath(), "fork_1"));
  Assert.assertTrue(lfs.exists(path));
  Assert.assertEquals(lfs.listStatus(path).length, 2);
  Assert.assertEquals(taskState.getPropAsLong(ConfigurationKeys.WRITER_RECORDS_WRITTEN + ".1"),
    TestExtractor.TOTAL_RECORDS);
 }
}

代码示例来源:origin: apache/incubator-gobblin

ForkOperatorUtils.getPathForBranch(state, state.getExtract().getOutputFilePath(), 2, 0));
fs.mkdirs(writerStagingPath0);
  ForkOperatorUtils.getPathForBranch(state, state.getExtract().getOutputFilePath(), 2, 1));
fs.mkdirs(writerStagingPath1);
  ForkOperatorUtils.getPathForBranch(state, state.getExtract().getOutputFilePath(), 2, 0));
fs.mkdirs(writerOutputPath0);
  ForkOperatorUtils.getPathForBranch(state, state.getExtract().getOutputFilePath(), 2, 1));
fs.mkdirs(writerOutputPath1);

相关文章