本文整理了Java中org.apache.gobblin.source.workunit.Extract.getOutputFilePath()
方法的一些代码示例,展示了Extract.getOutputFilePath()
的具体用法。这些代码示例主要来源于Github
/Stackoverflow
/Maven
等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。Extract.getOutputFilePath()
方法的具体详情如下:
包路径:org.apache.gobblin.source.workunit.Extract
类名称:Extract
方法名:getOutputFilePath
[英]Get the writer output file path corresponding to this Extract.
[中]获取与此提取对应的写入器输出文件路径。
代码示例来源:origin: apache/incubator-gobblin
@Test
public void testGetDefaultWriterFilePath() {
String namespace = "gobblin.test";
String tableName = "test-table";
SourceState sourceState = new SourceState();
WorkUnit state = WorkUnit.create(new Extract(sourceState, TableType.APPEND_ONLY, namespace, tableName));
Assert.assertEquals(WriterUtils.getWriterFilePath(state, 0, 0), new Path(state.getExtract().getOutputFilePath()));
Assert.assertEquals(WriterUtils.getWriterFilePath(state, 2, 0), new Path(state.getExtract().getOutputFilePath(),
ConfigurationKeys.DEFAULT_FORK_BRANCH_NAME + "0"));
}
代码示例来源:origin: apache/incubator-gobblin
@Test
public void testGetDefaultWriterFilePathWithWorkUnitState() {
String namespace = "gobblin.test";
String tableName = "test-table";
SourceState sourceState = new SourceState();
WorkUnit workUnit = WorkUnit.create(new Extract(sourceState, TableType.APPEND_ONLY, namespace, tableName));
WorkUnitState workUnitState = new WorkUnitState(workUnit);
Assert.assertEquals(WriterUtils.getWriterFilePath(workUnitState, 0, 0), new Path(workUnitState.getExtract()
.getOutputFilePath()));
Assert.assertEquals(WriterUtils.getWriterFilePath(workUnitState, 2, 0), new Path(workUnitState.getExtract()
.getOutputFilePath(), ConfigurationKeys.DEFAULT_FORK_BRANCH_NAME + "0"));
}
代码示例来源:origin: apache/incubator-gobblin
public void runTestWithFork(Properties jobProps) throws Exception {
String jobName = jobProps.getProperty(ConfigurationKeys.JOB_NAME_KEY);
String jobId = JobLauncherUtils.newJobId(jobName).toString();
jobProps.setProperty(ConfigurationKeys.JOB_ID_KEY, jobId);
try (JobLauncher jobLauncher = JobLauncherFactory.newJobLauncher(this.launcherProps, jobProps)) {
jobLauncher.launchJob(null);
}
List<JobState.DatasetState> datasetStateList = this.datasetStateStore.getAll(jobName, sanitizeJobNameForDatasetStore(jobId) + ".jst");
DatasetState datasetState = datasetStateList.get(0);
Assert.assertEquals(datasetState.getState(), JobState.RunningState.COMMITTED);
Assert.assertEquals(datasetState.getCompletedTasks(), 4);
Assert.assertEquals(datasetState.getJobFailures(), 0);
FileSystem lfs = FileSystem.getLocal(new Configuration());
for (TaskState taskState : datasetState.getTaskStates()) {
Assert.assertEquals(taskState.getWorkingState(), WorkUnitState.WorkingState.COMMITTED);
Path path = new Path(this.launcherProps.getProperty(ConfigurationKeys.DATA_PUBLISHER_FINAL_DIR),
new Path(taskState.getExtract().getOutputFilePath(), "fork_0"));
Assert.assertTrue(lfs.exists(path));
Assert.assertEquals(lfs.listStatus(path).length, 2);
Assert.assertEquals(taskState.getPropAsLong(ConfigurationKeys.WRITER_RECORDS_WRITTEN + ".0"),
TestExtractor.TOTAL_RECORDS);
path = new Path(this.launcherProps.getProperty(ConfigurationKeys.DATA_PUBLISHER_FINAL_DIR),
new Path(taskState.getExtract().getOutputFilePath(), "fork_1"));
Assert.assertTrue(lfs.exists(path));
Assert.assertEquals(lfs.listStatus(path).length, 2);
Assert.assertEquals(taskState.getPropAsLong(ConfigurationKeys.WRITER_RECORDS_WRITTEN + ".1"),
TestExtractor.TOTAL_RECORDS);
}
}
代码示例来源:origin: apache/incubator-gobblin
ForkOperatorUtils.getPathForBranch(state, state.getExtract().getOutputFilePath(), 2, 0));
fs.mkdirs(writerStagingPath0);
ForkOperatorUtils.getPathForBranch(state, state.getExtract().getOutputFilePath(), 2, 1));
fs.mkdirs(writerStagingPath1);
ForkOperatorUtils.getPathForBranch(state, state.getExtract().getOutputFilePath(), 2, 0));
fs.mkdirs(writerOutputPath0);
ForkOperatorUtils.getPathForBranch(state, state.getExtract().getOutputFilePath(), 2, 1));
fs.mkdirs(writerOutputPath1);
内容来源于网络,如有侵权,请联系作者删除!