org.apache.beam.sdk.io.WriteFilesResult.expand()方法的使用及代码示例

x33g5p2x  于2022-02-03 转载在 其他  
字(2.6k)|赞(0)|评价(0)|浏览(91)

本文整理了Java中org.apache.beam.sdk.io.WriteFilesResult.expand()方法的一些代码示例,展示了WriteFilesResult.expand()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。WriteFilesResult.expand()方法的具体详情如下:
包路径:org.apache.beam.sdk.io.WriteFilesResult
类名称:WriteFilesResult
方法名:expand

WriteFilesResult.expand介绍

暂无

代码示例

代码示例来源:origin: org.apache.beam/beam-runners-google-cloud-dataflow-java

private void testStreamingWriteOverride(PipelineOptions options, int expectedNumShards) {
 TestPipeline p = TestPipeline.fromOptions(options);
 StreamingShardedWriteFactory<Object, Void, Object> factory =
   new StreamingShardedWriteFactory<>(p.getOptions());
 WriteFiles<Object, Void, Object> original = WriteFiles.to(new TestSink(tmpFolder.toString()));
 PCollection<Object> objs = (PCollection) p.apply(Create.empty(VoidCoder.of()));
 AppliedPTransform<PCollection<Object>, WriteFilesResult<Void>, WriteFiles<Object, Void, Object>>
   originalApplication =
     AppliedPTransform.of("writefiles", objs.expand(), Collections.emptyMap(), original, p);
 WriteFiles<Object, Void, Object> replacement =
   (WriteFiles<Object, Void, Object>)
     factory.getReplacementTransform(originalApplication).getTransform();
 assertThat(replacement, not(equalTo((Object) original)));
 assertThat(replacement.getNumShardsProvider().get(), equalTo(expectedNumShards));
 WriteFilesResult<Void> originalResult = objs.apply(original);
 WriteFilesResult<Void> replacementResult = objs.apply(replacement);
 Map<PValue, ReplacementOutput> res =
   factory.mapOutputs(originalResult.expand(), replacementResult);
 assertEquals(1, res.size());
 assertEquals(
   originalResult.getPerDestinationOutputFilenames(),
   res.get(replacementResult.getPerDestinationOutputFilenames()).getOriginal().getValue());
}

代码示例来源:origin: org.apache.beam/beam-runners-core-construction-java

@Test
public void testExtractionDirectFromTransform() throws Exception {
 PCollection<String> input = p.apply(Create.of("hello"));
 WriteFilesResult<Void> output = input.apply(writeFiles);
 AppliedPTransform<PCollection<String>, WriteFilesResult<Void>, WriteFiles<String, Void, String>>
   appliedPTransform =
     AppliedPTransform.of("foo", input.expand(), output.expand(), writeFiles, p);
 assertThat(
   WriteFilesTranslation.isRunnerDeterminedSharding(appliedPTransform),
   equalTo(
     writeFiles.getNumShardsProvider() == null && writeFiles.getComputeNumShards() == null));
 assertThat(
   WriteFilesTranslation.isWindowedWrites(appliedPTransform),
   equalTo(writeFiles.getWindowedWrites()));
 assertThat(
   WriteFilesTranslation.<String, Void, String>getSink(appliedPTransform),
   equalTo(writeFiles.getSink()));
}

相关文章