本文整理了Java中org.apache.beam.sdk.io.WriteFilesResult.expand()
方法的一些代码示例,展示了WriteFilesResult.expand()
的具体用法。这些代码示例主要来源于Github
/Stackoverflow
/Maven
等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。WriteFilesResult.expand()
方法的具体详情如下:
包路径:org.apache.beam.sdk.io.WriteFilesResult
类名称:WriteFilesResult
方法名:expand
暂无
代码示例来源:origin: org.apache.beam/beam-runners-google-cloud-dataflow-java
private void testStreamingWriteOverride(PipelineOptions options, int expectedNumShards) {
TestPipeline p = TestPipeline.fromOptions(options);
StreamingShardedWriteFactory<Object, Void, Object> factory =
new StreamingShardedWriteFactory<>(p.getOptions());
WriteFiles<Object, Void, Object> original = WriteFiles.to(new TestSink(tmpFolder.toString()));
PCollection<Object> objs = (PCollection) p.apply(Create.empty(VoidCoder.of()));
AppliedPTransform<PCollection<Object>, WriteFilesResult<Void>, WriteFiles<Object, Void, Object>>
originalApplication =
AppliedPTransform.of("writefiles", objs.expand(), Collections.emptyMap(), original, p);
WriteFiles<Object, Void, Object> replacement =
(WriteFiles<Object, Void, Object>)
factory.getReplacementTransform(originalApplication).getTransform();
assertThat(replacement, not(equalTo((Object) original)));
assertThat(replacement.getNumShardsProvider().get(), equalTo(expectedNumShards));
WriteFilesResult<Void> originalResult = objs.apply(original);
WriteFilesResult<Void> replacementResult = objs.apply(replacement);
Map<PValue, ReplacementOutput> res =
factory.mapOutputs(originalResult.expand(), replacementResult);
assertEquals(1, res.size());
assertEquals(
originalResult.getPerDestinationOutputFilenames(),
res.get(replacementResult.getPerDestinationOutputFilenames()).getOriginal().getValue());
}
代码示例来源:origin: org.apache.beam/beam-runners-core-construction-java
@Test
public void testExtractionDirectFromTransform() throws Exception {
PCollection<String> input = p.apply(Create.of("hello"));
WriteFilesResult<Void> output = input.apply(writeFiles);
AppliedPTransform<PCollection<String>, WriteFilesResult<Void>, WriteFiles<String, Void, String>>
appliedPTransform =
AppliedPTransform.of("foo", input.expand(), output.expand(), writeFiles, p);
assertThat(
WriteFilesTranslation.isRunnerDeterminedSharding(appliedPTransform),
equalTo(
writeFiles.getNumShardsProvider() == null && writeFiles.getComputeNumShards() == null));
assertThat(
WriteFilesTranslation.isWindowedWrites(appliedPTransform),
equalTo(writeFiles.getWindowedWrites()));
assertThat(
WriteFilesTranslation.<String, Void, String>getSink(appliedPTransform),
equalTo(writeFiles.getSink()));
}
内容来源于网络,如有侵权,请联系作者删除!