org.apache.beam.sdk.transforms.Flatten.pCollections()方法的使用及代码示例

x33g5p2x  于2022-01-19 转载在 其他  
字(10.5k)|赞(0)|评价(0)|浏览(156)

本文整理了Java中org.apache.beam.sdk.transforms.Flatten.pCollections()方法的一些代码示例，展示了Flatten.pCollections()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Flatten.pCollections()方法的具体详情如下：
包路径：org.apache.beam.sdk.transforms.Flatten
类名称：Flatten
方法名：pCollections

Flatten.pCollections介绍

[英]Returns a PTransform that flattens a PCollectionList into a PCollectioncontaining all the elements of all the PCollections in its input.

All inputs must have equal WindowFns. The output elements of Flatten are in the same windows and have the same timestamps as their corresponding input elements. The output PCollection will have the same WindowFn as all of the inputs.
[中]返回一个PTransform，该PTransform将pCollection列表展平为pCollection，该pCollection包含其输入中所有pCollection的所有元素。
所有输入必须具有相等的窗口fn。Flatten的输出元素位于相同的窗口中，并且与相应的输入元素具有相同的时间戳。输出PCollection将具有与所有输入相同的窗口fn。

代码示例

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Override
 public PCollection<T> expand(PBegin input) {
  PCollectionList<T> empty = PCollectionList.empty(input.getPipeline());
  return empty.apply(Flatten.pCollections());
 }
}

代码示例来源：origin: org.apache.beam/beam-runners-core-construction-java

@Test
public void getInputEmptySucceeds() {
 PTransformReplacement<PCollectionList<Long>, PCollection<Long>> replacement =
   factory.getReplacementTransform(
     AppliedPTransform.of(
       "nonEmptyInput",
       Collections.emptyMap(),
       Collections.emptyMap(),
       Flatten.pCollections(),
       pipeline));
 assertThat(replacement.getInput().getAll(), emptyIterable());
}

代码示例来源：origin: org.apache.beam/beam-runners-core-construction-java

@Test
public void emptyFlattenWithEmptyFlatten() {
 AppliedPTransform application =
   AppliedPTransform.of(
     "EmptyFlatten",
     Collections.emptyMap(),
     Collections.singletonMap(
       new TupleTag<Integer>(),
       PCollection.createPrimitiveOutputInternal(
         p, WindowingStrategy.globalDefault(), IsBounded.BOUNDED, VarIntCoder.of())),
     Flatten.pCollections(),
     p);
 assertThat(PTransformMatchers.emptyFlatten().matches(application), is(true));
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(ValidatesRunner.class)
public void testFlattenPCollectionsSingletonList() {
 PCollection<String> input = p.apply(Create.of(LINES));
 PCollection<String> output = PCollectionList.of(input).apply(Flatten.pCollections());
 assertThat(output, not(equalTo(input)));
 PAssert.that(output).containsInAnyOrder(LINES);
 p.run();
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(ValidatesRunner.class)
public void testFlattenPCollections() {
 List<List<String>> inputs = Arrays.asList(LINES, NO_LINES, LINES2, NO_LINES, LINES, NO_LINES);
 PCollection<String> output =
   makePCollectionListOfStrings(p, inputs).apply(Flatten.pCollections());
 PAssert.that(output).containsInAnyOrder(flattenLists(inputs));
 p.run();
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
public void testFlattenGetName() {
 Assert.assertEquals("Flatten.Iterables", Flatten.<String>iterables().getName());
 Assert.assertEquals("Flatten.PCollections", Flatten.<String>pCollections().getName());
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(NeedsRunner.class)
public void testFlattenNoListsNoCoder() {
 // not ValidatesRunner because it should fail at pipeline construction time anyhow.
 thrown.expect(IllegalStateException.class);
 thrown.expectMessage("Unable to return a default Coder");
 PCollectionList.<ClassWithoutCoder>empty(p).apply(Flatten.pCollections());
 p.run();
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(ValidatesRunner.class)
public void testFlattenPCollectionsThenParDo() {
 List<List<String>> inputs = Arrays.asList(LINES, NO_LINES, LINES2, NO_LINES, LINES, NO_LINES);
 PCollection<String> output =
   makePCollectionListOfStrings(p, inputs)
     .apply(Flatten.pCollections())
     .apply(ParDo.of(new IdentityFn<>()));
 PAssert.that(output).containsInAnyOrder(flattenLists(inputs));
 p.run();
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(ValidatesRunner.class)
public void testFlattenPCollectionsEmpty() {
 PCollection<String> output =
   PCollectionList.<String>empty(p)
     .apply(Flatten.pCollections())
     .setCoder(StringUtf8Coder.of());
 PAssert.that(output).empty();
 p.run();
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(NeedsRunner.class)
public void testDroppedPartition() {
 // Compute the set of integers either 1 or 2 mod 3, the hard way.
 PCollectionList<Integer> outputs =
   pipeline
     .apply(Create.of(2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12))
     .apply(Partition.of(3, new ModFn()));
 List<PCollection<Integer>> outputsList = new ArrayList<>(outputs.getAll());
 outputsList.remove(0);
 outputs = PCollectionList.of(outputsList);
 assertTrue(outputs.size() == 2);
 PCollection<Integer> output = outputs.apply(Flatten.pCollections());
 PAssert.that(output).containsInAnyOrder(2, 4, 5, 7, 8, 10, 11);
 pipeline.run();
}

代码示例来源：origin: org.apache.beam/beam-runners-apex

@Test
 public void testFlattenSingleCollection() {
  ApexPipelineOptions options = PipelineOptionsFactory.as(ApexPipelineOptions.class);
  Pipeline p = Pipeline.create();
  PCollection<String> single = p.apply(Create.of(Collections.singletonList("1")));
  PCollectionList.of(single)
    .apply(Flatten.pCollections())
    .apply(ParDo.of(new EmbeddedCollector()));
  DAG dag = TestApexRunner.translate(p, options);
  Assert.assertNotNull(
    dag.getOperatorMeta("ParDo(EmbeddedCollector)/ParMultiDo(EmbeddedCollector)"));
 }
}

代码示例来源：origin: org.apache.beam/beam-runners-direct-java

@Test
public void getValueToConsumersWithDuplicateInputSucceeds() {
 PCollection<String> created = p.apply(Create.of("1", "2", "3"));
 PCollection<String> flattened =
   PCollectionList.of(created).and(created).apply(Flatten.pCollections());
 p.traverseTopologically(visitor);
 DirectGraph graph = visitor.getGraph();
 AppliedPTransform<?, ?, ?> flattenedProducer = graph.getProducer(flattened);
 assertThat(
   graph.getPerElementConsumers(created),
   Matchers.containsInAnyOrder(new Object[] {flattenedProducer, flattenedProducer}));
 assertThat(graph.getPerElementConsumers(flattened), emptyIterable());
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(ValidatesRunner.class)
public void testFlattenPCollectionsEmptyThenParDo() {
 PCollection<String> output =
   PCollectionList.<String>empty(p)
     .apply(Flatten.pCollections())
     .setCoder(StringUtf8Coder.of())
     .apply(ParDo.of(new IdentityFn<>()));
 PAssert.that(output).empty();
 p.run();
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category({ValidatesRunner.class, UsesParDoLifecycle.class})
public void testFnCallSequence() {
 PCollectionList.of(p.apply("Impolite", Create.of(1, 2, 4)))
   .and(p.apply("Polite", Create.of(3, 5, 6, 7)))
   .apply(Flatten.pCollections())
   .apply(ParDo.of(new CallSequenceEnforcingFn<>()));
 p.run();
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category({ValidatesRunner.class, UsesParDoLifecycle.class})
public void testFnCallSequenceMulti() {
 PCollectionList.of(p.apply("Impolite", Create.of(1, 2, 4)))
   .and(p.apply("Polite", Create.of(3, 5, 6, 7)))
   .apply(Flatten.pCollections())
   .apply(
     ParDo.of(new CallSequenceEnforcingFn<Integer>())
       .withOutputTags(new TupleTag<Integer>() {}, TupleTagList.empty()));
 p.run();
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(ValidatesRunner.class)
public void testMultipleApply() {
 PTransform<PCollection<? extends String>, PCollection<String>> myTransform = addSuffix("+");
 PCollection<String> input = pipeline.apply(Create.of(ImmutableList.of("a", "b")));
 PCollection<String> left = input.apply("Left1", myTransform).apply("Left2", myTransform);
 PCollection<String> right = input.apply("Right", myTransform);
 PCollection<String> both = PCollectionList.of(left).and(right).apply(Flatten.pCollections());
 PAssert.that(both).containsInAnyOrder("a++", "b++", "a+", "b+");
 pipeline.run();
}

代码示例来源：origin: org.apache.beam/beam-runners-core-construction-java

@Test
 public void outputMapping() {
  final PCollectionList<String> inputList =
    PCollectionList.of(first).and(second).and(first).and(first);
  PCollection<String> original = inputList.apply(Flatten.pCollections());
  PCollection<String> replacement = inputList.apply(new FlattenWithoutDuplicateInputs<>());

  assertThat(
    factory.mapOutputs(original.expand(), replacement),
    Matchers.hasEntry(
      replacement,
      ReplacementOutput.of(
        TaggedPValue.ofExpandedValue(original),
        TaggedPValue.ofExpandedValue(replacement))));
 }
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category({ValidatesRunner.class, UsesStatefulParDo.class, UsesParDoLifecycle.class})
public void testFnCallSequenceStateful() {
 PCollectionList.of(p.apply("Impolite", Create.of(KV.of("a", 1), KV.of("b", 2), KV.of("a", 4))))
   .and(
     p.apply(
       "Polite", Create.of(KV.of("b", 3), KV.of("a", 5), KV.of("c", 6), KV.of("c", 7))))
   .apply(Flatten.pCollections())
   .apply(
     ParDo.of(new CallSequenceEnforcingStatefulFn<String, Integer>())
       .withOutputTags(new TupleTag<KV<String, Integer>>() {}, TupleTagList.empty()));
 p.run();
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(NeedsRunner.class)
public void testEqualWindowFnPropagation() {
 PCollection<String> input1 =
   p.apply("CreateInput1", Create.of("Input1"))
     .apply("Window1", Window.into(FixedWindows.of(Duration.standardMinutes(1))));
 PCollection<String> input2 =
   p.apply("CreateInput2", Create.of("Input2"))
     .apply("Window2", Window.into(FixedWindows.of(Duration.standardMinutes(1))));
 PCollection<String> output =
   PCollectionList.of(input1).and(input2).apply(Flatten.pCollections());
 p.run();
 Assert.assertTrue(
   output
     .getWindowingStrategy()
     .getWindowFn()
     .isCompatible(FixedWindows.of(Duration.standardMinutes(1))));
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(NeedsRunner.class)
public void testCompatibleWindowFnPropagation() {
 PCollection<String> input1 =
   p.apply("CreateInput1", Create.of("Input1"))
     .apply("Window1", Window.into(Sessions.withGapDuration(Duration.standardMinutes(1))));
 PCollection<String> input2 =
   p.apply("CreateInput2", Create.of("Input2"))
     .apply("Window2", Window.into(Sessions.withGapDuration(Duration.standardMinutes(2))));
 PCollection<String> output =
   PCollectionList.of(input1).and(input2).apply(Flatten.pCollections());
 p.run();
 Assert.assertTrue(
   output
     .getWindowingStrategy()
     .getWindowFn()
     .isCompatible(Sessions.withGapDuration(Duration.standardMinutes(2))));
}

内容来源于网络，如有侵权，请联系作者删除！

Flatten

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com