org.apache.beam.sdk.transforms.Flatten.iterables()方法的使用及代码示例

x33g5p2x  于2022-01-19 转载在 其他  
字(8.3k)|赞(0)|评价(0)|浏览(153)

本文整理了Java中org.apache.beam.sdk.transforms.Flatten.iterables()方法的一些代码示例，展示了Flatten.iterables()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Flatten.iterables()方法的具体详情如下：
包路径：org.apache.beam.sdk.transforms.Flatten
类名称：Flatten
方法名：iterables

Flatten.iterables介绍

[英]Returns a PTransform that takes a PCollection> and returns a PCollection containing all the elements from all the Iterables.

Example of use:

PCollection> pcOfIterables = ...;

By default, the output PCollection encodes its elements using the same Coderthat the input uses for the elements in its Iterable.
[中]返回一个PTransform，该PTransform接受PCollection>并返回一个PCollection，该PCollection包含所有ITerable中的所有元素。
使用示例：

PCollection> pcOfIterables = ...;

默认情况下，输出PCollection使用与输入用于Iterable中元素相同的代码对其元素进行编码。

代码示例

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Override
public PCollection<T> expand(PCollection<T> in) {
 return in.apply(Combine.globally(new SampleAnyCombineFn<T>(limit)).withoutDefaults())
   .apply(Flatten.iterables());
}

代码示例来源：origin: org.apache.beam/beam-runners-core-construction-java

@Test
public void emptyFlattenWithNonFlatten() {
 AppliedPTransform application =
   AppliedPTransform
     .<PCollection<Iterable<Integer>>, PCollection<Integer>, Flatten.Iterables<Integer>>of(
       "EmptyFlatten",
       Collections.emptyMap(),
       Collections.singletonMap(
         new TupleTag<Integer>(),
         PCollection.createPrimitiveOutputInternal(
           p, WindowingStrategy.globalDefault(), IsBounded.BOUNDED, VarIntCoder.of())),
       /* This isn't actually possible to construct, but for the sake of example */
       Flatten.iterables(),
       p);
 assertThat(PTransformMatchers.emptyFlatten().matches(application), is(false));
}

代码示例来源：origin: org.apache.beam/beam-runners-core-construction-java

@Test
public void flattenWithDuplicateInputsNonFlatten() {
 AppliedPTransform application =
   AppliedPTransform
     .<PCollection<Iterable<Integer>>, PCollection<Integer>, Flatten.Iterables<Integer>>of(
       "EmptyFlatten",
       Collections.emptyMap(),
       Collections.singletonMap(
         new TupleTag<Integer>(),
         PCollection.createPrimitiveOutputInternal(
           p, WindowingStrategy.globalDefault(), IsBounded.BOUNDED, VarIntCoder.of())),
       /* This isn't actually possible to construct, but for the sake of example */
       Flatten.iterables(),
       p);
 assertThat(PTransformMatchers.flattenWithDuplicateInputs().matches(application), is(false));
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
public void testFlattenGetName() {
 Assert.assertEquals("Flatten.Iterables", Flatten.<String>iterables().getName());
 Assert.assertEquals("Flatten.PCollections", Flatten.<String>pCollections().getName());
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Override
 public PCollection<T> expand(PCollection<T> input) {
  return input
    .apply(WithKeys.of(""))
    .apply(GroupByKey.create())
    .apply(Values.create())
    .apply(Flatten.iterables());
 }
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Override
 public PCollectionView<Integer> expand(PCollection<String> input) {
  return input
    .apply(
      ParDo.of(
        new DoFn<String, Integer>() {
         @ProcessElement
         public void toInteger(ProcessContext ctxt) {
          ctxt.output(Integer.valueOf(ctxt.element()));
         }
        }))
    .apply(Top.largest(1))
    .apply(Flatten.iterables())
    .apply(View.asSingleton());
 }
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-extensions-sql

.apply("flatten", Flatten.iterables())
.setSchema(
  CalciteUtils.toSchema(getRowType()),

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

p.apply("ReadMyFile", TextIO.read().from(inputFile.getPath()))
  .apply(sample)
  .apply(Flatten.iterables())
  .apply("WriteMyFile", TextIO.write().to(outputFile.getPath()));

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Override
 public PCollectionView<ActualT> expand(PBegin input) {
  final Coder<T> coder = actual.getCoder();
  return actual
    .apply("FilterActuals", rewindowActuals.prepareActuals())
    .apply("GatherPanes", GatherAllPanes.globally())
    .apply("ExtractPane", MapElements.via(extractPane))
    .setCoder(IterableCoder.of(actual.getCoder()))
    .apply(Flatten.iterables())
    .apply("RewindowActuals", rewindowActuals.windowActuals())
    .apply(
      ParDo.of(
        new DoFn<T, T>() {
         @ProcessElement
         public void processElement(ProcessContext context) throws CoderException {
          context.output(CoderUtils.clone(coder, context.element()));
         }
        }))
    .apply(actualView);
 }
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(ValidatesRunner.class)
public void testFlattenIterablesEmpty() {
 PCollection<Iterable<String>> input =
   p.apply(
     Create.<Iterable<String>>of(NO_LINES)
       .withCoder(IterableCoder.of(StringUtf8Coder.of())));
 PCollection<String> output = input.apply(Flatten.iterables());
 PAssert.that(output).containsInAnyOrder(NO_LINES_ARRAY);
 p.run();
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(ValidatesRunner.class)
public void testFlattenIterables() {
 PCollection<Iterable<String>> input =
   p.apply(
     Create.<Iterable<String>>of(LINES).withCoder(IterableCoder.of(StringUtf8Coder.of())));
 PCollection<String> output = input.apply(Flatten.iterables());
 PAssert.that(output).containsInAnyOrder(LINES_ARRAY);
 p.run();
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(ValidatesRunner.class)
public void testFlattenIterablesLists() {
 PCollection<List<String>> input =
   p.apply(Create.<List<String>>of(LINES).withCoder(ListCoder.of(StringUtf8Coder.of())));
 PCollection<String> output = input.apply(Flatten.iterables());
 PAssert.that(output).containsInAnyOrder(LINES_ARRAY);
 p.run();
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(ValidatesRunner.class)
public void testFlattenIterablesSets() {
 Set<String> linesSet = ImmutableSet.copyOf(LINES);
 PCollection<Set<String>> input =
   p.apply(Create.<Set<String>>of(linesSet).withCoder(SetCoder.of(StringUtf8Coder.of())));
 PCollection<String> output = input.apply(Flatten.iterables());
 PAssert.that(output).containsInAnyOrder(LINES_ARRAY);
 p.run();
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(ValidatesRunner.class)
public void testFlattenIterablesCollections() {
 Set<String> linesSet = ImmutableSet.copyOf(LINES);
 PCollection<Collection<String>> input =
   p.apply(
     Create.<Collection<String>>of(linesSet)
       .withCoder(CollectionCoder.of(StringUtf8Coder.of())));
 PCollection<String> output = input.apply(Flatten.iterables());
 PAssert.that(output).containsInAnyOrder(LINES_ARRAY);
 p.run();
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(NeedsRunner.class)
public void testOutputCoders() {
 Schema keySchema = Schema.builder().addStringField("field1").build();
 PCollection<KV<Row, Iterable<POJO>>> grouped =
   pipeline
     .apply(Create.of(new POJO("key1", 1, "value1")))
     .apply(Group.byFieldNames("field1"));
 // Make sure that the key has the right schema.
 PCollection<Row> keys = grouped.apply(Keys.create());
 assertTrue(keys.getSchema().equivalent(keySchema));
 // Make sure that the value has the right schema.
 PCollection<POJO> values = grouped.apply(Values.create()).apply(Flatten.iterables());
 assertTrue(values.getSchema().equivalent(POJO_SCHEMA));
 pipeline.run();
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

@Test
@Category({NeedsRunner.class, UsesTestStream.class})
public void testElementsAtAlmostPositiveInfinity() {
 Instant endOfGlobalWindow = GlobalWindow.INSTANCE.maxTimestamp();
 TestStream<String> stream =
   TestStream.create(StringUtf8Coder.of())
     .addElements(
       TimestampedValue.of("foo", endOfGlobalWindow),
       TimestampedValue.of("bar", endOfGlobalWindow))
     .advanceWatermarkToInfinity();
 FixedWindows windows = FixedWindows.of(Duration.standardHours(6));
 PCollection<String> windowedValues =
   p.apply(stream)
     .apply(Window.into(windows))
     .apply(WithKeys.of(1))
     .apply(GroupByKey.create())
     .apply(Values.create())
     .apply(Flatten.iterables());
 PAssert.that(windowedValues)
   .inWindow(windows.assignWindow(endOfGlobalWindow))
   .containsInAnyOrder("foo", "bar");
 p.run();
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

.apply(GroupByKey.create())
    .apply(Values.create())
    .apply(Flatten.iterables());
PCollection<Long> count =
  windowed.apply(Combine.globally(Count.<Integer>combineFn()).withoutDefaults());

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

.apply(GroupByKey.create())
.apply(Values.create())
.apply(Flatten.iterables());

内容来源于网络，如有侵权，请联系作者删除！

Flatten

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com