本文整理了Java中org.apache.beam.sdk.transforms.Flatten.iterables()
方法的一些代码示例,展示了Flatten.iterables()
的具体用法。这些代码示例主要来源于Github
/Stackoverflow
/Maven
等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。Flatten.iterables()
方法的具体详情如下:
包路径:org.apache.beam.sdk.transforms.Flatten
类名称:Flatten
方法名:iterables
[英]Returns a PTransform that takes a PCollection> and returns a PCollection containing all the elements from all the Iterables.
Example of use:
PCollection> pcOfIterables = ...;
By default, the output PCollection encodes its elements using the same Coderthat the input uses for the elements in its Iterable.
[中]返回一个PTransform,该PTransform接受PCollection>并返回一个PCollection,该PCollection包含所有ITerable中的所有元素。
使用示例:
PCollection> pcOfIterables = ...;
默认情况下,输出PCollection使用与输入用于Iterable中元素相同的代码对其元素进行编码。
代码示例来源:origin: org.apache.beam/beam-sdks-java-core
@Override
public PCollection<T> expand(PCollection<T> in) {
return in.apply(Combine.globally(new SampleAnyCombineFn<T>(limit)).withoutDefaults())
.apply(Flatten.iterables());
}
代码示例来源:origin: org.apache.beam/beam-runners-core-construction-java
@Test
public void emptyFlattenWithNonFlatten() {
AppliedPTransform application =
AppliedPTransform
.<PCollection<Iterable<Integer>>, PCollection<Integer>, Flatten.Iterables<Integer>>of(
"EmptyFlatten",
Collections.emptyMap(),
Collections.singletonMap(
new TupleTag<Integer>(),
PCollection.createPrimitiveOutputInternal(
p, WindowingStrategy.globalDefault(), IsBounded.BOUNDED, VarIntCoder.of())),
/* This isn't actually possible to construct, but for the sake of example */
Flatten.iterables(),
p);
assertThat(PTransformMatchers.emptyFlatten().matches(application), is(false));
}
代码示例来源:origin: org.apache.beam/beam-runners-core-construction-java
@Test
public void flattenWithDuplicateInputsNonFlatten() {
AppliedPTransform application =
AppliedPTransform
.<PCollection<Iterable<Integer>>, PCollection<Integer>, Flatten.Iterables<Integer>>of(
"EmptyFlatten",
Collections.emptyMap(),
Collections.singletonMap(
new TupleTag<Integer>(),
PCollection.createPrimitiveOutputInternal(
p, WindowingStrategy.globalDefault(), IsBounded.BOUNDED, VarIntCoder.of())),
/* This isn't actually possible to construct, but for the sake of example */
Flatten.iterables(),
p);
assertThat(PTransformMatchers.flattenWithDuplicateInputs().matches(application), is(false));
}
代码示例来源:origin: org.apache.beam/beam-sdks-java-core
@Test
public void testFlattenGetName() {
Assert.assertEquals("Flatten.Iterables", Flatten.<String>iterables().getName());
Assert.assertEquals("Flatten.PCollections", Flatten.<String>pCollections().getName());
}
代码示例来源:origin: org.apache.beam/beam-sdks-java-core
@Override
public PCollection<T> expand(PCollection<T> input) {
return input
.apply(WithKeys.of(""))
.apply(GroupByKey.create())
.apply(Values.create())
.apply(Flatten.iterables());
}
}
代码示例来源:origin: org.apache.beam/beam-sdks-java-core
@Override
public PCollectionView<Integer> expand(PCollection<String> input) {
return input
.apply(
ParDo.of(
new DoFn<String, Integer>() {
@ProcessElement
public void toInteger(ProcessContext ctxt) {
ctxt.output(Integer.valueOf(ctxt.element()));
}
}))
.apply(Top.largest(1))
.apply(Flatten.iterables())
.apply(View.asSingleton());
}
}
代码示例来源:origin: org.apache.beam/beam-sdks-java-extensions-sql
.apply("flatten", Flatten.iterables())
.setSchema(
CalciteUtils.toSchema(getRowType()),
代码示例来源:origin: org.apache.beam/beam-sdks-java-core
p.apply("ReadMyFile", TextIO.read().from(inputFile.getPath()))
.apply(sample)
.apply(Flatten.iterables())
.apply("WriteMyFile", TextIO.write().to(outputFile.getPath()));
代码示例来源:origin: org.apache.beam/beam-sdks-java-core
@Override
public PCollectionView<ActualT> expand(PBegin input) {
final Coder<T> coder = actual.getCoder();
return actual
.apply("FilterActuals", rewindowActuals.prepareActuals())
.apply("GatherPanes", GatherAllPanes.globally())
.apply("ExtractPane", MapElements.via(extractPane))
.setCoder(IterableCoder.of(actual.getCoder()))
.apply(Flatten.iterables())
.apply("RewindowActuals", rewindowActuals.windowActuals())
.apply(
ParDo.of(
new DoFn<T, T>() {
@ProcessElement
public void processElement(ProcessContext context) throws CoderException {
context.output(CoderUtils.clone(coder, context.element()));
}
}))
.apply(actualView);
}
}
代码示例来源:origin: org.apache.beam/beam-sdks-java-core
@Test
@Category(ValidatesRunner.class)
public void testFlattenIterablesEmpty() {
PCollection<Iterable<String>> input =
p.apply(
Create.<Iterable<String>>of(NO_LINES)
.withCoder(IterableCoder.of(StringUtf8Coder.of())));
PCollection<String> output = input.apply(Flatten.iterables());
PAssert.that(output).containsInAnyOrder(NO_LINES_ARRAY);
p.run();
}
代码示例来源:origin: org.apache.beam/beam-sdks-java-core
@Test
@Category(ValidatesRunner.class)
public void testFlattenIterables() {
PCollection<Iterable<String>> input =
p.apply(
Create.<Iterable<String>>of(LINES).withCoder(IterableCoder.of(StringUtf8Coder.of())));
PCollection<String> output = input.apply(Flatten.iterables());
PAssert.that(output).containsInAnyOrder(LINES_ARRAY);
p.run();
}
代码示例来源:origin: org.apache.beam/beam-sdks-java-core
@Test
@Category(ValidatesRunner.class)
public void testFlattenIterablesLists() {
PCollection<List<String>> input =
p.apply(Create.<List<String>>of(LINES).withCoder(ListCoder.of(StringUtf8Coder.of())));
PCollection<String> output = input.apply(Flatten.iterables());
PAssert.that(output).containsInAnyOrder(LINES_ARRAY);
p.run();
}
代码示例来源:origin: org.apache.beam/beam-sdks-java-core
@Test
@Category(ValidatesRunner.class)
public void testFlattenIterablesSets() {
Set<String> linesSet = ImmutableSet.copyOf(LINES);
PCollection<Set<String>> input =
p.apply(Create.<Set<String>>of(linesSet).withCoder(SetCoder.of(StringUtf8Coder.of())));
PCollection<String> output = input.apply(Flatten.iterables());
PAssert.that(output).containsInAnyOrder(LINES_ARRAY);
p.run();
}
代码示例来源:origin: org.apache.beam/beam-sdks-java-core
@Test
@Category(ValidatesRunner.class)
public void testFlattenIterablesCollections() {
Set<String> linesSet = ImmutableSet.copyOf(LINES);
PCollection<Collection<String>> input =
p.apply(
Create.<Collection<String>>of(linesSet)
.withCoder(CollectionCoder.of(StringUtf8Coder.of())));
PCollection<String> output = input.apply(Flatten.iterables());
PAssert.that(output).containsInAnyOrder(LINES_ARRAY);
p.run();
}
代码示例来源:origin: org.apache.beam/beam-sdks-java-core
@Test
@Category(NeedsRunner.class)
public void testOutputCoders() {
Schema keySchema = Schema.builder().addStringField("field1").build();
PCollection<KV<Row, Iterable<POJO>>> grouped =
pipeline
.apply(Create.of(new POJO("key1", 1, "value1")))
.apply(Group.byFieldNames("field1"));
// Make sure that the key has the right schema.
PCollection<Row> keys = grouped.apply(Keys.create());
assertTrue(keys.getSchema().equivalent(keySchema));
// Make sure that the value has the right schema.
PCollection<POJO> values = grouped.apply(Values.create()).apply(Flatten.iterables());
assertTrue(values.getSchema().equivalent(POJO_SCHEMA));
pipeline.run();
}
代码示例来源:origin: org.apache.beam/beam-sdks-java-core
@Test
@Category({NeedsRunner.class, UsesTestStream.class})
public void testElementsAtAlmostPositiveInfinity() {
Instant endOfGlobalWindow = GlobalWindow.INSTANCE.maxTimestamp();
TestStream<String> stream =
TestStream.create(StringUtf8Coder.of())
.addElements(
TimestampedValue.of("foo", endOfGlobalWindow),
TimestampedValue.of("bar", endOfGlobalWindow))
.advanceWatermarkToInfinity();
FixedWindows windows = FixedWindows.of(Duration.standardHours(6));
PCollection<String> windowedValues =
p.apply(stream)
.apply(Window.into(windows))
.apply(WithKeys.of(1))
.apply(GroupByKey.create())
.apply(Values.create())
.apply(Flatten.iterables());
PAssert.that(windowedValues)
.inWindow(windows.assignWindow(endOfGlobalWindow))
.containsInAnyOrder("foo", "bar");
p.run();
}
代码示例来源:origin: org.apache.beam/beam-sdks-java-core
.apply(GroupByKey.create())
.apply(Values.create())
.apply(Flatten.iterables());
PCollection<Long> count =
windowed.apply(Combine.globally(Count.<Integer>combineFn()).withoutDefaults());
代码示例来源:origin: org.apache.beam/beam-sdks-java-core
.apply(GroupByKey.create())
.apply(Values.create())
.apply(Flatten.iterables());
内容来源于网络,如有侵权,请联系作者删除!