org.apache.tez.dag.api.Vertex.addDataSource()方法的使用及代码示例

x33g5p2x  于2022-02-01 转载在 其他  
字(5.3k)|赞(0)|评价(0)|浏览(97)

本文整理了Java中org.apache.tez.dag.api.Vertex.addDataSource()方法的一些代码示例,展示了Vertex.addDataSource()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。Vertex.addDataSource()方法的具体详情如下:
包路径:org.apache.tez.dag.api.Vertex
类名称:Vertex
方法名:addDataSource

Vertex.addDataSource介绍

[英]Specifies an external data source for a Vertex. This is meant to be used when a Vertex reads Input directly from an external source

For vertices which read data generated by another vertex - use the DAG method. If a vertex needs to use data generated by another vertex in the DAG and also from an external source, a combination of this API and the DAG.addEdge API can be used.

Note: If more than one RootInput exists on a vertex, which generates events which need to be routed, or generates information to set parallelism, a custom vertex manager should be setup to handle this. Not using a custom vertex manager for such a scenario will lead to a runtime failure.
[中]指定顶点的外部数据源。这意味着当顶点直接从外部源读取输入时使用
对于读取由另一个顶点生成的数据的顶点,请使用DAG方法。如果一个顶点需要使用DAG中另一个顶点生成的数据以及来自外部源的数据,则使用此API和DAG的组合。可以使用addEdge API。
注意:如果一个顶点上存在多个RootInput,这会生成需要路由的事件,或生成设置并行度的信息,则应设置自定义顶点管理器来处理此问题。在这种情况下不使用自定义顶点管理器将导致运行时失败。

代码示例

代码示例来源:origin: apache/hive

conf.set(Utilities.INPUT_NAME, mapWork.getName());
LOG.info("Going through each work and adding MultiMRInput");
mergeVx.addDataSource(mapWork.getName(),
  MultiMRInput.createConfigBuilder(conf, HiveInputFormat.class).build());

代码示例来源:origin: apache/drill

conf.set(Utilities.INPUT_NAME, mapWork.getName());
LOG.info("Going through each work and adding MultiMRInput");
mergeVx.addDataSource(mapWork.getName(),
  MultiMRInput.createConfigBuilder(conf, HiveInputFormat.class).build());

代码示例来源:origin: apache/drill

map.addDataSource(alias, dataSource);

代码示例来源:origin: apache/hive

map.addDataSource(alias, dataSource);
map.addTaskLocalFiles(localResources);
return map;

代码示例来源:origin: org.apache.tez/tez-tests

TokenProcessor.class.getName()), numMaps).addDataSource("MRInput", dataSource);
TokenProcessor.class.getName()), numMaps).addDataSource("MRInput", dataSource);
TokenProcessor.class.getName()), numMaps).addDataSource("MRInput", dataSource);

代码示例来源:origin: org.apache.tez/tez-tests

v1.addDataSource(INPUT, dataSourceDescriptor);
Vertex v2 = Vertex.create(VERTEX2, ProcessorDescriptor.create(TokenProcessor.class.getName()));
v2.addDataSource(INPUT, dataSourceDescriptor);

代码示例来源:origin: org.apache.tez/tez-tests

MRHelpers.getResourceForMRMapper(mapStageConf));
mapVertex.addTaskLocalFiles(commonLocalResources)
  .addDataSource("MRInput", dataSource)
  .setTaskLaunchCmdOpts(MRHelpers.getJavaOptsForMRMapper(mapStageConf)).setTaskEnvironment(mapEnv);
vertices.add(mapVertex);

代码示例来源:origin: org.apache.tez/tez-examples

Vertex v1 = Vertex.create(VERTEX1, ProcessorDescriptor.create(TokenProcessor.class.getName()));
v1.addDataSource(INPUT,
 MRInput.createConfigBuilder(new Configuration(tezConf), TextInputFormat.class, inputPath1)
     .groupSplits(false).build());
Vertex v2 = Vertex.create(VERTEX2, ProcessorDescriptor.create(TokenProcessor.class.getName()));
v2.addDataSource(INPUT,
 MRInput.createConfigBuilder(new Configuration(tezConf), TextInputFormat.class, inputPath2)
     .groupSplits(false).build());
Vertex v3 = Vertex.create(VERTEX3, ProcessorDescriptor.create(TokenProcessor.class.getName()));
v3.addDataSource(INPUT,
 MRInput.createConfigBuilder(new Configuration(tezConf), TextInputFormat.class, inputPath3)
  .groupSplits(false).build());

代码示例来源:origin: cwensel/cascading

vertex.addDataSource( FlowElements.id( flowElement ), dataSourceDescriptor );

代码示例来源:origin: org.apache.tez/tez-examples

Vertex.create(hashSide,
  ProcessorDescriptor.create(ForwardingProcessor.class.getName()))
  .addDataSource(
    inputFile,
    MRInput
Vertex.create(streamingSide,
  ProcessorDescriptor.create(ForwardingProcessor.class.getName()))
  .addDataSource(
    inputFile,
    MRInput

代码示例来源:origin: org.apache.tez/tez-tests

.groupSplits(false).build();
stage1Vertex.addDataSource("MRInput", dsd);

代码示例来源:origin: cascading/cascading-hadoop2-tez

vertex.addDataSource( FlowElements.id( flowElement ), dataSourceDescriptor );

代码示例来源:origin: org.apache.tez/tez-examples

ForwardingProcessor.class.getName())).addDataSource("lhs",
MRInput
  .createConfigBuilder(new Configuration(tezConf), TextInputFormat.class,
ForwardingProcessor.class.getName())).addDataSource("rhs",
MRInput
  .createConfigBuilder(new Configuration(tezConf), TextInputFormat.class,

代码示例来源:origin: org.apache.tez/tez-examples

tokenizerVertex.addDataSource(INPUT, dataSource);

代码示例来源:origin: org.apache.tez/tez-examples

TokenProcessor.class.getName())).addDataSource(INPUT, dataSource);

代码示例来源:origin: org.apache.tez/tez-tests

.groupSplits(false).build();
stage1Vertex.addDataSource("MRInput", dsd);

代码示例来源:origin: org.apache.tez/tez-examples

Vertex.create("input1",
  ProcessorDescriptor.create(ForwardingProcessor.class.getName()))
  .addDataSource(
    inputFile,
    MRInput
Vertex.create("input2",
  ProcessorDescriptor.create(ForwardingProcessor.class.getName()))
  .addDataSource(
    inputFile,
    MRInput

代码示例来源:origin: org.apache.tez/tez-mapreduce

vertex.addDataSource("MRInput",
  configureMRInputWithLegacySplitsGenerated(stageConf, true));

代码示例来源:origin: com.facebook.presto.hive/hive-apache

conf.set(Utilities.INPUT_NAME, mapWork.getName());
LOG.info("Going through each work and adding MultiMRInput");
mergeVx.addDataSource(mapWork.getName(),
  MultiMRInput.createConfigBuilder(conf, HiveInputFormat.class).build());

代码示例来源:origin: com.facebook.presto.hive/hive-apache

map.addDataSource(alias, dataSource);

相关文章