本文整理了Java中org.apache.tez.dag.api.Vertex.addDataSource()
方法的一些代码示例,展示了Vertex.addDataSource()
的具体用法。这些代码示例主要来源于Github
/Stackoverflow
/Maven
等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。Vertex.addDataSource()
方法的具体详情如下:
包路径:org.apache.tez.dag.api.Vertex
类名称:Vertex
方法名:addDataSource
[英]Specifies an external data source for a Vertex. This is meant to be used when a Vertex reads Input directly from an external source
For vertices which read data generated by another vertex - use the DAG method. If a vertex needs to use data generated by another vertex in the DAG and also from an external source, a combination of this API and the DAG.addEdge API can be used.
Note: If more than one RootInput exists on a vertex, which generates events which need to be routed, or generates information to set parallelism, a custom vertex manager should be setup to handle this. Not using a custom vertex manager for such a scenario will lead to a runtime failure.
[中]指定顶点的外部数据源。这意味着当顶点直接从外部源读取输入时使用
对于读取由另一个顶点生成的数据的顶点,请使用DAG方法。如果一个顶点需要使用DAG中另一个顶点生成的数据以及来自外部源的数据,则使用此API和DAG的组合。可以使用addEdge API。
注意:如果一个顶点上存在多个RootInput,这会生成需要路由的事件,或生成设置并行度的信息,则应设置自定义顶点管理器来处理此问题。在这种情况下不使用自定义顶点管理器将导致运行时失败。
代码示例来源:origin: apache/hive
conf.set(Utilities.INPUT_NAME, mapWork.getName());
LOG.info("Going through each work and adding MultiMRInput");
mergeVx.addDataSource(mapWork.getName(),
MultiMRInput.createConfigBuilder(conf, HiveInputFormat.class).build());
代码示例来源:origin: apache/drill
conf.set(Utilities.INPUT_NAME, mapWork.getName());
LOG.info("Going through each work and adding MultiMRInput");
mergeVx.addDataSource(mapWork.getName(),
MultiMRInput.createConfigBuilder(conf, HiveInputFormat.class).build());
代码示例来源:origin: apache/drill
map.addDataSource(alias, dataSource);
代码示例来源:origin: apache/hive
map.addDataSource(alias, dataSource);
map.addTaskLocalFiles(localResources);
return map;
代码示例来源:origin: org.apache.tez/tez-tests
TokenProcessor.class.getName()), numMaps).addDataSource("MRInput", dataSource);
TokenProcessor.class.getName()), numMaps).addDataSource("MRInput", dataSource);
TokenProcessor.class.getName()), numMaps).addDataSource("MRInput", dataSource);
代码示例来源:origin: org.apache.tez/tez-tests
v1.addDataSource(INPUT, dataSourceDescriptor);
Vertex v2 = Vertex.create(VERTEX2, ProcessorDescriptor.create(TokenProcessor.class.getName()));
v2.addDataSource(INPUT, dataSourceDescriptor);
代码示例来源:origin: org.apache.tez/tez-tests
MRHelpers.getResourceForMRMapper(mapStageConf));
mapVertex.addTaskLocalFiles(commonLocalResources)
.addDataSource("MRInput", dataSource)
.setTaskLaunchCmdOpts(MRHelpers.getJavaOptsForMRMapper(mapStageConf)).setTaskEnvironment(mapEnv);
vertices.add(mapVertex);
代码示例来源:origin: org.apache.tez/tez-examples
Vertex v1 = Vertex.create(VERTEX1, ProcessorDescriptor.create(TokenProcessor.class.getName()));
v1.addDataSource(INPUT,
MRInput.createConfigBuilder(new Configuration(tezConf), TextInputFormat.class, inputPath1)
.groupSplits(false).build());
Vertex v2 = Vertex.create(VERTEX2, ProcessorDescriptor.create(TokenProcessor.class.getName()));
v2.addDataSource(INPUT,
MRInput.createConfigBuilder(new Configuration(tezConf), TextInputFormat.class, inputPath2)
.groupSplits(false).build());
Vertex v3 = Vertex.create(VERTEX3, ProcessorDescriptor.create(TokenProcessor.class.getName()));
v3.addDataSource(INPUT,
MRInput.createConfigBuilder(new Configuration(tezConf), TextInputFormat.class, inputPath3)
.groupSplits(false).build());
代码示例来源:origin: cwensel/cascading
vertex.addDataSource( FlowElements.id( flowElement ), dataSourceDescriptor );
代码示例来源:origin: org.apache.tez/tez-examples
Vertex.create(hashSide,
ProcessorDescriptor.create(ForwardingProcessor.class.getName()))
.addDataSource(
inputFile,
MRInput
Vertex.create(streamingSide,
ProcessorDescriptor.create(ForwardingProcessor.class.getName()))
.addDataSource(
inputFile,
MRInput
代码示例来源:origin: org.apache.tez/tez-tests
.groupSplits(false).build();
stage1Vertex.addDataSource("MRInput", dsd);
代码示例来源:origin: cascading/cascading-hadoop2-tez
vertex.addDataSource( FlowElements.id( flowElement ), dataSourceDescriptor );
代码示例来源:origin: org.apache.tez/tez-examples
ForwardingProcessor.class.getName())).addDataSource("lhs",
MRInput
.createConfigBuilder(new Configuration(tezConf), TextInputFormat.class,
ForwardingProcessor.class.getName())).addDataSource("rhs",
MRInput
.createConfigBuilder(new Configuration(tezConf), TextInputFormat.class,
代码示例来源:origin: org.apache.tez/tez-examples
tokenizerVertex.addDataSource(INPUT, dataSource);
代码示例来源:origin: org.apache.tez/tez-examples
TokenProcessor.class.getName())).addDataSource(INPUT, dataSource);
代码示例来源:origin: org.apache.tez/tez-tests
.groupSplits(false).build();
stage1Vertex.addDataSource("MRInput", dsd);
代码示例来源:origin: org.apache.tez/tez-examples
Vertex.create("input1",
ProcessorDescriptor.create(ForwardingProcessor.class.getName()))
.addDataSource(
inputFile,
MRInput
Vertex.create("input2",
ProcessorDescriptor.create(ForwardingProcessor.class.getName()))
.addDataSource(
inputFile,
MRInput
代码示例来源:origin: org.apache.tez/tez-mapreduce
vertex.addDataSource("MRInput",
configureMRInputWithLegacySplitsGenerated(stageConf, true));
代码示例来源:origin: com.facebook.presto.hive/hive-apache
conf.set(Utilities.INPUT_NAME, mapWork.getName());
LOG.info("Going through each work and adding MultiMRInput");
mergeVx.addDataSource(mapWork.getName(),
MultiMRInput.createConfigBuilder(conf, HiveInputFormat.class).build());
代码示例来源:origin: com.facebook.presto.hive/hive-apache
map.addDataSource(alias, dataSource);
内容来源于网络,如有侵权,请联系作者删除!