org.apache.tez.dag.api.Vertex.create()方法的使用及代码示例

x33g5p2x  于2022-02-01 转载在 其他  
字(13.8k)|赞(0)|评价(0)|浏览(91)

本文整理了Java中org.apache.tez.dag.api.Vertex.create()方法的一些代码示例，展示了Vertex.create()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Vertex.create()方法的具体详情如下：
包路径：org.apache.tez.dag.api.Vertex
类名称：Vertex
方法名：create

Vertex.create介绍

[英]Create a new vertex with the given name.
The vertex task resource will be picked from configuration
The vertex parallelism will be inferred. If it cannot be inferred then an error will be reported. This constructor may be used for vertices that have data sources, or connected via 1-1 edges or have runtime parallelism estimation via data source initializers or vertex managers. Calling this constructor is equivalent to calling Vertex#Vertex(String,ProcessorDescriptor,int) with the parallelism set to -1.
[中]使用给定名称创建一个新顶点。
顶点任务资源将从配置中选取
将推断顶点平行度。如果无法推断，则会报告错误。此构造函数可用于具有数据源的顶点，或通过1-1边连接的顶点，或通过数据源初始值设定项或顶点管理器进行运行时并行度估计的顶点。调用此构造函数相当于调用Vertex#Vertex（String，ProcessorDescriptor，int），并行度设置为-1。

代码示例

代码示例来源：origin: apache/hive

@Override
 public Vertex answer(InvocationOnMock invocation) throws Throwable {
  Object[] args = invocation.getArguments();
  return Vertex.create(((BaseWork)args[1]).getName(),
    mock(ProcessorDescriptor.class), 0, mock(Resource.class));
 }
});

代码示例来源：origin: apache/drill

private Vertex createVertex(JobConf conf, ReduceWork reduceWork,
  LocalResource appJarLr, List<LocalResource> additionalLr, FileSystem fs,
  Path mrScratchDir, Context ctx) throws Exception {
 // set up operator plan
 conf.set(Utilities.INPUT_NAME, reduceWork.getName());
 Utilities.setReduceWork(conf, reduceWork, mrScratchDir, false);
 // create the directories FileSinkOperators need
 Utilities.createTmpDirs(conf, reduceWork);
 VertexExecutionContext vertexExecutionContext = createVertexExecutionContext(reduceWork);
 // create the vertex
 Vertex reducer = Vertex.create(reduceWork.getName(),
   ProcessorDescriptor.create(ReduceTezProcessor.class.getName()).
     setUserPayload(TezUtils.createUserPayloadFromConf(conf)),
   reduceWork.isAutoReduceParallelism() ?
     reduceWork.getMaxReduceTasks() :
     reduceWork.getNumReduceTasks(), getContainerResource(conf));
 reducer.setTaskEnvironment(getContainerEnvironment(conf, false));
 reducer.setExecutionContext(vertexExecutionContext);
 reducer.setTaskLaunchCmdOpts(getContainerJavaOpts(conf));
 Map<String, LocalResource> localResources = new HashMap<String, LocalResource>();
 localResources.put(getBaseName(appJarLr), appJarLr);
 for (LocalResource lr: additionalLr) {
  localResources.put(getBaseName(lr), lr);
 }
 reducer.addTaskLocalFiles(localResources);
 return reducer;
}

代码示例来源：origin: apache/drill

map = Vertex.create(mapWork.getName(), ProcessorDescriptor.create(procClassName)
  .setUserPayload(serializedConf), numTasks, getContainerResource(conf));

代码示例来源：origin: apache/hive

private Vertex createVertex(JobConf conf, ReduceWork reduceWork, FileSystem fs,
  Path mrScratchDir, Context ctx, Map<String, LocalResource> localResources)
    throws Exception {
 // set up operator plan
 conf.set(Utilities.INPUT_NAME, reduceWork.getName());
 Utilities.setReduceWork(conf, reduceWork, mrScratchDir, false);
 // create the directories FileSinkOperators need
 Utilities.createTmpDirs(conf, reduceWork);
 VertexExecutionContext vertexExecutionContext = createVertexExecutionContext(reduceWork);
 // create the vertex
 Vertex reducer = Vertex.create(reduceWork.getName(),
   ProcessorDescriptor.create(ReduceTezProcessor.class.getName()).
     setUserPayload(TezUtils.createUserPayloadFromConf(conf)),
   reduceWork.isAutoReduceParallelism() ?
     reduceWork.getMaxReduceTasks() :
     reduceWork.getNumReduceTasks(), getContainerResource(conf));
 reducer.setTaskEnvironment(getContainerEnvironment(conf, false));
 reducer.setExecutionContext(vertexExecutionContext);
 reducer.setTaskLaunchCmdOpts(getContainerJavaOpts(conf));
 reducer.addTaskLocalFiles(localResources);
 return reducer;
}

代码示例来源：origin: apache/hive

map = Vertex.create(mapWork.getName(), ProcessorDescriptor.create(procClassName)
  .setUserPayload(serializedConf), numTasks, getContainerResource(conf));

代码示例来源：origin: cwensel/cascading

private Vertex newVertex( FlowNode flowNode, Configuration conf, int parallelism )
 {
 conf.set( FlowNode.CASCADING_FLOW_NODE, pack( flowNode, conf ) ); // todo: pack into payload directly
 ProcessorDescriptor descriptor = ProcessorDescriptor.create( FlowProcessor.class.getName() );
 descriptor.setUserPayload( getPayload( conf ) );
 Vertex vertex = Vertex.create( flowNode.getID(), descriptor, parallelism );
 if( environment != null )
  vertex.setTaskEnvironment( environment );
 return vertex;
 }

代码示例来源：origin: cascading/cascading-hadoop2-tez

private Vertex newVertex( FlowNode flowNode, Configuration conf, int parallelism )
 {
 conf.set( FlowNode.CASCADING_FLOW_NODE, pack( flowNode, conf ) ); // todo: pack into payload directly
 ProcessorDescriptor descriptor = ProcessorDescriptor.create( FlowProcessor.class.getName() );
 descriptor.setUserPayload( getPayload( conf ) );
 Vertex vertex = Vertex.create( flowNode.getID(), descriptor, parallelism );
 if( environment != null )
  vertex.setTaskEnvironment( environment );
 return vertex;
 }

代码示例来源：origin: org.apache.tez/tez-tests

private DAG createDAG(TezConfiguration conf, int numTasks, int maxSleepTimeMillis,
           int payloadSize, String mode) throws IOException {
 Map<String, LocalResource> localResourceMap = new HashMap<String, LocalResource>();
 UserPayload payload =
   createUserPayload(conf, maxSleepTimeMillis, payloadSize, mode, localResourceMap);
 Vertex vertex = Vertex.create("RPCLoadVertex",
   ProcessorDescriptor.create(RPCSleepProcessor.class.getName()).setUserPayload(
     payload), numTasks).addTaskLocalFiles(localResourceMap);
 return DAG.create("RPCLoadGen").addVertex(vertex);
}

代码示例来源：origin: org.apache.tez/tez-tests

private DAG createDAG(int numGenTasks, int totalSourceDataSize, int numFetcherTasks) {
 int bytesPerSource = totalSourceDataSize / numGenTasks;
 LOG.info("DataPerSourceTask(bytes)=" + bytesPerSource);
 ByteBuffer payload = ByteBuffer.allocate(4);
 payload.putInt(0, bytesPerSource);
 Vertex broadcastVertex = Vertex.create("DataGen",
   ProcessorDescriptor.create(InputGenProcessor.class.getName())
     .setUserPayload(UserPayload.create(payload)), numGenTasks);
 Vertex fetchVertex = Vertex.create("FetchVertex",
   ProcessorDescriptor.create(InputFetchProcessor.class.getName()), numFetcherTasks);
 UnorderedKVEdgeConfig edgeConf = UnorderedKVEdgeConfig.newBuilder(NullWritable.class
 .getName(), IntWritable.class.getName()).setCompression(false, null, null).build();
 DAG dag = DAG.create("BroadcastLoadGen");
 dag.addVertex(broadcastVertex).addVertex(fetchVertex).addEdge(
   Edge.create(broadcastVertex, fetchVertex, edgeConf.createDefaultBroadcastEdgeProperty()));
 return dag;
}

代码示例来源：origin: org.apache.tez/tez-examples

private DAG createDag(TezConfiguration tezConf, Path largeOutPath, Path smallOutPath,
  Path expectedOutputPath, int numTasks, long largeOutSize, long smallOutSize)
  throws IOException {
 long largeOutSizePerTask = largeOutSize / numTasks;
 long smallOutSizePerTask = smallOutSize / numTasks;
 DAG dag = DAG.create("JoinDataGen");
 Vertex genDataVertex = Vertex.create("datagen", ProcessorDescriptor.create(
   GenDataProcessor.class.getName()).setUserPayload(
   UserPayload.create(ByteBuffer.wrap(GenDataProcessor.createConfiguration(largeOutSizePerTask,
     smallOutSizePerTask)))), numTasks);
 genDataVertex.addDataSink(STREAM_OUTPUT_NAME, 
   MROutput.createConfigBuilder(new Configuration(tezConf),
     TextOutputFormat.class, largeOutPath.toUri().toString()).build());
 genDataVertex.addDataSink(HASH_OUTPUT_NAME, 
   MROutput.createConfigBuilder(new Configuration(tezConf),
     TextOutputFormat.class, smallOutPath.toUri().toString()).build());
 genDataVertex.addDataSink(EXPECTED_OUTPUT_NAME, 
   MROutput.createConfigBuilder(new Configuration(tezConf),
     TextOutputFormat.class, expectedOutputPath.toUri().toString()).build());
 dag.addVertex(genDataVertex);
 return dag;
}

代码示例来源：origin: org.apache.tez/tez-tests

DataSourceDescriptor dataSource = configurer.generateSplitsInAM(false).build();
Vertex mapVertex1 = Vertex.create("map1", ProcessorDescriptor.create(
  TokenProcessor.class.getName()), numMaps).addDataSource("MRInput", dataSource);
Vertex mapVertex2 = Vertex.create("map2", ProcessorDescriptor.create(
  TokenProcessor.class.getName()), numMaps).addDataSource("MRInput", dataSource);
Vertex mapVertex3 = Vertex.create("map3", ProcessorDescriptor.create(
  TokenProcessor.class.getName()), numMaps).addDataSource("MRInput", dataSource);
Vertex checkerVertex = Vertex.create("checker", ProcessorDescriptor.create(
  UnionProcessor.class.getName()), 1);

代码示例来源：origin: org.apache.tez/tez-tests

DataSourceDescriptor.create(inputDescriptor, inputInitializerDescriptor, null);
Vertex v1 = Vertex.create(VERTEX1, ProcessorDescriptor.create(TokenProcessor.class.getName()));
v1.addDataSource(INPUT, dataSourceDescriptor);
Vertex v2 = Vertex.create(VERTEX2, ProcessorDescriptor.create(TokenProcessor.class.getName()));
v2.addDataSource(INPUT, dataSourceDescriptor);
UserPayload userPayload = cartesianProductConfig.toUserPayload(tezConf);
Vertex v3 = Vertex.create(VERTEX3, ProcessorDescriptor.create(JoinProcessor.class.getName()));
v3.addDataSink(OUTPUT, dataSinkDescriptor);
v3.setVertexManagerPlugin(

代码示例来源：origin: org.apache.tez/tez-tests

Vertex broadcastVertex = Vertex.create("Broadcast", ProcessorDescriptor.create(
  InputProcessor.class.getName()), numBroadcastTasks);
Vertex inputVertex = Vertex.create("Input", ProcessorDescriptor.create(
  InputProcessor.class.getName()).setUserPayload(procPayload), numOneToOneTasks);
Vertex oneToOneVertex = Vertex.create("OneToOne",
  ProcessorDescriptor.create(
    OneToOneProcessor.class.getName()).setUserPayload(procPayload));

代码示例来源：origin: org.apache.tez/tez-examples

TextOutputFormat.class, outputPath).build();
Vertex tokenizerVertex = Vertex.create(TOKENIZER, ProcessorDescriptor.create(
  TokenProcessor.class.getName()));
tokenizerVertex.addDataSource(INPUT, dataSource);
Vertex summationVertex = Vertex.create(SUMMATION, ProcessorDescriptor.create(
  SumProcessor.class.getName()), numPartitions);
Vertex sorterVertex = Vertex.create(SORTER, ProcessorDescriptor.create(
  NoOpSorter.class.getName()), 1);
sorterVertex.addDataSink(OUTPUT, dataSink);

代码示例来源：origin: org.apache.tez/tez-examples

private DAG createDAG(TezConfiguration tezConf, String inputPath1, String inputPath2,
           String inputPath3, String outputPath, boolean isPartitioned)
 throws IOException {
 Vertex v1 = Vertex.create(VERTEX1, ProcessorDescriptor.create(TokenProcessor.class.getName()));
 Vertex v2 = Vertex.create(VERTEX2, ProcessorDescriptor.create(TokenProcessor.class.getName()));
 v2.addDataSource(INPUT,
  MRInput.createConfigBuilder(new Configuration(tezConf), TextInputFormat.class, inputPath2)
      .groupSplits(false).build());
 Vertex v3 = Vertex.create(VERTEX3, ProcessorDescriptor.create(TokenProcessor.class.getName()));
 v3.addDataSource(INPUT,
  MRInput.createConfigBuilder(new Configuration(tezConf), TextInputFormat.class, inputPath3)
 Vertex v4 = Vertex.create(VERTEX4, ProcessorDescriptor.create(JoinProcessor.class.getName()));
 v4.addDataSink(OUTPUT,
  MROutput.createConfigBuilder(new Configuration(tezConf), TextOutputFormat.class, outputPath)

代码示例来源：origin: org.apache.tez/tez-examples

Vertex tokenizerVertex = Vertex.create(TOKENIZER, ProcessorDescriptor.create(
  TokenProcessor.class.getName())).addDataSource(INPUT, dataSource);
Vertex summationVertex = Vertex.create(SUMMATION,
  ProcessorDescriptor.create(SumProcessor.class.getName()), numPartitions)
  .addDataSink(OUTPUT, dataSink);

代码示例来源：origin: com.facebook.presto.hive/hive-apache

private Vertex createVertex(JobConf conf, ReduceWork reduceWork,
  LocalResource appJarLr, List<LocalResource> additionalLr, FileSystem fs,
  Path mrScratchDir, Context ctx) throws Exception {
 // set up operator plan
 conf.set(Utilities.INPUT_NAME, reduceWork.getName());
 Utilities.setReduceWork(conf, reduceWork, mrScratchDir, false);
 // create the directories FileSinkOperators need
 Utilities.createTmpDirs(conf, reduceWork);
 // create the vertex
 Vertex reducer = Vertex.create(reduceWork.getName(),
   ProcessorDescriptor.create(ReduceTezProcessor.class.getName()).
   setUserPayload(TezUtils.createUserPayloadFromConf(conf)),
     reduceWork.isAutoReduceParallelism() ? reduceWork.getMaxReduceTasks() : reduceWork
       .getNumReduceTasks(), getContainerResource(conf));
 reducer.setTaskEnvironment(getContainerEnvironment(conf, false));
 reducer.setTaskLaunchCmdOpts(getContainerJavaOpts(conf));
 Map<String, LocalResource> localResources = new HashMap<String, LocalResource>();
 localResources.put(getBaseName(appJarLr), appJarLr);
 for (LocalResource lr: additionalLr) {
  localResources.put(getBaseName(lr), lr);
 }
 reducer.addTaskLocalFiles(localResources);
 return reducer;
}

代码示例来源：origin: org.apache.tez/tez-examples

.build();
Vertex lhsVertex = Vertex.create(LHS_INPUT_NAME, ProcessorDescriptor.create(
  ForwardingProcessor.class.getName())).addDataSource("lhs",
  MRInput
setVertexExecutionContext(lhsVertex, getLhsExecutionContext());
Vertex rhsVertex = Vertex.create(RHS_INPUT_NAME, ProcessorDescriptor.create(
  ForwardingProcessor.class.getName())).addDataSource("rhs",
  MRInput
setVertexExecutionContext(rhsVertex, getRhsExecutionContext());
Vertex joinValidateVertex = Vertex.create("joinvalidate", ProcessorDescriptor.create(
  JoinValidateProcessor.class.getName()), numPartitions);
setVertexExecutionContext(joinValidateVertex, getValidateExecutionContext());

代码示例来源：origin: org.apache.tez/tez-tests

DAG dag = DAG.create("multipleCommitsDAG");
dag.setConf(TezConfiguration.TEZ_AM_COMMIT_ALL_OUTPUTS_ON_DAG_SUCCESS, !commitOnVertexSuccess + "");
Vertex v1 = Vertex.create("v1", ProcessorDescriptor.create(MultipleOutputProcessor.class.getName())
  .setUserPayload(
    new MultipleOutputProcessor.MultipleOutputProcessorConfig(
      V1OutputNamePrefix, v1OutputNum, UV12OutputNamePrefix, uv12OutputNum)
     .toUserPayload()), 2);
Vertex v2 = Vertex.create("v2", ProcessorDescriptor.create(MultipleOutputProcessor.class.getName())
  .setUserPayload(
    new MultipleOutputProcessor.MultipleOutputProcessorConfig(
Vertex v3 = Vertex.create("v3", ProcessorDescriptor.create(MultipleOutputProcessor.class.getName())
  .setUserPayload(
    new MultipleOutputProcessor.MultipleOutputProcessorConfig(V3OutputNamePrefix, v3OutputNum)

代码示例来源：origin: org.apache.tez/tez-mapreduce

Vertex vertex = Vertex.create(vertexName,
  ProcessorDescriptor.create(processorName).setUserPayload(vertexUserPayload),
  numTasks, taskResource);

内容来源于网络，如有侵权，请联系作者删除！

vertex

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com