cascading.tap.Tap.getNodeConfigDef()方法的使用及代码示例

x33g5p2x  于2022-01-30 转载在 其他  
字(7.2k)|赞(0)|评价(0)|浏览(127)

本文整理了Java中cascading.tap.Tap.getNodeConfigDef()方法的一些代码示例,展示了Tap.getNodeConfigDef()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。Tap.getNodeConfigDef()方法的具体详情如下:
包路径:cascading.tap.Tap
类名称:Tap
方法名:getNodeConfigDef

Tap.getNodeConfigDef介绍

[英]Returns a ConfigDef instance that allows for process level properties to be set and made available via a resulting cascading.flow.FlowProcess instance when the tap is invoked.

Any properties set on the nodeConfigDef will not show up in any Flow configuration, but will show up in the current process cascading.flow.FlowNode (in Apache Tez the Vertex configuration). Any value set in the nodeConfigDef will be overridden by the pipe local #getConfigDef instance.

Use this method to tweak properties in the process node this tap instance is planned into.
[中]返回一个ConfigDef实例,该实例允许设置进程级属性,并通过结果级联使其可用。流调用tap时的FlowProcess实例。
在nodeConfigDef上设置的任何属性都不会显示在任何流配置中,但会显示在当前进程级联中。流FlowNode(在Apache Tez中是顶点配置)。nodeConfigDef中设置的任何值都将被管道本地#getConfigDef实例覆盖。
使用此方法可以调整该tap实例计划进入的流程节点中的属性。

代码示例

代码示例来源:origin: cwensel/cascading

@Override
public ConfigDef getNodeConfigDef()
 {
 return original.getNodeConfigDef();
 }

代码示例来源:origin: cwensel/cascading

@Override
public ConfigDef getNodeConfigDef()
 {
 return original.getNodeConfigDef();
 }

代码示例来源:origin: cwensel/cascading

@Test
public void testTapSourceConfigDef() throws IOException
 {
 getPlatform().copyFromLocal( inputFileNums20 );
 Scheme scheme = getPlatform().getTestConfigDefScheme();
 Tap source = getPlatform().getTap( scheme, inputFileNums20, SinkMode.KEEP );
 // process -> after sink/sourceConfInit are called
 // default -> Wrapper for all cluster side calls
 source.getConfigDef().setProperty( Mode.DEFAULT, "default", "source-default" );
 // steps on above value
 source.getStepConfigDef().setProperty( Mode.DEFAULT, "default", "process-default" );
 source.getConfigDef().setProperty( Mode.DEFAULT, "replace", "source-default" );
 source.getConfigDef().setProperty( Mode.REPLACE, "replace", "source-replace" );
 source.getNodeConfigDef().setProperty( Mode.REPLACE, "default-node", "node-replace" );
 source.getStepConfigDef().setProperty( Mode.DEFAULT, "replace", "process-default" );
 source.getStepConfigDef().setProperty( Mode.REPLACE, "replace", "process-replace" );
 source.getStepConfigDef().setProperty( Mode.DEFAULT, "default-node", "process-default" );
 Pipe pipe = new Pipe( "test" );
 pipe = new Each( pipe, new Insert( new Fields( "value" ), "nada" ), Fields.ALL );
 Tap sink = getPlatform().getTextFile( getOutputPath( "tapsourceconfigdef" ), SinkMode.REPLACE );
 Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );
 flow.complete();
 assertTrue( flow.resourceExists( sink ) );
 }

代码示例来源:origin: cascading/cascading-platform

@Test
public void testTapSourceConfigDef() throws IOException
 {
 getPlatform().copyFromLocal( inputFileNums20 );
 Scheme scheme = getPlatform().getTestConfigDefScheme();
 Tap source = getPlatform().getTap( scheme, inputFileNums20, SinkMode.KEEP );
 // process -> after sink/sourceConfInit are called
 // default -> Wrapper for all cluster side calls
 source.getConfigDef().setProperty( Mode.DEFAULT, "default", "source-default" );
 // steps on above value
 source.getStepConfigDef().setProperty( Mode.DEFAULT, "default", "process-default" );
 source.getConfigDef().setProperty( Mode.DEFAULT, "replace", "source-default" );
 source.getConfigDef().setProperty( Mode.REPLACE, "replace", "source-replace" );
 source.getNodeConfigDef().setProperty( Mode.REPLACE, "default-node", "node-replace" );
 source.getStepConfigDef().setProperty( Mode.DEFAULT, "replace", "process-default" );
 source.getStepConfigDef().setProperty( Mode.REPLACE, "replace", "process-replace" );
 source.getStepConfigDef().setProperty( Mode.DEFAULT, "default-node", "process-default" );
 Pipe pipe = new Pipe( "test" );
 pipe = new Each( pipe, new Insert( new Fields( "value" ), "nada" ), Fields.ALL );
 Tap sink = getPlatform().getTextFile( getOutputPath( "tapsourceconfigdef" ), SinkMode.REPLACE );
 Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );
 flow.complete();
 assertTrue( flow.resourceExists( sink ) );
 }

代码示例来源:origin: cascading/cascading-platform

@Test
public void testTapSinkConfigDef() throws IOException
 {
 getPlatform().copyFromLocal( inputFileNums20 );
 Tap source = getPlatform().getTextFile( new Fields( "line" ), inputFileNums20, SinkMode.KEEP );
 Pipe pipe = new Pipe( "test" );
 pipe = new Each( pipe, new Insert( new Fields( "value" ), "nada" ), Fields.ALL );
 Scheme scheme = getPlatform().getTestConfigDefScheme();
 Tap sink = getPlatform().getTap( scheme, getOutputPath( "tapsinkconfigdef" ), SinkMode.REPLACE );
 // process -> after sink/sourceConfInit are called
 // default -> Wrapper for all cluster side calls
 sink.getConfigDef().setProperty( Mode.DEFAULT, "default", "sink-default" );
 // steps on above value
 sink.getStepConfigDef().setProperty( Mode.DEFAULT, "default", "process-default" );
 sink.getConfigDef().setProperty( Mode.DEFAULT, "replace", "sink-default" );
 sink.getConfigDef().setProperty( Mode.REPLACE, "replace", "sink-replace" );
 sink.getNodeConfigDef().setProperty( Mode.REPLACE, "default-node", "node-replace" );
 sink.getStepConfigDef().setProperty( Mode.DEFAULT, "replace", "process-default" );
 sink.getStepConfigDef().setProperty( Mode.REPLACE, "replace", "process-replace" );
 sink.getStepConfigDef().setProperty( Mode.DEFAULT, "default-node", "process-default" );
 Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );
 flow.complete();
 assertTrue( flow.resourceExists( sink ) );
 }

代码示例来源:origin: cwensel/cascading

@Test
public void testTapSinkConfigDef() throws IOException
 {
 getPlatform().copyFromLocal( inputFileNums20 );
 Tap source = getPlatform().getTextFile( new Fields( "line" ), inputFileNums20, SinkMode.KEEP );
 Pipe pipe = new Pipe( "test" );
 pipe = new Each( pipe, new Insert( new Fields( "value" ), "nada" ), Fields.ALL );
 Scheme scheme = getPlatform().getTestConfigDefScheme();
 Tap sink = getPlatform().getTap( scheme, getOutputPath( "tapsinkconfigdef" ), SinkMode.REPLACE );
 // process -> after sink/sourceConfInit are called
 // default -> Wrapper for all cluster side calls
 sink.getConfigDef().setProperty( Mode.DEFAULT, "default", "sink-default" );
 // steps on above value
 sink.getStepConfigDef().setProperty( Mode.DEFAULT, "default", "process-default" );
 sink.getConfigDef().setProperty( Mode.DEFAULT, "replace", "sink-default" );
 sink.getConfigDef().setProperty( Mode.REPLACE, "replace", "sink-replace" );
 sink.getNodeConfigDef().setProperty( Mode.REPLACE, "default-node", "node-replace" );
 sink.getStepConfigDef().setProperty( Mode.DEFAULT, "replace", "process-default" );
 sink.getStepConfigDef().setProperty( Mode.REPLACE, "replace", "process-replace" );
 sink.getStepConfigDef().setProperty( Mode.DEFAULT, "default-node", "process-default" );
 Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );
 flow.complete();
 assertTrue( flow.resourceExists( sink ) );
 }

相关文章