cascading.util.Util类的使用及代码示例

x33g5p2x  于2022-02-01 转载在 其他  
字(10.3k)|赞(0)|评价(0)|浏览(185)

本文整理了Java中cascading.util.Util类的一些代码示例，展示了Util类的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Util类的具体详情如下：
包路径：cascading.util.Util
类名称：Util

Util介绍

[英]Class Util provides reusable operations.
[中]类Util提供可重用的操作。

代码示例

代码示例来源：origin: cwensel/cascading

  1. /**
  2.  * This method joins each value in the collection with the given delimiter. All results are appended to the
  3.  * given {@link StringBuffer} instance.
  4.  *
  5.  * @param buffer
  6.  * @param collection
  7.  * @param delim
  8.  */
  9. public static void join( StringBuffer buffer, Collection collection, String delim )
  10.  {
  11.  join( buffer, collection, delim, false );
  12.  }

代码示例来源：origin: cwensel/cascading

  1. @Override
  2. public String getID()
  3.  {
  4.  if( id == null )
  5.   id = Util.createUniqueID();
  6.  return id;
  7.  }

代码示例来源：origin: cwensel/cascading

  1. protected void sleepForPollingInterval()
  2.  {
  3.  Util.safeSleep( pollingInterval );
  4.  }

代码示例来源：origin: elastic/elasticsearch-hadoop

  1. cfg.set(TupleSerializationProps.SERIALIZATION_TOKENS, Util.join(",", Util.removeNulls(tokens, id + "=" + lmw)));
  2. LogFactory.getLog(EsTap.class).trace(String.format("Registered Cascading serialization token %s for %s", id, lmw));
  3. return;

代码示例来源：origin: cwensel/cascading

  1. public static Object invokeStaticMethod( String typeString, String methodName, Object[] parameters, Class[] parameterTypes )
  2.  {
  3.  Class type = loadClass( typeString );
  4.  return invokeStaticMethod( type, methodName, parameters, parameterTypes );
  5.  }

代码示例来源：origin: cwensel/cascading

  1. private void runComprehensiveCase( Boolean[] testCase, boolean useCollectionsComparator ) throws IOException
  2.  {
  3.  getPlatform().copyFromLocal( inputFileCrossNulls );
  4.  String test = Util.join( testCase, "_", true ) + "_" + useCollectionsComparator;
  5.  String path = "comprehensive/" + test;
  6.  Tap source = getPlatform().getTextFile( new Fields( "line" ), inputFileCrossNulls );
  7.  Tap sink = getPlatform().getDelimitedFile( new Fields( "num", "lower", "upper" ).applyTypes( Long.class, String.class, String.class ), " ", getOutputPath( path ), SinkMode.REPLACE );
  8.  sink.getScheme().setNumSinkParts( 1 );
  9.  Pipe pipe = new Pipe( "comprehensivesort" );
  10.  pipe = new Each( pipe, new Fields( "line" ), new RegexSplitter( new Fields( "num", "lower", "upper" ), "\\s" ) );
  11.  pipe = new Each( pipe, new Fields( "num" ), new Identity( Long.class ), Fields.REPLACE );
  12.  Fields groupFields = new Fields( "num" );
  13.  if( testCase[ 0 ] )
  14.   groupFields.setComparator( "num", useCollectionsComparator ? new NullSafeReverseComparator() : getPlatform().getLongComparator( true ) );
  15.  Fields sortFields = null;
  16.  if( testCase[ 1 ] != null )
  17.   {
  18.   sortFields = new Fields( "upper" );
  19.   if( testCase[ 1 ] )
  20.    sortFields.setComparator( "upper", useCollectionsComparator ? new NullSafeReverseComparator() : getPlatform().getStringComparator( true ) );
  21.   }
  22.  pipe = new GroupBy( pipe, groupFields, sortFields, testCase[ 2 ] );
  23.  Map<Object, Object> properties = getProperties();
  24.  if( getPlatform().isMapReduce() && getPlatform().getNumMapTasks( properties ) != null )
  25.   getPlatform().setNumMapTasks( properties, 13 );
  26.  Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );
  27.  flow.complete();
  28.  validateCase( test, testCase, sink );
  29.  }

代码示例来源：origin: cascading/cascading-hadoop2-common

  1. getPlatform().copyFromLocal( inputFileUpper );
  2. Tap sourceLower = new Hfs( new TextLine( new Fields( "offset", "line" ) ), inputFileLower );
  3. Tap sourceUpper = new Hfs( new TextLine( new Fields( "offset", "line" ) ), inputFileUpper );
  4. Pipe pipeLower = new Each( new Pipe( "lower" ), new Fields( "line" ), splitter );
  5. pipeLower = new GroupBy( pipeLower, new Fields( "num" ) );
  6. Pipe pipeUpper = new Each( new Pipe( "upper" ), new Fields( "line" ), splitter );
  7. pipeUpper = new GroupBy( pipeUpper, new Fields( "num" ) );
  8. flow.start();
  9. Util.safeSleep( 5000 );

代码示例来源：origin: cwensel/cascading

  1. Tap source = getPlatform().getDelimitedFile( new Fields( "number", "lower", "upper" ), " ", inputPath );
  2. Flow firstFlow = getPlatform().getFlowConnector().connect( "first", source, sinkTap, new Pipe( "copy" ) );
  3. Flow secondFlow = getPlatform().getFlowConnector().connect( "second", source, sinkTap, new Pipe( "copy" ) );
  4. Util.safeSleep( 1000 ); // be safe, delay execution
  5. Flow thirdFlow = getPlatform().getFlowConnector().connect( "third", source, sinkTap, new Pipe( "copy" ) );

代码示例来源：origin: cwensel/cascading

  1. public DelimitedPartition( Fields partitionFields, String delimiter, String postfix )
  2.  {
  3.  if( partitionFields == null )
  4.   throw new IllegalArgumentException( "partitionFields must not be null" );
  5.  if( !partitionFields.isDefined() )
  6.   throw new IllegalArgumentException( "partitionFields must be defined, got: " + partitionFields.printVerbose() );
  7.  this.partitionFields = partitionFields;
  8.  this.delimiter = delimiter == null ? PATH_DELIM : delimiter;
  9.  postfix = Util.isEmpty( postfix ) ? null : postfix.startsWith( this.delimiter ) ? postfix.substring( this.delimiter.length() ) : postfix;
  10.  this.numSplits = partitionFields.size() + ( postfix != null ? postfix.split( this.delimiter ).length : 0 );
  11.  this.postfix = postfix == null ? null : delimiter + postfix; // prefix the postfix w/ the delimiter
  12.  }

代码示例来源：origin: cwensel/cascading

  1. private static String emptyOrValue( Object value )
  2.  {
  3.  if( value == null )
  4.   return "";
  5.  if( Util.isEmpty( value.toString() ) )
  6.   return "";
  7.  return value.toString();
  8.  }

代码示例来源：origin: cwensel/cascading

  1. String tailName = tail.getName();
  2.   String tailName = pipe.getName();
  3.  throw new PlannerException( "not all tail pipes bound to sink taps, remaining tail pipe names: [" + join( quote( tailNames, "'" ), ", " ) + "], remaining sink tap names: [" + join( quote( remainingSinks, "'" ), ", " ) + "]" );
  4. remainingSinks.removeAll( asList( Pipe.names( flowTails ) ) );
  5.  throw new PlannerException( "not all sink taps bound to tail pipes, remaining sink tap names: [" + join( quote( remainingSinks, "'" ), ", " ) + "]" );
  6.  throw new PlannerException( "not all head pipes bound to source taps, remaining head pipe names: [" + join( quote( headNames, "'" ), ", " ) + "], remaining source tap names: [" + join( quote( remainingSources, "'" ), ", " ) + "]" );
  7.  throw new PlannerException( "not all source taps bound to head pipes, remaining source tap names: [" + join( quote( remainingSources, "'" ), ", " ) + "], remaining head pipe names: [" + join( quote( headNames, "'" ), ", " ) + "]" );

代码示例来源：origin: cwensel/cascading

  1. private String makeName( Pipe[] pipes )
  2.  {
  3.  String[] names = new String[ pipes.length ];
  4.  for( int i = 0; i < pipes.length; i++ )
  5.   names[ i ] = pipes[ i ].getName();
  6.  String name = Util.join( names, "+" );
  7.  if( name.length() > 32 )
  8.   name = name.substring( 0, 32 );
  9.  return name;
  10.  }
  11. }

代码示例来源：origin: cascading/cascading-platform

  1. public static Map<String, Pipe> buildOpPipes( String prefix, Pipe pipe, AssemblyFactory assemblyFactory, Fields[] args_fields, Fields[] decl_fields, Fields[] select_fields, String functionValue, String runOnly )
  2.  {
  3.  Map<String, Pipe> pipes = new LinkedHashMap<String, Pipe>();
  4.  for( int arg = 0; arg < args_fields.length; arg++ )
  5.   {
  6.   Fields argFields = args_fields[ arg ];
  7.   for( int decl = 0; decl < decl_fields.length; decl++ )
  8.    {
  9.    Fields declFields = decl_fields[ decl ];
  10.    for( int select = 0; select < select_fields.length; select++ )
  11.     {
  12.     Fields selectFields = select_fields[ select ];
  13.     String name;
  14.     if( prefix != null )
  15.      name = prefix + "." + Util.join( Fields.fields( argFields, declFields, selectFields ), "_" );
  16.     else
  17.      name = Util.join( Fields.fields( argFields, declFields, selectFields ), "_" );
  18.     if( runOnly != null && !runOnly.equalsIgnoreCase( name ) )
  19.      continue;
  20.     pipes.put( name, assemblyFactory.createAssembly( pipe, argFields, declFields, functionValue, selectFields ) );
  21.     }
  22.    }
  23.   }
  24.  return pipes;
  25.  }

代码示例来源：origin: cwensel/cascading

  1. public Integer getFirstOrdinal()
  2.  {
  3.  if( getOrdinals().isEmpty() )
  4.   return null;
  5.  return Util.getFirst( getOrdinals() );
  6.  }

代码示例来源：origin: cwensel/cascading

  1. @Override
  2. public String toString()
  3.  {
  4.  String string;
  5.  if( isOrdered() )
  6.   string = orderedToString();
  7.  else
  8.   string = unorderedToString();
  9.  if( types != null )
  10.   string += " | " + Util.join( Util.simpleTypeNames( types ), ", " );
  11.  return string;
  12.  }

代码示例来源：origin: cwensel/cascading

  1. private Tap decorateTap( Pipe pipe, Tap tempTap, String decoratorClassProp, String defaultDecoratorClassName )
  2.  {
  3.  String decoratorClassName = PropertyUtil.getProperty( defaultProperties, pipe, decoratorClassProp );
  4.  if( Util.isEmpty( decoratorClassName ) )
  5.   decoratorClassName = defaultDecoratorClassName;
  6.  if( Util.isEmpty( decoratorClassName ) )
  7.   return tempTap;
  8.  LOG.info( "found decorator property: {}, with value: {}, wrapping tap: {}", decoratorClassProp, decoratorClassName, tempTap );
  9.  tempTap = Util.newInstance( decoratorClassName, tempTap );
  10.  return tempTap;
  11.  }

代码示例来源：origin: cwensel/cascading

  1. private void verifyPipelines()
  2.  {
  3.  if( pipelineGraphs == null || pipelineGraphs.isEmpty() )
  4.   return;
  5.  Set<FlowElement> allElements = createIdentitySet( nodeSubGraph.vertexSet() );
  6.  for( ElementGraph pipelineGraph : pipelineGraphs )
  7.   allElements.removeAll( pipelineGraph.vertexSet() );
  8.  if( !allElements.isEmpty() )
  9.   throw new IllegalStateException( "union of pipeline graphs for flow node are missing elements: " + Util.join( allElements, ", " ) );
  10.  }

代码示例来源：origin: cwensel/cascading

  1. protected static Job getJob( RunningJob runningJob )
  2.  {
  3.  if( runningJob == null ) // if null, job hasn't been submitted
  4.   return null;
  5.  Job job = Util.returnInstanceFieldIfExistsSafe( runningJob, "job" );
  6.  if( job == null )
  7.   {
  8.   LOG.warn( "unable to get underlying org.apache.hadoop.mapreduce.Job from org.apache.hadoop.mapred.RunningJob, task level task counters will be unavailable" );
  9.   return null;
  10.   }
  11.  return job;
  12.  }

代码示例来源：origin: cwensel/cascading

  1. label = ( (Pipe) object ).print( scope ).replaceAll( "\"", "\'" ).replaceAll( "(\\)|\\])(\\[)", "$1|$2" ).replaceAll( "(^[^(\\[]+)(\\(|\\[)", "$1|$2" );
  2. label += "|{" + Util.join( annotations, "|" ) + "}";

代码示例来源：origin: cwensel/cascading

  1. public static String setLog4jLevel( String logger, String level )
  2.  {
  3.  // removing logj4 dependency
  4.  // org.apache.log4j.Logger.getLogger( logger[ 0 ] ).setLevel( org.apache.log4j.Level.toLevel( logger[ 1 ] ) );
  5.  Object loggerObject = Util.invokeStaticMethod( "org.apache.log4j.Logger", "getLogger",
  6.   new Object[]{logger}, new Class[]{String.class} );
  7.  Object levelObject = null;
  8.  if( level != null )
  9.   levelObject = Util.invokeStaticMethod( "org.apache.log4j.Level", "toLevel",
  10.    new Object[]{level}, new Class[]{String.class} );
  11.  Object oldLevel = Util.invokeInstanceMethod( loggerObject, "getLevel",
  12.   new Object[]{}, new Class[]{} );
  13.  Util.invokeInstanceMethod( loggerObject, "setLevel",
  14.   new Object[]{levelObject}, new Class[]{Util.loadClass( "org.apache.log4j.Level" )} );
  15.  if( oldLevel == null )
  16.   return null;
  17.  return oldLevel.toString();
  18.  }

内容来源于网络，如有侵权，请联系作者删除！

Util

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com