com.alibaba.datax.common.util.Configuration.getListConfiguration()方法的使用及代码示例

x33g5p2x  于2022-01-18 转载在 其他  
字(12.4k)|赞(0)|评价(0)|浏览(194)

本文整理了Java中com.alibaba.datax.common.util.Configuration.getListConfiguration()方法的一些代码示例，展示了Configuration.getListConfiguration()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Configuration.getListConfiguration()方法的具体详情如下：
包路径：com.alibaba.datax.common.util.Configuration
类名称：Configuration
方法名：getListConfiguration

Configuration.getListConfiguration介绍

[英]根据用户提供的json path，寻址包含Configuration的List，如果对象不存在，返回默认null
[中]根据用户提供的json路径寻址包含配置的列表如果对象不存在，返回默认无效的

代码示例

代码示例来源：origin: ECNU-1X/DataX-Masking

private int calculateTaskCount(List<Configuration> configurations) {
  int totalTasks = 0;
  for (Configuration taskGroupConfiguration : configurations) {
    totalTasks += taskGroupConfiguration.getListConfiguration(
        CoreConstant.DATAX_JOB_CONTENT).size();
  }
  return totalTasks;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

List<Configuration> contentConfig = jobConfiguration.getListConfiguration(CoreConstant.DATAX_JOB_CONTENT);

代码示例来源：origin: ECNU-1X/DataX-Masking

public HbaseAbstractTask(com.alibaba.datax.common.util.Configuration configuration) {
  this.htable = Hbase094xHelper.getTable(configuration);
  this.columns = configuration.getListConfiguration(Key.COLUMN);
  this.rowkeyColumn = configuration.getListConfiguration(Key.ROWKEY_COLUMN);
  this.versionColumn = configuration.getConfiguration(Key.VERSION_COLUMN);
  this.encoding = configuration.getString(Key.ENCODING,Constant.DEFAULT_ENCODING);
  this.nullMode = NullModeType.getByTypeName(configuration.getString(Key.NULL_MODE,Constant.DEFAULT_NULL_MODE));
  this.walFlag = configuration.getBool(Key.WAL_FLAG, false);
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public HbaseAbstractTask(com.alibaba.datax.common.util.Configuration configuration) {
  //this.htable = Hbase11xHelper.getTable(configuration);
  this.bufferedMutator = Hbase11xHelper.getBufferedMutator(configuration);
  this.columns = configuration.getListConfiguration(Key.COLUMN);
  this.rowkeyColumn = configuration.getListConfiguration(Key.ROWKEY_COLUMN);
  this.versionColumn = configuration.getConfiguration(Key.VERSION_COLUMN);
  this.encoding = configuration.getString(Key.ENCODING,Constant.DEFAULT_ENCODING);
  this.nullMode = NullModeType.getByTypeName(configuration.getString(Key.NULL_MODE,Constant.DEFAULT_NULL_MODE));
  this.walFlag = configuration.getBool(Key.WAL_FLAG, false);
}

代码示例来源：origin: ECNU-1X/DataX-Masking

/**
 * 执行reader和writer最细粒度的切分，需要注意的是，writer的切分结果要参照reader的切分结果，
 * 达到切分后数目相等，才能满足1：1的通道模型，所以这里可以将reader和writer的配置整合到一起，
 * 然后，为避免顺序给读写端带来长尾影响，将整合的结果shuffler掉
 */
private int split() {
  this.adjustChannelNumber();
  if (this.needChannelNumber <= 0) {
    this.needChannelNumber = 1;
  }
  List<Configuration> readerTaskConfigs = this
      .doReaderSplit(this.needChannelNumber);
  int taskNumber = readerTaskConfigs.size();
  List<Configuration> writerTaskConfigs = this
      .doWriterSplit(taskNumber);
  List<Configuration> transformerList = this.configuration.getListConfiguration(CoreConstant.DATAX_JOB_CONTENT_TRANSFORMER);
  LOG.debug("transformer configuration: "+ JSON.toJSONString(transformerList));
  /**
   * 输入是reader和writer的parameter list，输出是content下面元素的list
   */
  List<Configuration> contentConfig = mergeReaderAndWriterTaskConfigs(
      readerTaskConfigs, writerTaskConfigs, transformerList);
  LOG.debug("contentConfig configuration: "+ JSON.toJSONString(contentConfig));
  this.configuration.set(CoreConstant.DATAX_JOB_CONTENT, contentConfig);
  return contentConfig.size();
}

代码示例来源：origin: ECNU-1X/DataX-Masking

.getListConfiguration(com.alibaba.datax.plugin.unstructuredstorage.reader.Key.COLUMN);
if (null != column
    && 1 == column.size()
      .getListConfiguration(com.alibaba.datax.plugin.unstructuredstorage.reader.Key.COLUMN);

代码示例来源：origin: ECNU-1X/DataX-Masking

public static void validateColumn(com.alibaba.datax.common.util.Configuration originalConfig){
  List<Configuration> columns = originalConfig.getListConfiguration(Key.COLUMN);
  if (columns == null || columns.isEmpty()) {
    throw DataXException.asDataXException(Hbase094xWriterErrorCode.REQUIRED_VALUE, "column为必填项，其形式为：column:[{\"index\": 0,\"name\": \"cf0:column0\",\"type\": \"string\"},{\"index\": 1,\"name\": \"cf1:column1\",\"type\": \"long\"}]");
  }
  for (Configuration aColumn : columns) {
    Integer index = aColumn.getInt(Key.INDEX);
    String type = aColumn.getNecessaryValue(Key.TYPE, Hbase094xWriterErrorCode.REQUIRED_VALUE);
    String name = aColumn.getNecessaryValue(Key.NAME, Hbase094xWriterErrorCode.REQUIRED_VALUE);
    ColumnType.getByTypeName(type);
    if(name.split(":").length != 2){
      throw DataXException.asDataXException(Hbase094xWriterErrorCode.ILLEGAL_VALUE, String.format("您column配置项中name配置的列格式[%s]不正确，name应该配置为 列族:列名  的形式, 如 {\"index\": 1,\"name\": \"cf1:q1\",\"type\": \"long\"}", name));
    }
    if(index == null || index < 0){
      throw DataXException.asDataXException(Hbase094xWriterErrorCode.ILLEGAL_VALUE, "您的column配置项不正确,配置项中中index为必填项,且为非负数，请检查并修改.");
    }
  }
}

代码示例来源：origin: ECNU-1X/DataX-Masking

private static void validateColumn(com.alibaba.datax.common.util.Configuration originalConfig){
  List<Configuration> columns = originalConfig.getListConfiguration(Key.COLUMN);
  if (columns == null || columns.isEmpty()) {
    throw DataXException.asDataXException(Hbase11xWriterErrorCode.REQUIRED_VALUE, "column为必填项，其形式为：column:[{\"index\": 0,\"name\": \"cf0:column0\",\"type\": \"string\"},{\"index\": 1,\"name\": \"cf1:column1\",\"type\": \"long\"}]");
  }
  for (Configuration aColumn : columns) {
    Integer index = aColumn.getInt(Key.INDEX);
    String type = aColumn.getNecessaryValue(Key.TYPE,Hbase11xWriterErrorCode.REQUIRED_VALUE);
    String name = aColumn.getNecessaryValue(Key.NAME,Hbase11xWriterErrorCode.REQUIRED_VALUE);
    ColumnType.getByTypeName(type);
    if(name.split(":").length != 2){
      throw DataXException.asDataXException(Hbase11xWriterErrorCode.ILLEGAL_VALUE, String.format("您column配置项中name配置的列格式[%s]不正确，name应该配置为 列族:列名  的形式, 如 {\"index\": 1,\"name\": \"cf1:q1\",\"type\": \"long\"}", name));
    }
    if(index == null || index < 0){
      throw DataXException.asDataXException(Hbase11xWriterErrorCode.ILLEGAL_VALUE, "您的column配置项不正确,配置项中中index为必填项,且为非负数，请检查并修改.");
    }
  }
}

代码示例来源：origin: ECNU-1X/DataX-Masking

"每个channel的平均task数[averTaskPerChannel]，channel数目[channelNumber]，每个taskGroup的平均channel数[channelsPerTaskGroup]都应该为正数");
List<Configuration> taskConfigs = this.configuration
    .getListConfiguration(CoreConstant.DATAX_JOB_CONTENT);
int taskGroupNumber = channelNumber / channelsPerTaskGroup;
int leftChannelNumber = channelNumber % channelsPerTaskGroup;
      .getListConfiguration(CoreConstant.DATAX_JOB_CONTENT));
  taskGroupConfig.set(CoreConstant.DATAX_CORE_CONTAINER_TASKGROUP_CHANNEL,
      channelNumber);
  Configuration taskGroupConfig = this.configuration.clone();
  List<Configuration> taskGroupJobContent = taskGroupConfig
      .getListConfiguration(CoreConstant.DATAX_JOB_CONTENT);
  taskGroupJobContent.clear();
  taskGroupConfig.set(CoreConstant.DATAX_JOB_CONTENT, taskGroupJobContent);
    for (int i = 0; i < averTaskPerChannel; i++) {
      List<Configuration> taskGroupJobContent = taskGroupConfig
          .getListConfiguration(CoreConstant.DATAX_JOB_CONTENT);
      taskGroupJobContent.add(taskConfigs.get(taskConfigIndex++));
      taskGroupConfig.set(CoreConstant.DATAX_JOB_CONTENT,
    Configuration taskGroupConfig = taskGroupConfigs.get(taskGroupConfigIndex);
    List<Configuration> taskGroupJobContent = taskGroupConfig
        .getListConfiguration(CoreConstant.DATAX_JOB_CONTENT);
    taskGroupJobContent.add(taskConfigs.get(taskConfigIndex++));
    taskGroupConfig.set(

代码示例来源：origin: ECNU-1X/DataX-Masking

Validate.isTrue(configuration != null, "框架获得的 Job 不能为 null.");
List<Configuration> contentConfig = configuration.getListConfiguration(CoreConstant.DATAX_JOB_CONTENT);
Validate.isTrue(contentConfig.size() > 0, "框架获得的切分后的 Job 无内容.");

代码示例来源：origin: ECNU-1X/DataX-Masking

.getListConfiguration(com.alibaba.datax.plugin.unstructuredstorage.reader.Key.COLUMN);
if (null == columns || columns.size() == 0) {
  throw DataXException.asDataXException(UnstructuredStorageReaderErrorCode.REQUIRED_VALUE, "您需要指定 columns");

代码示例来源：origin: ECNU-1X/DataX-Masking

private static void validateRowkeyColumn(com.alibaba.datax.common.util.Configuration originalConfig){
  List<Configuration> rowkeyColumn = originalConfig.getListConfiguration(Key.ROWKEY_COLUMN);
  if (rowkeyColumn == null || rowkeyColumn.isEmpty()) {
    throw DataXException.asDataXException(Hbase11xWriterErrorCode.REQUIRED_VALUE, "rowkeyColumn为必填项，其形式为：rowkeyColumn:[{\"index\": 0,\"type\": \"string\"},{\"index\": -1,\"type\": \"string\",\"value\": \"_\"}]");
  }
  int rowkeyColumnSize = rowkeyColumn.size();
  //包含{"index":0,"type":"string"} 或者 {"index":-1,"type":"string","value":"_"}
  for (Configuration aRowkeyColumn : rowkeyColumn) {
    Integer index = aRowkeyColumn.getInt(Key.INDEX);
    String type = aRowkeyColumn.getNecessaryValue(Key.TYPE,Hbase11xWriterErrorCode.REQUIRED_VALUE);
    ColumnType.getByTypeName(type);
    if(index == null ){
      throw DataXException.asDataXException(Hbase11xWriterErrorCode.REQUIRED_VALUE, "rowkeyColumn配置项中index为必填项");
    }
    //不能只有-1列,即rowkey连接串
    if(rowkeyColumnSize ==1 && index == -1){
      throw DataXException.asDataXException(Hbase11xWriterErrorCode.ILLEGAL_VALUE, "rowkeyColumn配置项不能全为常量列,至少指定一个rowkey列");
    }
    if(index == -1){
      aRowkeyColumn.getNecessaryValue(Key.VALUE,Hbase11xWriterErrorCode.REQUIRED_VALUE);
    }
  }
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public static void validateRowkeyColumn(com.alibaba.datax.common.util.Configuration originalConfig){
  List<Configuration> rowkeyColumn = originalConfig.getListConfiguration(Key.ROWKEY_COLUMN);
  if (rowkeyColumn == null || rowkeyColumn.isEmpty()) {
    throw DataXException.asDataXException(Hbase094xWriterErrorCode.REQUIRED_VALUE, "rowkeyColumn为必填项，其形式为：rowkeyColumn:[{\"index\": 0,\"type\": \"string\"},{\"index\": -1,\"type\": \"string\",\"value\": \"_\"}]");
  }
  int rowkeyColumnSize = rowkeyColumn.size();
  //包含{"index":0,"type":"string"} 或者 {"index":-1,"type":"string","value":"_"}
  for (Configuration aRowkeyColumn : rowkeyColumn) {
    Integer index = aRowkeyColumn.getInt(Key.INDEX);
    String type = aRowkeyColumn.getNecessaryValue(Key.TYPE, Hbase094xWriterErrorCode.REQUIRED_VALUE);
    ColumnType.getByTypeName(type);
    if(index == null ){
      throw DataXException.asDataXException(Hbase094xWriterErrorCode.REQUIRED_VALUE, "rowkeyColumn配置项中index为必填项");
    }
    //不能只有-1列,即rowkey连接串
    if(rowkeyColumnSize ==1 && index == -1){
      throw DataXException.asDataXException(Hbase094xWriterErrorCode.ILLEGAL_VALUE, "rowkeyColumn配置项不能全为常量列,至少指定一个rowkey列");
    }
    if(index == -1){
      aRowkeyColumn.getNecessaryValue(Key.VALUE, Hbase094xWriterErrorCode.REQUIRED_VALUE);
    }
  }
}

代码示例来源：origin: ECNU-1X/DataX-Masking

this.columns = this.writerSliceConfig.getListConfiguration(Key.COLUMN);
if (null == columns || columns.size() == 0) {
  throw DataXException.asDataXException(HdfsWriterErrorCode.REQUIRED_VALUE, "您需要指定 columns");

代码示例来源：origin: ECNU-1X/DataX-Masking

TaskPluginCollector taskPluginCollector){
char fieldDelimiter = config.getChar(Key.FIELD_DELIMITER);
List<Configuration>  columns = config.getListConfiguration(Key.COLUMN);
String compress = config.getString(Key.COMPRESS,null);

代码示例来源：origin: ECNU-1X/DataX-Masking

.getListConfiguration(Constant.PARSED_COLUMNS);
List<Pair<String, ColumnType>> parsedColumns = new ArrayList<Pair<String, ColumnType>>();
for (int i = 0; i < parsedColumnsTmp.size(); i++) {

代码示例来源：origin: ECNU-1X/DataX-Masking

.getListConfiguration(com.alibaba.datax.plugin.unstructuredstorage.reader.Key.COLUMN);

代码示例来源：origin: ECNU-1X/DataX-Masking

.getListConfiguration(com.alibaba.datax.plugin.unstructuredstorage.reader.Key.COLUMN);
if (null != column
    && 1 == column.size()
      .getListConfiguration(com.alibaba.datax.plugin.unstructuredstorage.reader.Key.COLUMN);

代码示例来源：origin: ECNU-1X/DataX-Masking

public static List<TransformerExecution> buildTransformerInfo(Configuration taskConfig) {
  List<Configuration> tfConfigs = taskConfig.getListConfiguration(CoreConstant.JOB_TRANSFORMER);
  if (tfConfigs == null || tfConfigs.size() == 0) {
    return null;

代码示例来源：origin: ECNU-1X/DataX-Masking

List<Configuration>  columns = config.getListConfiguration(Key.COLUMN);
String compress = config.getString(Key.COMPRESS, null);
List<String> columnNames = getColumnNames(columns);

内容来源于网络，如有侵权，请联系作者删除！

Configuration

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com