com.alibaba.datax.common.util.Configuration.clone()方法的使用及代码示例

x33g5p2x  于2022-01-18 转载在 其他  
字(12.0k)|赞(0)|评价(0)|浏览(177)

本文整理了Java中com.alibaba.datax.common.util.Configuration.clone()方法的一些代码示例，展示了Configuration.clone()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Configuration.clone()方法的具体详情如下：
包路径：com.alibaba.datax.common.util.Configuration
类名称：Configuration
方法名：clone

Configuration.clone介绍

[英]拷贝当前Configuration，注意，这里使用了深拷贝，避免冲突
[中]拷贝当前配置注意，这里使用了深拷贝，避免冲突

代码示例

代码示例来源：origin: ECNU-1X/DataX-Masking

@Override
public List<Configuration> split(int mandatoryNumber) {
  List<Configuration> splitResultConfigs = new ArrayList<Configuration>();
  for (int j = 0; j < mandatoryNumber; j++) {
    splitResultConfigs.add(originConfig.clone());
  }
  return splitResultConfigs;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

@Override
public List<Configuration> split(int mandatoryNumber) {
  ArrayList<Configuration> configList = new ArrayList<Configuration>();
  for (int i = 0; i < mandatoryNumber; i++) {
    configList.add(this.configuration.clone());
  }
  return configList;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

@Override
public List<Configuration> split(int adviceNumber) {
  List<Configuration> configurations = new ArrayList<Configuration>();
  for (int i = 0; i < adviceNumber; i++) {
    configurations.add(this.originalConfig.clone());
  }
  return configurations;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

@Override
public List<Configuration> split(int mandatoryNumber) {
  List<Configuration> configList = new ArrayList<Configuration>();
  for(int i = 0; i < mandatoryNumber; i++) {
    configList.add(this.originalConfig.clone());
  }
  return configList;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

@Override
public List<Configuration> split(int adviceNumber) {
  List<Configuration> configurations = new ArrayList<Configuration>();
  for (int i = 0; i < adviceNumber; i++) {
    configurations.add(this.originalConfig.clone());
  }
  return configurations;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

@Override
public List<Configuration> split(int mandatoryNumber) {
  List<Configuration> splitResultConfigs = new ArrayList<Configuration>();
  for (int j = 0; j < mandatoryNumber; j++) {
    splitResultConfigs.add(originConfig.clone());
  }
  return splitResultConfigs;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

@Override
public List<Configuration> split(int mandatoryNumber) {
  if(Constant.LOADMODE.equalsIgnoreCase(this.writeMode)) {
    return this.odpsWriterJobProxy.split(mandatoryNumber);
  } else {
    List<Configuration> splitResult = new ArrayList<Configuration>();
    for(int i = 0; i < mandatoryNumber; i++) {
      splitResult.add(this.originalConfig.clone());
    }
    return splitResult;
  }
}

代码示例来源：origin: ECNU-1X/DataX-Masking

@Override
public List<Configuration> split(int mandatoryNumber) {
  List<Configuration> splitResultConfigs = new ArrayList<Configuration>();
  for (int j = 0; j < mandatoryNumber; j++) {
    splitResultConfigs.add(config.getOriginalConfig().clone());
  }
  return splitResultConfigs;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public static List<Configuration> split(Configuration writerSliceConfig,
    Set<String> originAllFileExists, int mandatoryNumber) {
  LOG.info("begin do split...");
  Set<String> allFileExists = new HashSet<String>();
  allFileExists.addAll(originAllFileExists);
  List<Configuration> writerSplitConfigs = new ArrayList<Configuration>();
  String filePrefix = writerSliceConfig.getString(Key.FILE_NAME);
  String fileSuffix;
  for (int i = 0; i < mandatoryNumber; i++) {
    // handle same file name
    Configuration splitedTaskConfig = writerSliceConfig.clone();
    String fullFileName = null;
    fileSuffix = UUID.randomUUID().toString().replace('-', '_');
    fullFileName = String.format("%s__%s", filePrefix, fileSuffix);
    while (allFileExists.contains(fullFileName)) {
      fileSuffix = UUID.randomUUID().toString().replace('-', '_');
      fullFileName = String.format("%s__%s", filePrefix, fileSuffix);
    }
    allFileExists.add(fullFileName);
    splitedTaskConfig.set(Key.FILE_NAME, fullFileName);
    LOG.info(String
        .format("splited write file name:[%s]", fullFileName));
    writerSplitConfigs.add(splitedTaskConfig);
  }
  LOG.info("end do split.");
  return writerSplitConfigs;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

@Override
public List<Configuration> split(int adviceNumber) {
  LOG.debug("split() begin...");
  List<Configuration> readerSplitConfigs = new ArrayList<Configuration>();
  // warn:每个slice拖且仅拖一个文件,
  // int splitNumber = adviceNumber;
  int splitNumber = this.sourceFiles.size();
  if (0 == splitNumber) {
    throw DataXException.asDataXException(FtpReaderErrorCode.EMPTY_DIR_EXCEPTION,
        String.format("未能找到待读取的文件,请确认您的配置项path: %s", this.originConfig.getString(Key.PATH)));
  }
  List<List<String>> splitedSourceFiles = this.splitSourceFiles(new ArrayList(this.sourceFiles), splitNumber);
  for (List<String> files : splitedSourceFiles) {
    Configuration splitedConfig = this.originConfig.clone();
    splitedConfig.set(Constant.SOURCE_FILES, files);
    readerSplitConfigs.add(splitedConfig);
  }
  LOG.debug("split() ok and end...");
  return readerSplitConfigs;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

@Override
public List<Configuration> split(int adviceNumber) {
  LOG.debug("split() begin...");
  List<Configuration> readerSplitConfigs = new ArrayList<Configuration>();
  // warn:每个slice拖且仅拖一个文件,
  // int splitNumber = adviceNumber;
  int splitNumber = this.sourceFiles.size();
  if (0 == splitNumber) {
    throw DataXException.asDataXException(
        TxtFileReaderErrorCode.EMPTY_DIR_EXCEPTION, String
            .format("未能找到待读取的文件,请确认您的配置项path: %s",
                this.originConfig.getString(Key.PATH)));
  }
  List<List<String>> splitedSourceFiles = this.splitSourceFiles(
      this.sourceFiles, splitNumber);
  for (List<String> files : splitedSourceFiles) {
    Configuration splitedConfig = this.originConfig.clone();
    splitedConfig.set(Constant.SOURCE_FILES, files);
    readerSplitConfigs.add(splitedConfig);
  }
  LOG.debug("split() ok and end...");
  return readerSplitConfigs;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

@Override
public List<Configuration> split(int adviceNumber) {
  LOG.info("split() begin...");
  List<Configuration> readerSplitConfigs = new ArrayList<Configuration>();
  // warn:每个slice拖且仅拖一个文件,
  // int splitNumber = adviceNumber;
  int splitNumber = this.sourceFiles.size();
  if (0 == splitNumber) {
    throw DataXException.asDataXException(HdfsReaderErrorCode.EMPTY_DIR_EXCEPTION,
        String.format("未能找到待读取的文件,请确认您的配置项path: %s", this.readerOriginConfig.getString(Key.PATH)));
  }
  List<List<String>> splitedSourceFiles = this.splitSourceFiles(new ArrayList<String>(this.sourceFiles), splitNumber);
  for (List<String> files : splitedSourceFiles) {
    Configuration splitedConfig = this.readerOriginConfig.clone();
    splitedConfig.set(Constant.SOURCE_FILES, files);
    readerSplitConfigs.add(splitedConfig);
  }
  return readerSplitConfigs;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

@Override
public List<Configuration> split(int adviceNumber) {
  LOG.debug("split() begin...");
  List<Configuration> readerSplitConfigs = new ArrayList<Configuration>();
  // 将每个单独的 object 作为一个 slice
  List<String> objects = parseOriginObjects(readerOriginConfig
      .getList(Constant.OBJECT, String.class));
  if (0 == objects.size()) {
    throw DataXException.asDataXException(
        OssReaderErrorCode.EMPTY_BUCKET_EXCEPTION,
        String.format(
            "未能找到待读取的Object,请确认您的配置项bucket: %s object: %s",
            this.readerOriginConfig.get(Key.BUCKET),
            this.readerOriginConfig.get(Key.OBJECT)));
  }
  for (String object : objects) {
    Configuration splitedConfig = this.readerOriginConfig.clone();
    splitedConfig.set(Constant.OBJECT, object);
    readerSplitConfigs.add(splitedConfig);
    LOG.info(String.format("OSS object to be read:%s", object));
  }
  LOG.debug("split() ok and end...");
  return readerSplitConfigs;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

private static List<Configuration> splitForNonPartitionedTable(Odps odps,
                                int adviceNum, Configuration sliceConfig) {
  List<Configuration> params = new ArrayList<Configuration>();
  String tunnelServer = sliceConfig.getString(Key.TUNNEL_SERVER);
  String tableName = sliceConfig.getString(Key.TABLE);
  String projectName = sliceConfig.getString(Key.PROJECT);
  DownloadSession session = OdpsUtil.createMasterSessionForNonPartitionedTable(odps,
      tunnelServer, projectName, tableName);
  String id = session.getId();
  long count = session.getRecordCount();
  List<Pair<Long, Long>> splitResult = splitRecordCount(count, adviceNum);
  for (Pair<Long, Long> pair : splitResult) {
    Configuration iParam = sliceConfig.clone();
    iParam.set(Constant.SESSION_ID, id);
    iParam.set(Constant.START_INDEX, pair.getLeft().longValue());
    iParam.set(Constant.STEP_COUNT, pair.getRight().longValue());
    params.add(iParam);
  }
  return params;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

Configuration tempConfig = this.originalConfig.clone();

代码示例来源：origin: ECNU-1X/DataX-Masking

public static String filterJobConfiguration(final Configuration configuration) {
  Configuration jobConfWithSetting = configuration.getConfiguration("job").clone();
  Configuration jobContent = jobConfWithSetting.getConfiguration("content");
  filterSensitiveConfiguration(jobContent);
  jobConfWithSetting.set("content",jobContent);
  return jobConfWithSetting.beautify();
}

代码示例来源：origin: ECNU-1X/DataX-Masking

public static List<Configuration> doSplit(
  Configuration originalSliceConfig, int adviceNumber, MongoClient mongoClient) {
  List<Configuration> confList = new ArrayList<Configuration>();
  String dbName = originalSliceConfig.getString(KeyConstant.MONGO_DB_NAME, originalSliceConfig.getString(KeyConstant.MONGO_DATABASE));
  String collName = originalSliceConfig.getString(KeyConstant.MONGO_COLLECTION_NAME);
  if(Strings.isNullOrEmpty(dbName) || Strings.isNullOrEmpty(collName) || mongoClient == null) {
    throw DataXException.asDataXException(MongoDBReaderErrorCode.ILLEGAL_VALUE,
      MongoDBReaderErrorCode.ILLEGAL_VALUE.getDescription());
  }
  boolean isObjectId = isPrimaryIdObjectId(mongoClient, dbName, collName);
  List<Range> rangeList = doSplitCollection(adviceNumber, mongoClient, dbName, collName, isObjectId);
  for(Range range : rangeList) {
    Configuration conf = originalSliceConfig.clone();
    conf.set(KeyConstant.LOWER_BOUND, range.lowerBound);
    conf.set(KeyConstant.UPPER_BOUND, range.upperBound);
    conf.set(KeyConstant.IS_OBJECTID, isObjectId);
    confList.add(conf);
  }
  return confList;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

private static List<Configuration> splitOnePartition(Odps odps,
                           String onePartition, int adviceNum, Configuration sliceConfig) {
  List<Configuration> params = new ArrayList<Configuration>();
  String tunnelServer = sliceConfig.getString(Key.TUNNEL_SERVER);
  String tableName = sliceConfig.getString(Key.TABLE);
  String  projectName = sliceConfig.getString(Key.PROJECT);
  DownloadSession session = OdpsUtil.createMasterSessionForPartitionedTable(odps,
      tunnelServer, projectName, tableName, onePartition);
  String id = session.getId();
  long count = session.getRecordCount();
  List<Pair<Long, Long>> splitResult = splitRecordCount(count, adviceNum);
  for (Pair<Long, Long> pair : splitResult) {
    Configuration iParam = sliceConfig.clone();
    iParam.set(Key.PARTITION, onePartition);
    iParam.set(Constant.SESSION_ID, id);
    iParam.set(Constant.START_INDEX, pair.getLeft().longValue());
    iParam.set(Constant.STEP_COUNT, pair.getRight().longValue());
    params.add(iParam);
  }
  return params;
}

代码示例来源：origin: ECNU-1X/DataX-Masking

splitResultConfigs.add(simplifiedConf.clone());
Configuration sliceConfig = simplifiedConf.clone();
  Configuration tempSlice = sliceConfig.clone();
  tempSlice.set(Key.TABLE, table);
  tempSlice.set(Key.PRE_SQL, renderPreOrPostSqls(preSqls, table));

代码示例来源：origin: ECNU-1X/DataX-Masking

public static Configuration generateConf(Configuration originalConfig, String odpsTableName, TableMeta tableMeta, TransferProjectConf transConf){
    Configuration newConfig = originalConfig.clone();
    newConfig.set(Key.ODPSTABLENAME, odpsTableName);
    newConfig.set(Key.ODPS_SERVER, transConf.getOdpsServer());
    newConfig.set(Key.TUNNEL_SERVER,transConf.getOdpsTunnel());
    newConfig.set(Key.ACCESS_ID,transConf.getAccessId());
    newConfig.set(Key.ACCESS_KEY,transConf.getAccessKey());
    newConfig.set(Key.PROJECT,transConf.getProject());
    newConfig.set(Key.TRUNCATE, true);
    newConfig.set(Key.PARTITION,null);
//        newConfig.remove(Key.PARTITION);
    List<FieldSchema> cols = tableMeta.getCols();
    List<String> allColumns = new ArrayList<String>();
    if(cols != null && !cols.isEmpty()){
      for(FieldSchema col:cols){
        allColumns.add(col.getName());
      }
    }
    newConfig.set(Key.COLUMN,allColumns);
    return newConfig;
  }

内容来源于网络，如有侵权，请联系作者删除！

Configuration

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com