com.alibaba.datax.common.util.Configuration.get()方法的使用及代码示例

x33g5p2x  于2022-01-18 转载在 其他  
字(11.4k)|赞(0)|评价(0)|浏览(129)

本文整理了Java中com.alibaba.datax.common.util.Configuration.get()方法的一些代码示例,展示了Configuration.get()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。Configuration.get()方法的具体详情如下:
包路径:com.alibaba.datax.common.util.Configuration
类名称:Configuration
方法名:get

Configuration.get介绍

[英]根据用户提供的json path,寻址具体的对象。

NOTE: 目前仅支持Map以及List下标寻址, 例如:

对于如下JSON

{"a": {"b": {"c": [0,1,2,3]}}}

config.get("") 返回整个Map
config.get("a") 返回a下属整个Map
config.get("a.b.c") 返回c对应的数组List
config.get("a.b.c[0]") 返回数字0
[中]根据用户提供的json路径寻址具体的对象。
注:目前仅支持地图以及列表下标寻址, 例如:
对于如下JSON
{“a”:{“b”:{“c”:[0,1,2,3]}}
配置。得到(“”)返回整个地图
配置。得到(“a”)返回A.下属整个地图
配置。获取(“a.b.c”)返回C对应的数组列表
配置。获取(“a.b.c[0]”)返回数字0

代码示例

代码示例来源:origin: ECNU-1X/DataX-Masking

/**
 * 根据用户提供的json path,寻址Map对象,如果对象不存在,返回null
 */
@SuppressWarnings("unchecked")
public Map<String, Object> getMap(final String path) {
  Map<String, Object> result = this.get(path, Map.class);
  if (null == result) {
    return null;
  }
  return result;
}

代码示例来源:origin: ECNU-1X/DataX-Masking

@Override
public void init() {
  this.readerSliceConfig = super.getPluginJobConf();
  this.userName = readerSliceConfig.getString(KeyConstant.MONGO_USER_NAME, readerSliceConfig.getString(KeyConstant.MONGO_USERNAME));
  this.password = readerSliceConfig.getString(KeyConstant.MONGO_USER_PASSWORD, readerSliceConfig.getString(KeyConstant.MONGO_PASSWORD));
  this.database = readerSliceConfig.getString(KeyConstant.MONGO_DB_NAME, readerSliceConfig.getString(KeyConstant.MONGO_DATABASE));
  this.authDb = readerSliceConfig.getString(KeyConstant.MONGO_AUTHDB, this.database);
  if(!Strings.isNullOrEmpty(userName) && !Strings.isNullOrEmpty(password)) {
    mongoClient = MongoUtil.initCredentialMongoClient(readerSliceConfig,userName,password,authDb);
  } else {
    mongoClient = MongoUtil.initMongoClient(readerSliceConfig);
  }
  this.collection = readerSliceConfig.getString(KeyConstant.MONGO_COLLECTION_NAME);
  this.query = readerSliceConfig.getString(KeyConstant.MONGO_QUERY);
  this.mongodbColumnMeta = JSON.parseArray(readerSliceConfig.getString(KeyConstant.MONGO_COLUMN));
  this.lowerBound = readerSliceConfig.get(KeyConstant.LOWER_BOUND);
  this.upperBound = readerSliceConfig.get(KeyConstant.UPPER_BOUND);
  this.isObjectId = readerSliceConfig.getBool(KeyConstant.IS_OBJECTID);
}

代码示例来源:origin: ECNU-1X/DataX-Masking

/**
 * 合并其他Configuration,并修改两者冲突的KV配置
 * 
 * @param another
 *            合并加入的第三方Configuration
 * @param updateWhenConflict
 *            当合并双方出现KV冲突时候,选择更新当前KV,或者忽略该KV
 * @return 返回合并后对象
 */
public Configuration merge(final Configuration another,
    boolean updateWhenConflict) {
  Set<String> keys = another.getKeys();
  for (final String key : keys) {
    // 如果使用更新策略,凡是another存在的key,均需要更新
    if (updateWhenConflict) {
      this.set(key, another.get(key));
      continue;
    }
    // 使用忽略策略,只有another Configuration存在但是当前Configuration不存在的key,才需要更新
    boolean isCurrentExists = this.get(key) != null;
    if (isCurrentExists) {
      continue;
    }
    this.set(key, another.get(key));
  }
  return this;
}

代码示例来源:origin: ECNU-1X/DataX-Masking

if (param.get(KEY_DATE) != null &&
    (param.getLong(KEY_START_TIMESTAMP_MILLIS) != null || param.getLong(KEY_END_TIMESTAMP_MILLIS) != null) &&
    (param.getLong(KEY_START_TIME_STRING) != null || param.getLong(KEY_END_TIME_STRING) != null)) {
if (param.get(KEY_DATE) != null &&
    (param.getLong(KEY_START_TIMESTAMP_MILLIS) != null || param.getLong(KEY_END_TIMESTAMP_MILLIS) != null)) {
  throw new OTSStreamReaderException("Can't set date and time range both, please check your config.");
if (param.get(KEY_DATE) != null &&
    (param.getLong(KEY_START_TIME_STRING) != null || param.getLong(KEY_END_TIME_STRING) != null)) {
  throw new OTSStreamReaderException("Can't set date and time range string both, please check your config.");

代码示例来源:origin: ECNU-1X/DataX-Masking

Configuration _conf = Configuration.from(con.toString());
if (Strings.isNullOrEmpty(_conf.getString("condition"))) {
  query.put(_conf.getString("name"), _conf.get("value"));
} else {
  query.put(_conf.getString("name"),
      new BasicDBObject(_conf.getString("condition"), _conf.get("value")));

代码示例来源:origin: ECNU-1X/DataX-Masking

public static Configuration filterSensitiveConfiguration(Configuration configuration){
  Set<String> keys = configuration.getKeys();
  for (final String key : keys) {
    boolean isSensitive = StringUtils.endsWithIgnoreCase(key, "password")
        || StringUtils.endsWithIgnoreCase(key, "accessKey");
    if (isSensitive && configuration.get(key) instanceof String) {
      configuration.set(key, configuration.getString(key).replaceAll(".", "*"));
    }
  }
  return configuration;
}

代码示例来源:origin: ECNU-1X/DataX-Masking

@Override
public List<Configuration> split(int adviceNumber) {
  LOG.debug("split() begin...");
  List<Configuration> readerSplitConfigs = new ArrayList<Configuration>();
  // 将每个单独的 object 作为一个 slice
  List<String> objects = parseOriginObjects(readerOriginConfig
      .getList(Constant.OBJECT, String.class));
  if (0 == objects.size()) {
    throw DataXException.asDataXException(
        OssReaderErrorCode.EMPTY_BUCKET_EXCEPTION,
        String.format(
            "未能找到待读取的Object,请确认您的配置项bucket: %s object: %s",
            this.readerOriginConfig.get(Key.BUCKET),
            this.readerOriginConfig.get(Key.OBJECT)));
  }
  for (String object : objects) {
    Configuration splitedConfig = this.readerOriginConfig.clone();
    splitedConfig.set(Constant.OBJECT, object);
    readerSplitConfigs.add(splitedConfig);
    LOG.info(String.format("OSS object to be read:%s", object));
  }
  LOG.debug("split() ok and end...");
  return readerSplitConfigs;
}

代码示例来源:origin: ECNU-1X/DataX-Masking

@Override
public void init() {
  writerSliceConfig = super.getPluginJobConf();
  this.writeMode = this.writerSliceConfig.getString(Key.WRITE_MODE);
  this.schema = writerSliceConfig.getString(Key.SCHEMA);
  this.table =  writerSliceConfig.getString(Key.ADS_TABLE);
  
  if(Constant.LOADMODE.equalsIgnoreCase(this.writeMode)) {
    odpsWriterTaskProxy.setPluginJobConf(writerSliceConfig);
    odpsWriterTaskProxy.init();
  } else if(Constant.INSERTMODE.equalsIgnoreCase(this.writeMode) || Constant.STREAMMODE.equalsIgnoreCase(this.writeMode)) {
    try {
      this.tableInfo = AdsUtil.createAdsHelper(this.writerSliceConfig).getTableInfo(this.table);
    } catch (AdsException e) {
      throw DataXException.asDataXException(AdsWriterErrorCode.CREATE_ADS_HELPER_FAILED, e);
    }
    List<String> allColumns = new ArrayList<String>();
    List<ColumnInfo> columnInfo =  this.tableInfo.getColumns();
    for (ColumnInfo eachColumn : columnInfo) {
      allColumns.add(eachColumn.getName());
    }
    LOG.info("table:[{}] all columns:[\n{}\n].", this.writerSliceConfig.get(Key.ADS_TABLE), StringUtils.join(allColumns, ","));
    AdsInsertUtil.dealColumnConf(writerSliceConfig, allColumns);
    List<String> userColumns = writerSliceConfig.getList(Key.COLUMN, String.class);
    this.columnNumber = userColumns.size();
  } else {
    throw DataXException.asDataXException(AdsWriterErrorCode.INVALID_CONFIG_VALUE, "writeMode 必须为 'load' 或者 'insert' 或者 'stream'");
  }
}

代码示例来源:origin: ECNU-1X/DataX-Masking

if (lastPathKey.length() > 1 && lastPathKey.charAt(0) == '*'
    && lastPathKey.charAt(1) != '*') {
  Object value = config.get(key);
  if (value instanceof String) {
    String newKey = key.substring(0, lastPathIndex)

代码示例来源:origin: ECNU-1X/DataX-Masking

/**
 * 根据用户提供的json path,寻址List对象,如果对象不存在,返回null
 */
@SuppressWarnings("unchecked")
public List<Object> getList(final String path) {
  List<Object> list = this.get(path, List.class);
  if (null == list) {
    return null;
  }
  return list;
}

代码示例来源:origin: ECNU-1X/DataX-Masking

/**
 * 根据用户提供的json path,寻址String对象
 * 
 * @return String对象,如果path不存在或者String不存在,返回null
 */
public String getString(final String path) {
  Object string = this.get(path);
  if (null == string) {
    return null;
  }
  return String.valueOf(string);
}

代码示例来源:origin: ECNU-1X/DataX-Masking

/**
 * 根据用户提供的json path,寻址List对象,如果对象不存在,返回null
 */
@SuppressWarnings("unchecked")
public <T> List<T> getList(final String path, Class<T> t) {
  Object object = this.get(path, List.class);
  if (null == object) {
    return null;
  }
  List<T> result = new ArrayList<T>();
  List<Object> origin = (List<Object>) object;
  for (final Object each : origin) {
    result.add((T) each);
  }
  return result;
}

代码示例来源:origin: ECNU-1X/DataX-Masking

/**
 * 根据用户提供的json path,寻址Map对象,如果对象不存在,返回null;
 */
@SuppressWarnings("unchecked")
public <T> Map<String, T> getMap(final String path, Class<T> t) {
  Map<String, Object> map = this.get(path, Map.class);
  if (null == map) {
    return null;
  }
  Map<String, T> result = new HashMap<String, T>();
  for (final String key : map.keySet()) {
    result.put(key, (T) map.get(key));
  }
  return result;
}

代码示例来源:origin: ECNU-1X/DataX-Masking

/**
 * 根据用户提供的json path,寻址包含Configuration的Map,如果对象不存在,返回默认null
 */
@SuppressWarnings("unchecked")
public Map<String, Configuration> getMapConfiguration(final String path) {
  Map<String, Object> map = this.get(path, Map.class);
  if (null == map) {
    return null;
  }
  Map<String, Configuration> result = new HashMap<String, Configuration>();
  for (final String key : map.keySet()) {
    result.put(key, Configuration.from(Configuration.toJSONString(map
        .get(key))));
  }
  return result;
}

代码示例来源:origin: ECNU-1X/DataX-Masking

/**
 * 根据用户提供的json path,寻址具体的对象,并转为用户提供的类型
 * <p/>
 * <br>
 * <p/>
 * NOTE: 目前仅支持Map以及List下标寻址, 例如:
 * <p/>
 * <br />
 * <p/>
 * 对于如下JSON
 * <p/>
 * {"a": {"b": {"c": [0,1,2,3]}}}
 * <p/>
 * config.get("") 返回整个Map <br>
 * config.get("a") 返回a下属整个Map <br>
 * config.get("a.b.c") 返回c对应的数组List <br>
 * config.get("a.b.c[0]") 返回数字0
 * 
 * @return Java表示的JSON对象,如果转型失败,将抛出异常
 */
@SuppressWarnings("unchecked")
public <T> T get(final String path, Class<T> clazz) {
  this.checkPath(path);
  return (T) this.get(path);
}

代码示例来源:origin: ECNU-1X/DataX-Masking

/**
 * 删除path对应的值,如果path不存在,将抛出异常。
 */
public Object remove(final String path) {
  final Object result = this.get(path);
  if (null == result) {
    throw DataXException.asDataXException(
        CommonErrorCode.RUNTIME_ERROR,
        String.format("配置文件对应Key[%s]并不存在,该情况是代码编程错误. 请联系DataX团队的同学.", path));
  }
  this.set(path, null);
  return result;
}

代码示例来源:origin: ECNU-1X/DataX-Masking

/**
 * 用户指定部分path,获取Configuration的子集
 * <p/>
 * <br>
 * 如果path获取的路径或者对象不存在,返回null
 */
public Configuration getConfiguration(final String path) {
  Object object = this.get(path);
  if (null == object) {
    return null;
  }
  return Configuration.from(Configuration.toJSONString(object));
}

代码示例来源:origin: ECNU-1X/DataX-Masking

public void init() {
  try {
    OTSStreamReaderConfig config = GsonParser.jsonToConfig(
        (String) this.getPluginJobConf().get(OTSStreamReaderConstants.CONF));
    StreamJob streamJob = StreamJob.fromJson(
        (String) this.getPluginJobConf().get(OTSStreamReaderConstants.STREAM_JOB));
    List<String> ownedShards = GsonParser.jsonToList(
        (String) this.getPluginJobConf().get(OTSStreamReaderConstants.OWNED_SHARDS));
    List<StreamShard> allShards = GsonParser.fromJson(
        (String) this.getPluginJobConf().get(OTSStreamReaderConstants.ALL_SHARDS));
    proxy.init(config, streamJob, allShards, new HashSet<String>(ownedShards));
  } catch (TableStoreException ex) {
    throw DataXException.asDataXException(new OTSReaderError(ex.getErrorCode(), "OTS ERROR"), ex.toString(), ex);
  } catch (Exception ex) {
    throw DataXException.asDataXException(OTSReaderError.ERROR, ex.toString(), ex);
  }
}

代码示例来源:origin: ECNU-1X/DataX-Masking

/**
 * 根据用户提供的json path,插入指定对象,并返回之前存在的对象(如果存在)
 * <p/>
 * <br>
 * <p/>
 * 目前仅支持.以及数组下标寻址, 例如:
 * <p/>
 * <br />
 * <p/>
 * config.set("a.b.c[3]", object);
 * <p/>
 * <br>
 * 对于插入对象,Configuration不做任何限制,但是请务必保证该对象是简单对象(包括Map<String,
 * Object>、List<Object>),不要使用自定义对象,否则后续对于JSON序列化等情况会出现未定义行为。
 * 
 * @param path
 *            JSON path对象
 * @param object
 *            需要插入的对象
 * @return Java表示的JSON对象
 */
public Object set(final String path, final Object object) {
  checkPath(path);
  Object result = this.get(path);
  setObject(path, extractConfiguration(object));
  return result;
}

相关文章