org.apache.gobblin.config.store.zip.ZipFileConfigStore类的使用及代码示例

x33g5p2x  于2022-02-05 转载在 其他  
字(10.4k)|赞(0)|评价(0)|浏览(82)

本文整理了Java中org.apache.gobblin.config.store.zip.ZipFileConfigStore类的一些代码示例，展示了ZipFileConfigStore类的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。ZipFileConfigStore类的具体详情如下：
包路径：org.apache.gobblin.config.store.zip.ZipFileConfigStore
类名称：ZipFileConfigStore

ZipFileConfigStore介绍

[英]ConfigStore that uses a zipped file containing all the config store paths. Similar to SimpleHadoopFilesystemConfigStore but using java APIs instead of Hadoop APIs for the Filesystem to allow reading the file without unzipping. It is assumed that the version passed in the constructor will be the only version used.
[中]使用包含所有配置存储路径的压缩文件的ConfigStore。与SimpleHadoopFilesystemConfigStore类似，但对文件系统使用java API而不是Hadoop API，以允许在不解压缩的情况下读取文件。假设构造函数中传递的版本将是唯一使用的版本。

代码示例

代码示例来源：origin: apache/incubator-gobblin

@BeforeClass
public void setUp() throws URISyntaxException, ConfigStoreCreationException, IOException {
 Path path = Paths.get(this.getClass().getClassLoader().getResource("zipStoreTest.zip").getPath());
 FileSystem fs = FileSystems.newFileSystem(path, null);
 this.store = new ZipFileConfigStore((ZipFileSystem) fs, path.toUri(), this.version, "_CONFIG_STORE");
}

代码示例来源：origin: apache/incubator-gobblin

/**
 * Retrieves all the {@link ConfigKeyPath}s that are imported by the given {@link ConfigKeyPath}. Similar to
 * {@link SimpleHadoopFilesystemConfigStore#getOwnImports}
 */
@Override
public List<ConfigKeyPath> getOwnImports(ConfigKeyPath configKey, String version) {
 return getOwnImports(configKey, version, Optional.<Config>absent());
}

代码示例来源：origin: apache/incubator-gobblin

public List<ConfigKeyPath> getOwnImports(ConfigKeyPath configKey, String version, Optional<Config> runtimeConfig)
  throws VersionDoesNotExistException {
 Preconditions.checkNotNull(configKey, "configKey cannot be null!");
 Preconditions.checkArgument(version.equals(getCurrentVersion()));
 List<ConfigKeyPath> configKeyPaths = new ArrayList<>();
 Path datasetDir = getDatasetDirForKey(configKey);
 Path includesFile = this.fs.getPath(datasetDir.toString(), SimpleHadoopFilesystemConfigStore.INCLUDES_CONF_FILE_NAME);
 try {
  if (!Files.exists(includesFile)) {
   return configKeyPaths;
  }
  if (!Files.isDirectory(includesFile)) {
   try (InputStream includesConfInStream = Files.newInputStream(includesFile)) {
    configKeyPaths = SimpleHadoopFilesystemConfigStore.getResolvedConfigKeyPaths(includesConfInStream, runtimeConfig);
   }
  }
 } catch (IOException e) {
  throw new RuntimeException(String.format("Error while getting config for configKey: \"%s\"", configKey), e);
 }
 return configKeyPaths;
}

代码示例来源：origin: apache/incubator-gobblin

@Test
public void testGetOwnConfig() {
 Config config1 = this.store.getOwnConfig(this.rootPath, this.version);
 Assert.assertEquals(config1.getString("gobblin.property.test1"), "prop1");
 Assert.assertEquals(config1.getString("gobblin.property.test2"), "prop2");
 Config config2 = this.store.getOwnConfig(this.testPath, this.version);
 Assert.assertEquals(config2.getString("gobblin.test.property"), "string1");
 Config config3 = this.store.getOwnConfig(this.child1Path, this.version);
 Assert.assertEquals(config3.getString("gobblin.test.property"), "string2");
 Config config4 = this.store.getOwnConfig(this.child2Path, this.version);
 Assert.assertEquals(config4.getString("gobblin.test.property"), "string3");
}

代码示例来源：origin: apache/incubator-gobblin

@Test
 public void testGetChildren() {
  Collection<ConfigKeyPath> children1 = this.store.getChildren(this.rootPath, this.version);
  Assert.assertEquals(children1.size(), 1);
  Assert.assertTrue(children1.contains(this.testPath));

  Collection<ConfigKeyPath> children2 = this.store.getChildren(this.testPath, this.version);
  Assert.assertEquals(children2.size(), 2);
  Assert.assertTrue(children2.contains(this.child1Path));
  Assert.assertTrue(children2.contains(this.child2Path));

  Collection<ConfigKeyPath> children3 = this.store.getChildren(this.child1Path, this.version);
  Assert.assertEquals(children3.size(), 0);
 }
}

代码示例来源：origin: apache/incubator-gobblin

/**
 * Retrieves the {@link Config} for the given {@link ConfigKeyPath}. Similar to
 * {@link SimpleHadoopFilesystemConfigStore#getOwnConfig}
 */
@Override
public Config getOwnConfig(ConfigKeyPath configKey, String version) throws VersionDoesNotExistException {
 Preconditions.checkNotNull(configKey, "configKey cannot be null!");
 Preconditions.checkArgument(version.equals(getCurrentVersion()));
 Path datasetDir = getDatasetDirForKey(configKey);
 Path mainConfFile = this.fs.getPath(datasetDir.toString(), SimpleHadoopFilesystemConfigStore.MAIN_CONF_FILE_NAME);
 try {
  if (!Files.exists(mainConfFile)) {
   return ConfigFactory.empty();
  }
  if (!Files.isDirectory(mainConfFile)) {
   try (InputStream mainConfInputStream = Files.newInputStream(mainConfFile)) {
    return ConfigFactory.parseReader(new InputStreamReader(mainConfInputStream, Charsets.UTF_8));
   }
  }
  return ConfigFactory.empty();
 } catch (IOException e) {
  throw new RuntimeException(String.format("Error while getting config for configKey: \"%s\"", configKey), e);
 }
}

代码示例来源：origin: apache/incubator-gobblin

@Test
public void testGetOwnImports() {
 Collection<ConfigKeyPath> imports1 = this.store.getOwnImports(this.child1Path, this.version);
 Assert.assertEquals(imports1.size(), 1);
 Assert.assertTrue(imports1.contains(this.child1Path));
 Collection<ConfigKeyPath> imports2 = this.store.getOwnImports(this.child2Path, this.version);
 Assert.assertEquals(imports2.size(), 0);
}

代码示例来源：origin: apache/incubator-gobblin

return new ZipFileConfigStore((ZipFileSystem) zipFs, getBaseURI(configKey), currentVersion, factoryProps.getProperty(STORE_PREFIX_KEY, ""));
} catch (IOException | URISyntaxException e) {
 throw new ConfigStoreCreationException(configKey, e);

代码示例来源：origin: apache/incubator-gobblin

/**
 * Retrieves all the children of the given {@link ConfigKeyPath} using {@link Files#walk} to list files
 */
@Override
public Collection<ConfigKeyPath> getChildren(ConfigKeyPath configKey, String version)
  throws VersionDoesNotExistException {
 Preconditions.checkNotNull(configKey, "configKey cannot be null!");
 Preconditions.checkArgument(version.equals(getCurrentVersion()));
 List<ConfigKeyPath> children = new ArrayList<>();
 Path datasetDir = getDatasetDirForKey(configKey);
 try {
  if (!Files.exists(this.fs.getPath(datasetDir.toString()))) {
   return children;
  }
  Stream<Path> files = Files.walk(datasetDir, 1);
  for (Iterator<Path> it = files.iterator(); it.hasNext();) {
   Path path = it.next();
   if (Files.isDirectory(path) && !path.equals(datasetDir)) {
    children.add(configKey.createChild(StringUtils.removeEnd(path.getName(path.getNameCount() - 1).toString(),
      SingleLinkedListConfigKeyPath.PATH_DELIMETER)));
   }
  }
  return children;
 } catch (IOException e) {
  throw new RuntimeException(String.format("Error while getting children for configKey: \"%s\"", configKey), e);
 }
}

代码示例来源：origin: org.apache.gobblin/gobblin-config-core

/**
 * Retrieves all the {@link ConfigKeyPath}s that are imported by the given {@link ConfigKeyPath}. Similar to
 * {@link SimpleHadoopFilesystemConfigStore#getOwnImports}
 */
@Override
public List<ConfigKeyPath> getOwnImports(ConfigKeyPath configKey, String version) {
 return getOwnImports(configKey, version, Optional.<Config>absent());
}

代码示例来源：origin: org.apache.gobblin/gobblin-config-core

return new ZipFileConfigStore((ZipFileSystem) zipFs, getBaseURI(configKey), currentVersion, factoryProps.getProperty(STORE_PREFIX_KEY, ""));
} catch (IOException | URISyntaxException e) {
 throw new ConfigStoreCreationException(configKey, e);

代码示例来源：origin: org.apache.gobblin/gobblin-config-core

public List<ConfigKeyPath> getOwnImports(ConfigKeyPath configKey, String version, Optional<Config> runtimeConfig)
  throws VersionDoesNotExistException {
 Preconditions.checkNotNull(configKey, "configKey cannot be null!");
 Preconditions.checkArgument(version.equals(getCurrentVersion()));
 List<ConfigKeyPath> configKeyPaths = new ArrayList<>();
 Path datasetDir = getDatasetDirForKey(configKey);
 Path includesFile = this.fs.getPath(datasetDir.toString(), SimpleHadoopFilesystemConfigStore.INCLUDES_CONF_FILE_NAME);
 try {
  if (!Files.exists(includesFile)) {
   return configKeyPaths;
  }
  if (!Files.isDirectory(includesFile)) {
   try (InputStream includesConfInStream = Files.newInputStream(includesFile)) {
    configKeyPaths = SimpleHadoopFilesystemConfigStore.getResolvedConfigKeyPaths(includesConfInStream, runtimeConfig);
   }
  }
 } catch (IOException e) {
  throw new RuntimeException(String.format("Error while getting config for configKey: \"%s\"", configKey), e);
 }
 return configKeyPaths;
}

代码示例来源：origin: org.apache.gobblin/gobblin-config-core

/**
 * Retrieves all the children of the given {@link ConfigKeyPath} using {@link Files#walk} to list files
 */
@Override
public Collection<ConfigKeyPath> getChildren(ConfigKeyPath configKey, String version)
  throws VersionDoesNotExistException {
 Preconditions.checkNotNull(configKey, "configKey cannot be null!");
 Preconditions.checkArgument(version.equals(getCurrentVersion()));
 List<ConfigKeyPath> children = new ArrayList<>();
 Path datasetDir = getDatasetDirForKey(configKey);
 try {
  if (!Files.exists(this.fs.getPath(datasetDir.toString()))) {
   return children;
  }
  Stream<Path> files = Files.walk(datasetDir, 1);
  for (Iterator<Path> it = files.iterator(); it.hasNext();) {
   Path path = it.next();
   if (Files.isDirectory(path) && !path.equals(datasetDir)) {
    children.add(configKey.createChild(StringUtils.removeEnd(path.getName(path.getNameCount() - 1).toString(),
      SingleLinkedListConfigKeyPath.PATH_DELIMETER)));
   }
  }
  return children;
 } catch (IOException e) {
  throw new RuntimeException(String.format("Error while getting children for configKey: \"%s\"", configKey), e);
 }
}

代码示例来源：origin: org.apache.gobblin/gobblin-config-core

/**
 * Retrieves the {@link Config} for the given {@link ConfigKeyPath}. Similar to
 * {@link SimpleHadoopFilesystemConfigStore#getOwnConfig}
 */
@Override
public Config getOwnConfig(ConfigKeyPath configKey, String version) throws VersionDoesNotExistException {
 Preconditions.checkNotNull(configKey, "configKey cannot be null!");
 Preconditions.checkArgument(version.equals(getCurrentVersion()));
 Path datasetDir = getDatasetDirForKey(configKey);
 Path mainConfFile = this.fs.getPath(datasetDir.toString(), SimpleHadoopFilesystemConfigStore.MAIN_CONF_FILE_NAME);
 try {
  if (!Files.exists(mainConfFile)) {
   return ConfigFactory.empty();
  }
  if (!Files.isDirectory(mainConfFile)) {
   try (InputStream mainConfInputStream = Files.newInputStream(mainConfFile)) {
    return ConfigFactory.parseReader(new InputStreamReader(mainConfInputStream, Charsets.UTF_8));
   }
  }
  return ConfigFactory.empty();
 } catch (IOException e) {
  throw new RuntimeException(String.format("Error while getting config for configKey: \"%s\"", configKey), e);
 }
}

内容来源于网络，如有侵权，请联系作者删除！

ZipFileConfigStore

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com