org.apache.beam.sdk.io.FileSystems类的使用及代码示例

x33g5p2x  于2022-01-19 转载在 其他  
字(9.0k)|赞(0)|评价(0)|浏览(179)

本文整理了Java中org.apache.beam.sdk.io.FileSystems类的一些代码示例，展示了FileSystems类的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。FileSystems类的具体详情如下：
包路径：org.apache.beam.sdk.io.FileSystems
类名称：FileSystems

FileSystems介绍

[英]Clients facing FileSystem utility.
[中]面向文件系统实用程序的客户端。

代码示例

代码示例来源：origin: spotify/dbeam

public static void writeToFile(String filename, ByteBuffer contents) throws IOException {
 ResourceId resourceId = FileSystems.matchNewResource(filename, false);
 try (WritableByteChannel out = FileSystems.create(resourceId, MimeTypes.TEXT)) {
  out.write(contents);
 }
}

代码示例来源：origin: GoogleCloudPlatform/java-docs-samples

public static void runAvroToCsv(SampleOptions options)
  throws IOException, IllegalArgumentException {
 FileSystems.setDefaultPipelineOptions(options);
 // Get Avro Schema
 String schemaJson = getSchema(options.getAvroSchema());
 Schema schema = new Schema.Parser().parse(schemaJson);
 // Check schema field types before starting the Dataflow job
 checkFieldTypes(schema);
 // Create the Pipeline object with the options we defined above.
 Pipeline pipeline = Pipeline.create(options);
 // Convert Avro To CSV
 pipeline.apply("Read Avro files",
   AvroIO.readGenericRecords(schemaJson).from(options.getInputFile()))
   .apply("Convert Avro to CSV formatted data",
     ParDo.of(new ConvertAvroToCsv(schemaJson, options.getCsvDelimiter())))
   .apply("Write CSV formatted data", TextIO.write().to(options.getOutput())
     .withSuffix(".csv"));
 // Run the pipeline.
 pipeline.run().waitUntilFinish();
}

代码示例来源：origin: GoogleCloudPlatform/java-docs-samples

public static String getSchema(String schemaPath) throws IOException {
 ReadableByteChannel chan = FileSystems.open(FileSystems.matchNewResource(
   schemaPath, false));
 try (InputStream stream = Channels.newInputStream(chan)) {
  BufferedReader streamReader = new BufferedReader(new InputStreamReader(stream, "UTF-8"));
  StringBuilder dataBuilder = new StringBuilder();
  String line;
  while ((line = streamReader.readLine()) != null) {
   dataBuilder.append(line);
  }
  return dataBuilder.toString();
 }
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

/** Like {@link #match(String)}, but with a configurable {@link EmptyMatchTreatment}. */
public static MatchResult match(String spec, EmptyMatchTreatment emptyMatchTreatment)
  throws IOException {
 MatchResult res = match(spec);
 return maybeAdjustEmptyMatchResult(spec, res, emptyMatchTreatment);
}

代码示例来源：origin: GoogleCloudPlatform/cloud-bigtable-client

@Override
 public ResourceId apply(String input) {
  return FileSystems.matchNewResource(input, true);
 }
}

代码示例来源：origin: GoogleCloudPlatform/DataflowTemplates

ResourceId outputDir = FileSystems.matchNewResource(destinationLocation.get(), true);
ResourceId outputFile =
  outputDir.resolve(outputFilename, StandardResolveOptions.RESOLVE_FILE);
  compression.readDecompressed(FileSystems.open(inputFile))) {
 try (WritableByteChannel writerChannel = FileSystems.create(tempFile, MimeTypes.TEXT)) {
  ByteStreams.copy(readerChannel, writerChannel);
 FileSystems.rename(
   ImmutableList.of(tempFile),
   ImmutableList.of(outputFile),

代码示例来源：origin: org.apache.beam/beam-sdks-java-io-google-cloud-platform

static void removeTemporaryFiles(Iterable<String> files) throws IOException {
  ImmutableList.Builder<ResourceId> fileResources = ImmutableList.builder();
  for (String file : files) {
   fileResources.add(FileSystems.matchNewResource(file, false /* isDirectory */));
  }
  FileSystems.delete(fileResources.build());
 }
}

代码示例来源：origin: com.spotify/scio-core

/**
 * Check if a remote {@link URI} exists.
 */
public boolean remoteExists(URI uri) throws IOException {
 try {
  FileSystems.matchSingleFileSpec(uri.toString());
  return true;
 } catch (FileNotFoundException e) {
  return false;
 }
}

代码示例来源：origin: GoogleCloudPlatform/DataflowTemplates

/** utility method to copy binary (jar file) data from source to dest. */
 private static void copy(ResourceId source, ResourceId dest) throws IOException {
  try (ReadableByteChannel rbc = FileSystems.open(source)) {
   try (WritableByteChannel wbc = FileSystems.create(dest, MimeTypes.BINARY)) {
    ByteStreams.copy(rbc, wbc);
   }
  }
 }
}

代码示例来源：origin: com.spotify/scio-core

private static void copyToLocal(Metadata src, Path dst) throws IOException {
 FileChannel dstCh = FileChannel.open(
   dst, StandardOpenOption.WRITE, StandardOpenOption.CREATE_NEW);
 ReadableByteChannel srcCh = FileSystems.open(src.resourceId());
 long srcSize = src.sizeBytes();
 long copied = 0;
 do {
  copied += dstCh.transferFrom(srcCh, copied, srcSize - copied);
 } while (copied < srcSize);
 dstCh.close();
 srcCh.close();
 Preconditions.checkState(copied == srcSize);
}

代码示例来源：origin: spotify/dbeam

String readFromFile(String passwordFile) throws IOException {
 MatchResult.Metadata m = FileSystems.matchSingleFileSpec(passwordFile);
 LOGGER.info("Reading password from file: {}", m.resourceId().toString());
 InputStream inputStream = Channels.newInputStream(FileSystems.open(m.resourceId()));
 return CharStreams.toString(new InputStreamReader(inputStream, Charsets.UTF_8));
}

代码示例来源：origin: GoogleCloudPlatform/DataflowTemplates

MatchResult result = FileSystems.match(path);
checkArgument(
  result.status() == Status.OK && !result.metadata().isEmpty(),
       try (Reader reader =
         Channels.newReader(
           FileSystems.open(resourceId), StandardCharsets.UTF_8.name())) {
        return CharStreams.toString(reader);
       } catch (IOException e) {

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

/**
 * Like {@link #match(List)}, but for a single resource specification.
 *
 * <p>The function {@link #match(List)} is preferred when matching multiple patterns, as it allows
 * for bulk API calls to remote filesystems.
 */
public static MatchResult match(String spec) throws IOException {
 List<MatchResult> matches = match(Collections.singletonList(spec));
 verify(
   matches.size() == 1,
   "FileSystem implementation for %s did not return exactly one MatchResult: %s",
   spec,
   matches);
 return matches.get(0);
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

/**
 * Returns a write channel for the given {@link ResourceId}.
 *
 * <p>The resource is not expanded; it is used verbatim.
 *
 * @param resourceId the reference of the file-like resource to create
 * @param mimeType the mine type of the file-like resource to create
 */
public static WritableByteChannel create(ResourceId resourceId, String mimeType)
  throws IOException {
 return create(resourceId, StandardCreateOptions.builder().setMimeType(mimeType).build());
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

public final void cleanup() throws Exception {
 if (outputFile != null) {
  LOG.info("Deleting temporary file {}", outputFile);
  // outputFile may be null if open() was not called or failed.
  FileSystems.delete(
    Collections.singletonList(outputFile), StandardMoveOptions.IGNORE_MISSING_FILES);
 }
}

代码示例来源：origin: org.apache.beam/beam-sdks-java-core

MatchResult singleMatch =
    Iterables.getOnlyElement(
      FileSystems.match(Collections.singletonList(tempDir.toString() + "*")));
  for (Metadata matchResult : singleMatch.metadata()) {
   if (allMatches.add(matchResult.resourceId())) {
FileSystems.delete(allMatches, StandardMoveOptions.IGNORE_MISSING_FILES);
  FileSystems.delete(
    Collections.singletonList(tempDir), StandardMoveOptions.IGNORE_MISSING_FILES);
 } catch (Exception e) {

代码示例来源：origin: GoogleCloudPlatform/cloud-bigtable-client

@Override
 public ResourceId apply(String input) {
  return FileSystems.matchNewResource(input, true);
 }
}

代码示例来源：origin: GoogleCloudPlatform/DataflowTemplates

@ProcessElement
 public void processElement(ProcessContext context) {
  ResourceId inputFile = context.element().resourceId();
  Compression compression = compressionValue.get();
  // Add the compression extension to the output filename. Example: demo.txt -> demo.txt.gz
  String outputFilename = inputFile.getFilename() + compression.getSuggestedSuffix();
  // Resolve the necessary resources to perform the transfer
  ResourceId outputDir = FileSystems.matchNewResource(destinationLocation.get(), true);
  ResourceId outputFile =
    outputDir.resolve(outputFilename, StandardResolveOptions.RESOLVE_FILE);
  ResourceId tempFile =
    outputDir.resolve("temp-" + outputFilename, StandardResolveOptions.RESOLVE_FILE);
  // Perform the copy of the compressed channel to the destination.
  try (ReadableByteChannel readerChannel = FileSystems.open(inputFile)) {
   try (WritableByteChannel writerChannel =
     compression.writeCompressed(FileSystems.create(tempFile, MimeTypes.BINARY))) {
    // Execute the copy to the temporary file
    ByteStreams.copy(readerChannel, writerChannel);
   }
   // Rename the temporary file to the output file
   FileSystems.rename(ImmutableList.of(tempFile), ImmutableList.of(outputFile));
   // Output the path to the uncompressed file
   context.output(outputFile.toString());
  } catch (IOException e) {
   LOG.error("Error occurred during compression of {}", inputFile.toString(), e);
   context.output(DEADLETTER_TAG, KV.of(inputFile.toString(), e.getMessage()));
  }
 }
}

代码示例来源：origin: com.spotify/scio-core

private static Metadata getMetadata(URI src) throws IOException {
 return FileSystems.matchSingleFileSpec(src.toString());
}

代码示例来源：origin: org.apache.beam/beam-examples-java

public static String copyFile(ResourceId sourceFile, ResourceId destinationFile)
  throws IOException {
 try (WritableByteChannel writeChannel = FileSystems.create(destinationFile, "text/plain")) {
  try (ReadableByteChannel readChannel = FileSystems.open(sourceFile)) {
   final ByteBuffer buffer = ByteBuffer.allocateDirect(16 * 1024);
   while (readChannel.read(buffer) != -1) {
    buffer.flip();
    writeChannel.write(buffer);
    buffer.compact();
   }
   buffer.flip();
   while (buffer.hasRemaining()) {
    writeChannel.write(buffer);
   }
  }
 }
 return destinationFile.toString();
}

内容来源于网络，如有侵权，请联系作者删除！

FileSystems

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com