hadoop压缩

jjhzyzn0  于 2021-06-03  发布在  Hadoop
关注(0)|答案(0)|浏览(289)

我试图用下面的代码压缩一个文件。当文件的大小很小(比如1GB)时,压缩工作正常。但当文件大小在5gb左右时,程序不会失败,而是继续运行2天,没有任何结果。根据我得到的信息消息,这似乎是集群问题,虽然我不太确定。
下面是我得到的错误代码:
错误

我使用的代码

  1. public void compressData(final String inputFilePath,final String outputPath) throws DataFabricAppendException {
  2. CompressionOutputStream compressionOutputStream = null;
  3. FSDataOutputStream fsDataOutputStream = null;
  4. FSDataInputStream fsDataInputStream = null;
  5. CompressionCodec compressionCodec = null;
  6. CompressionCodecFactory compressionCodecFactory = null;
  7. try {
  8. compressionCodecFactory = new CompressionCodecFactory(conf);
  9. final Path compressionFilePath = new Path(outputPath);
  10. fsDataOutputStream = fs.create(compressionFilePath);
  11. compressionCodec = compressionCodecFactory
  12. .getCodecByClassName(BZip2Codec.class.getName());
  13. compressionOutputStream = compressionCodec
  14. .createOutputStream(fsDataOutputStream);
  15. fsDataInputStream = new FSDataInputStream(fs.open(new Path(
  16. inputFilePath)));
  17. IOUtils.copyBytes(fsDataInputStream, compressionOutputStream, conf,
  18. false);
  19. compressionOutputStream.finish();
  20. } catch (IOException ex) {
  21. throw new DataFabricAppendException(
  22. "Error while compressing non-partitioned file : "
  23. + inputFilePath, ex);
  24. } catch (Exception ex) {
  25. throw new DataFabricAppendException(
  26. "Error while compressing non-partitioned file : "
  27. + inputFilePath, ex);
  28. } finally {
  29. try {
  30. if (compressionOutputStream != null) {
  31. compressionOutputStream.close();
  32. }
  33. if (fsDataInputStream != null) {
  34. fsDataInputStream.close();
  35. }
  36. if (fsDataOutputStream != null) {
  37. fsDataOutputStream.close();
  38. }
  39. } catch (IOException e1) {
  40. LOG.warn("Could not close necessary objects");
  41. }
  42. }
  43. }

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题