jvm 使用IOUtils.copyLarge时出现Java堆空间不足的错误,我如何解决这个问题?

unftdfkk  于 2024-01-07  发布在  Java
关注(0)|答案(1)|浏览(353)

我有一个应用程序(Springboot API)运行un kubernetes,我有一个cpu:200mmemory:8Gi的示例。我的jvm参数是:_JAVA_OPTIONS: -Xmx7G -Xms7G我的目标是将一个或多个大小在30Mb1Gb之间的文件从一个s3 bucket复制到另一个带有预签名url的s3 bucket。
我得到java.lang.OutOfMemoryError: Java heap space错误时,我调用我的API超过2次在1GB的文件目录。
下面是我的代码:

  1. public ResponseEntity<String> transferData(Config config) throws IOException {
  2. var inputUrl = config.getInputdUrl();
  3. var outputUrl = config.getOutputdUrl();
  4. var uploadConnection = uploader.getUploadConnection(outputUrl);
  5. try (var inputStream = downloader.getDownloadConnection(inputUrl);
  6. var outputStream = uploadConnection.getOutputStream()) {
  7. IOUtils.copyLarge(inputStream, outputStream);
  8. var resp = uploadConnection.getResponseCode();
  9. return ResponseEntity
  10. .status(HttpStatus.OK)
  11. .body("Transfer done with response:" + resp);
  12. } catch (IOException ioe) {
  13. return ResponseEntity
  14. .status(HttpStatus.INTERNAL_SERVER_ERROR)
  15. .body("An error occurred while transferring data : " + ioe);
  16. }
  17. }

字符串
getUploadConnection返回HttpURLConnection

  1. var connection = (HttpURLConnection) new URL(presignedUrl).openConnection();
  2. connection.setDoOutput(true);
  3. connection.setRequestMethod(PUT.toString());
  4. return connection;


getDownloadConnection返回HttpURLConnection inputStream。

  1. HttpURLConnection connection = (HttpURLConnection) new URL(presignedUrl).openConnection();
  2. connection.setRequestMethod(GET.toString());
  3. connection.setDoOutput(true);
  4. return connection.getInputStream();


关于我的代码和配置,我认为对内存的影响不太重要,因为我流文件,并没有得到它完全在内存中。
我错过了什么?
编辑1:我添加了setChunkedStreamingMode,所以现在我的getUploadConnection返回:

  1. var connection = (HttpURLConnection) new URL(url).openConnection();
  2. connection.setDoOutput(true);
  3. connection.setDoInput(true);
  4. connection.setChunkedStreamingMode(-1);
  5. connection.setRequestMethod(PUT.toString());
  6. return connection;


但是我从我的s3 bucket得到一个411错误

xn1cxnb4

xn1cxnb41#

在我的输出连接中设置了setFixedLengthStreamingMode,我能够传输更多的文件,而不会出现Out of memory error java heap space问题。
我的输出连接:

  1. var connection = (HttpURLConnection) new URL(presignedUrl).openConnection();
  2. connection.setDoOutput(true);
  3. connection.setDoInput(true);
  4. connection.setRequestMethod("PUT");
  5. connection.setFixedLengthStreamingMode(downloadConnection.getContentLengthLong());

字符串

相关问题