我在异步下载~2k个文件(每个文件0,5mb)时发生outofmemoryerror。在我的本地机器上,一切正常,但在docker映像中这样做时,它缺少内存。我不想把我的docker映像内存分配增加到本地机器的大小(尽管这是唯一可能的方法)。
要下载每个文件,我使用以下方法:
static void download(String fileUrl, File localFile) throws HttpException, CannotReadException, TagException,InvalidAudioFrameException, ExecutionException,InterruptedException, IOException {
FileOutputStream stream = new FileOutputStream(localFile)
AsyncHttpClient client = Dsl.asyncHttpClient()
client.prepareGet(fileUrl)
.execute(new AsyncCompletionHandler<FileOutputStream>() {
@Override
AsyncHandler.State onStatusReceived(HttpResponseStatus responseStatus) throws Exception {
def status = responseStatus.getStatusCode()
if (status != 200) {
println(status + " file not downloaded: " + fileUrl)
return State.ABORT
}
return State.CONTINUE
}
@Override
AsyncHandler.State onBodyPartReceived(HttpResponseBodyPart bodyPart) throws Exception {
stream.getChannel()
.write(bodyPart.getBodyByteBuffer())
return State.CONTINUE
}
@Override
FileOutputStream onCompleted(Response response) throws Exception {
return stream
}
})
.get()
stream.getChannel().close()
client.close()
}
在每个下载的文件上,我执行一些检查(也以异步方式执行)并收集结果。
List filesThatMatchMyTest = Flux.fromIterable(items)
.parallel()
.runOn(Schedulers.boundedElastic())
.map { it ->
def uri = it.URI
File localFile = File.createTempFile("file-$uri", ".txt")
String fileUrl = createFileLink(uri)
download(fileUrl, localFile)
return check(localFile, locale, fileUrl)
}
.sequential()
.filter { !it.isBlank() }
.collectList()
.block()
我寻找一种解决方案,可以帮助在代码中处理这个问题。
另请注意,本地计算机的内存为16GB,而docker映像的内存为2GB
暂无答案!
目前还没有任何答案,快来回答吧!