如何使用PoolighttpClientConnectionManager管理数千个get请求?

lbsnaicq  于 2021-07-09  发布在  Java
关注(0)|答案(0)|浏览(245)

我正在使用 PoolingHttpClientConnectionManagerhttpclient 版本 4.5.3 获取URL数组。我遵循了ApacheHttpComponents教程第2.4节中描述的示例。但是,我在使用以下代码时遇到堆外内存问题:

PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
CloseableHttpClient httpClient = HttpClients.custom()
        .setConnectionManager(cm)
        .build();

// URIs to perform GETs on
String[] urisToGet = {
    "http://www.domain1.com/a",
    "http://www.domain1.com/b",
    "http://www.domain1.com/c",
    "http://www.domain1.com/d",
    // thousands more
    // ....
};

// create a thread for each URI
GetThread[] threads = new GetThread[urisToGet.length];
for (int i = 0; i < threads.length; i++) {
    HttpGet httpget = new HttpGet(urisToGet[i]);
    threads[i] = new GetThread(httpClient, httpget);
}

// start the threads
for (int j = 0; j < threads.length; j++) {
    threads[j].start();
}

// join the threads
for (int j = 0; j < threads.length; j++) {
    threads[j].join();
}

作为参考,getthread如下所示:

static class GetThread extends Thread {

    private final CloseableHttpClient httpClient;
    private final HttpContext context;
    private final HttpGet httpget;

    public GetThread(CloseableHttpClient httpClient, HttpGet httpget) {
        this.httpClient = httpClient;
        this.context = HttpClientContext.create();
        this.httpget = httpget;
    }

    @Override
    public void run() {
        try {
            CloseableHttpResponse response = httpClient.execute(
                    httpget, context);
            try {
                HttpEntity entity = response.getEntity();
            } finally {
                response.close();
            }
        } catch (ClientProtocolException ex) {
            // Handle protocol errors
        } catch (IOException ex) {
            // Handle I/O errors
        }
    }

}

我已经尝试将连接管理器传递到每个线程中,并关闭线程中过期的连接 finally 各线部分:

this.connectionManager.closeExpiredConnections();
this.connectionManager.closeIdleConnections(30, TimeUnit.SECONDS);

然而,这并不能解决这个问题。
可能是因为我创建了太多线程?我需要用吗 ThreadPoolExecutor 相反呢?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题