我需要在Java上运行多个cURL命令。我一直在寻找各种方法来实现这一点。其中一种方法是使用ProcessBuilder。我编写的代码如下所示:
private void performCurl() {
ProcessBuilder processBuilder = new ProcessBuilder();
List<String> curlArgs = getMyCurlArgs(); // curl -k -v https://www.amazon.com -H <and so on>
String listString = String.join(" ", curlArgs);
processBuilder.command(curlArgs);
processBuilder.redirectErrorStream(true);
Process proc = processBuilder.start();
ExecutorService fixedThreadPool = Executors.newFixedThreadPool(poolSize);
Future<String> futureOpt;
try {
futureOpt = fixedThreadPool.submit(() -> {
StringBuilder sb = new StringBuilder();
InputStream ins = proc.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(ins));
br.lines().forEach(sb::append);
try {
ins.close();
br.close();
} catch (IOException e) {
// my exception
}
return sb.toString();
});
boolean terminatedNormally = proc.waitFor(15, TimeUnit.SECONDS);
if (!terminatedNormally)
throw new SocketTimeoutException("Timed Out");
} finally {
fixedThreadPool.shutdown();
proc.destroy();
}
String content = futureOpt.get(); // This content is what I use.
}
现在上面的代码可以正常工作了。cURL抓取网站并提供HTML内容。The problem is that ProcessBuilder with cURL is extremely CPU intensive. Especially given the fact that ProcessBuilder makes use of Operating System resources.
我现在的问题是:
a)我可以更有效地使用ProcessBuilder吗?
B)或者,是否有其他机制可以触发Java上的并行cURL请求?
是否有其他方法可以并行运行cURL请求?
1条答案
按热度按时间sdnqo3pr1#
至少还有两种选择可能会更好。
首先,您要单独运行每个外部
curl
命令,您可以创建一个脚本来并行运行所有命令(例如shell脚本),然后执行一个shell命令来运行该脚本,这将减少所有ProcessBuilders
的开销第二,你可以使用普通的java HTTP库连接到服务器并下载内容,这将消除外部操作系统命令的开销。