HtmlUnit（Java） - 快速入门学习 - 无界面浏览器

转x33g5p2x  于2022-02-12 转载在 HTML5  
字(18.9k)|赞(0)|评价(0)|浏览(428)

注意： 对于百度翻译、百度搜索、腾讯翻译等页面依然抓取不了结果，对于加密的JS文件解析基本不生效 — 推荐使用Selenium爬复杂JS、以及加密JS页面的内容

1. 概述

官方文档： https://htmlunit.sourceforge.io/
 
有具体Demo的讲解文档（搭配官方文档效果更佳）：https://www.scrapingbee.com/java-webscraping-book/
 
作用： 一个"用于Java程序的无GUI浏览器"。它对HTML文档进行建模，并提供一个API，允许您调用页面，填写表单，单击链接等…就像您在"正常"浏览器中所做的那样

2. 注意

2.0 js解析问题

根据官方文档描述，仅能解析js库： htmx, jQuery, jQuery, MochiKit, GWT, Sarissa, MooTools, Prototype, Ext, Dojo, Dojo, YUI所以遇到经过加密的JS文件、以及其他库很可能会解析失败 === 所以模拟抓百度翻译、腾讯翻译、有道翻译这些加密的JS抓不了，建议使用Selenium（Java）进行抓，不过这工具比较重，好用是非常好用、直接爬就完事压根就不用分析浏览器的请求

2.1 关闭HtmlUnit日志

  1. java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(Level.OFF);

3. 使用

依赖： https://search.maven.org/artifact/net.sourceforge.htmlunit/htmlunit

  1. <dependency>
  2.   <groupId>net.sourceforge.htmlunit</groupId>
  3.   <artifactId>htmlunit</artifactId>
  4.   <version>2.58.0</version>
  5. </dependency>

3.1 抓取IT之家周榜内容 - 单页面

抓取IT之家周榜的内容

  1. /**
  2.      * IT之家
  3.      */
  4.     @Test
  5.     @SneakyThrows
  6.     public void test10() {
  7.         
  8.         //浏览器设置
  9.         WebClient webClient = new WebClient();
  10.         webClient.setAjaxController(new NicelyResynchronizingAjaxController());
  11.         webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
  12.         webClient.getOptions().setThrowExceptionOnScriptError(false);
  13.         webClient.getOptions().setCssEnabled(true);
  14.         webClient.getOptions().setJavaScriptEnabled(true);
  15.         webClient.getOptions().setActiveXNative(false);
  16.         
  17.         //打开页面
  18.         HtmlPage page = webClient.getPage("https://www.ithome.com/");
  20.         //鼠标悬浮到周榜上
  21.         DomElement inputEle = page.getFirstByXPath("//div[@id='rank']//li[@data-id='2']");
  22.         page = (HtmlPage) inputEle.mouseOver();
  23.         DomElement ulElement = page.getFirstByXPath("//div[@id='rank']//ul[@id='d-2']");
  25.         //周榜信息
  26.         System.out.println(ulElement.asNormalizedText());
  28.     }

抓取成功

3.2 抓取IT之家周榜第九篇文章的内容 - 双页面

  1. /**
  2.      * IT之家周榜第九篇内容
  3.      */
  4.     @Test
  5.     @SneakyThrows
  6.     public void test11() {
  8.         WebClient webClient = new WebClient();
  9.         webClient.setAjaxController(new NicelyResynchronizingAjaxController());
  10.         webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
  11.         webClient.getOptions().setThrowExceptionOnScriptError(false);
  12.         webClient.getOptions().setCssEnabled(true);
  13.         webClient.getOptions().setJavaScriptEnabled(true);
  14.         webClient.getOptions().setActiveXNative(false);
  16.         HtmlPage page = webClient.getPage("https://www.ithome.com/");
  18.         //鼠标悬浮到周榜上
  19.         DomElement inputEle = page.getFirstByXPath("//div[@id='rank']//li[@data-id='2']");
  20.         page = (HtmlPage) inputEle.mouseOver();
  21.         
  22.         //获取文章链接
  23.         List<DomElement> articleLinkElems = page.getByXPath("//div[@id='rank']//ul[@id='d-2']//a");
  24.         if(CollUtil.isNotEmpty(articleLinkElems)) {
  25.             //第九篇文章
  26.             page = articleLinkElems.get(8).click();
  27.             DomElement articleDivElem = page.getFirstByXPath("//div[@id='dt']//div[@class='fl content']");
  28.             System.out.println(articleDivElem.asNormalizedText());
  29.         }
  31.     }

抓取成功

3.3 模拟用户操作 - （这个功能个人感觉非常非常的鸡肋，只能用于非常简单的JS，但是一般网站的动作触发都会进行一系列复杂的JS操作，所以想爬虫还是推荐Selenium）

示例页面

  1. <!DOCTYPE html>
  2. <html lang="en">
  4. <head>
  5.     <meta charset="UTF-8">
  6.     <meta http-equiv="X-UA-Compatible" content="IE=edge">
  7.     <meta name="viewport" content="width=device-width, initial-scale=1.0">
  8.     <title>HtmlUnit测试</title>
  9. </head>
  11. <body>
  13.     <form id="form" onclick="return false;">
  14.         <div class="container">
  16.             <input type="hidden" placeholder="Enter Username" name="mark" id="mark" required value="ajax手动提交">
  17.             <label for="uname"><b>账号</b></label>
  18.             <input type="text" placeholder="Enter Username" name="uname" id="uname" required>
  20.             <label for="psw"><b>密码</b></label>
  21.             <input type="password" placeholder="Enter Password" name="psw"  id="psw" required>
  23.             <button id="loginBtn" type="button">登陆</button>
  25.         </div>
  27.     </form>
  29.     <form id="form2" method="post" action="http://127.0.0.1:8080/login">
  30.         <div class="container">
  31.             <input type="hidden" placeholder="Enter Username" name="mark" id="mark2" required value="form表单提交">
  32.             <label for="uname"><b>账号2</b></label>
  33.             <input type="text" placeholder="Enter Username" name="uname" id="uname2" required>
  35.             <label for="psw"><b>密码2</b></label>
  36.             <input type="password" placeholder="Enter Password" name="psw"  id="psw2" required>
  38.             <button id="loginBtn2" type="submit">登陆2</button>
  40.         </div>
  42.     </form>
  44. </body>
  46. <script src="file:///G:/VsCode/开源/jquery-3.5.1/jquery-3.5.1.min.js"></script>
  47. <script>
  48.     $(function () {
  50.         //登陆
  51.         function loginOperation() {
  52.             $.post("http://127.0.0.1:8080/login",$("#form").serialize(),responseData => {
  53.                 $("body").append(`<h1>${JSON.stringify(responseData)}</h1>`)
  54.                 $("form").hide();
  55.             },"json")
  57.             return false;
  58.         }
  60.         $("#loginBtn").click(loginOperation);
  62.     })
  63. </script>
  65. </html>

登录接口代码 == springboot == 注意下面是两个文件的代码

  1. @Configuration
  2. public class SystemConfig {
  3.     //允许跨域
  4.     @Bean
  5.     public CorsFilter corsFilter() {
  6.         CorsConfiguration corsConfiguration = new CorsConfiguration();
  7.         corsConfiguration.addAllowedOriginPattern("*");
  8.         corsConfiguration.setAllowCredentials(true);
  9.         corsConfiguration.addAllowedMethod("*");
  10.         corsConfiguration.addAllowedHeader("*");
  11.         UrlBasedCorsConfigurationSource configSource = new UrlBasedCorsConfigurationSource();
  12.         configSource.registerCorsConfiguration("/**", corsConfiguration);
  13.         return new CorsFilter(configSource);
  14.     }
  15. }
  18. @Controller
  19. @RequestMapping
  20. @ResponseBody
  21. public class LoginController {
  22.     @PostMapping("login")
  23.     public Map login(HttpServletRequest request) {
  24.         Map parameterMap = new HashMap(request.getParameterMap());
  25.         parameterMap.put("name", "嗯嗯*");
  26.         return parameterMap;
  27.     }
  29. }

模拟用户表单操作

  1. /**
  2.      * 模拟用户输入
  3.      */
  4.     @Test
  5.     @SneakyThrows
  6.     public void test12() {
  8.         WebClient webClient = new WebClient();
  9.         webClient.setAjaxController(new NicelyResynchronizingAjaxController());
  10.         webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
  11.         webClient.getOptions().setThrowExceptionOnScriptError(false);
  12.         webClient.getOptions().setCssEnabled(true);
  13.         webClient.getOptions().setJavaScriptEnabled(true);
  14.         webClient.getOptions().setActiveXNative(false);
  16.         //ajax手动提交的请求
  17.         HtmlPage page = webClient.getPage("file:///C:/Users/Administrator/Desktop/index5.html");
  19.         DomElement loginNameElem = page.getElementById("uname");
  20.         loginNameElem.setAttribute("value", "root");
  21.         DomElement passwordElem = page.getElementById("psw");
  22.         passwordElem.setAttribute("value", "pswroot");
  24.         //提交form1的表单
  25.         DomElement startLoginBtnElem = page.getElementById("loginBtn");
  26.         page = startLoginBtnElem.click();
  28.         DomElement userInfoDivElem = page.getFirstByXPath("//h1");
  29.         System.out.println(userInfoDivElem.asNormalizedText());
  31.         //==================================================
  33.         //表单提交 == 返回的是JSON结果的页面，不是htmlPage页面故需要将结果转成UnexpectedPage
  34.         page = webClient.getPage("file:///C:/Users/Administrator/Desktop/index5.html");
  35.         HtmlInput inputloginNameElem = (HtmlInput) page.getElementById("uname2");
  36.         inputloginNameElem.setAttribute("value", "root2");
  37.         HtmlInput inputpasswordElem = (HtmlInput) page.getElementById("psw2");
  38.         inputpasswordElem.setAttribute("value", "pswroot2");
  40.         //提交form2的表单
  41.         HtmlForm enclosingForm = inputloginNameElem.getEnclosingForm();
  42.         UnexpectedPage page2 = webClient.getPage(enclosingForm.getWebRequest(null));
  44.         //获取响应结果
  45.         System.out.println(page2.getWebResponse().getContentAsString(UTF_8));
  46.     }

3.4 文件下载

  1. <!DOCTYPE html>
  2. <html lang="en">
  4. 	<head>
  5. 		<meta charset="UTF-8">
  6. 		<meta http-equiv="X-UA-Compatible" content="IE=edge">
  7. 		<meta name="viewport" content="width=device-width, initial-scale=1.0">
  8. 		<title>HtmlUnit测试</title>
  9. 	</head>
  11. 	<body>
  13. 		<form id="form" onclick="return false;">
  14. 			<div class="container">
  16. 				<input type="hidden" placeholder="Enter Username" name="mark" id="mark" required value="ajax手动提交">
  17. 				<label for="uname"><b>账号</b></label>
  18. 				<input type="text" placeholder="Enter Username" name="uname" id="uname" required>
  20. 				<label for="psw"><b>密码</b></label>
  21. 				<input type="password" placeholder="Enter Password" name="psw" id="psw" required>
  23. 				<button id="loginBtn" type="button">登陆</button>
  25. 			</div>
  27. 		</form>
  29. 		<form id="form2" method="post" action="http://127.0.0.1:8080/login">
  30. 			<div class="container">
  31. 				<input type="hidden" placeholder="Enter Username" name="mark" id="mark2" required value="form表单提交">
  32. 				<label for="uname"><b>账号2</b></label>
  33. 				<input type="text" placeholder="Enter Username" name="uname" id="uname2" required>
  35. 				<label for="psw"><b>密码2</b></label>
  36. 				<input type="password" placeholder="Enter Password" name="psw" id="psw2" required>
  38. 				<button id="loginBtn2" type="submit">登陆2</button>
  39. 			</div>
  40. 		</form>
  41. 		
  42. 		
  43. 		
  44. 		<a href="http://127.0.0.1:8080/download" id="downloadBtn">下载按钮（当前页面）</a>
  45. 		<br/>
  46. 		<a href="http://127.0.0.1:8080/download" id="downloadBtn2" target="_blank">下载按钮2（新页面）</a>
  48. 	</body>
  50. 	<script src="file:///G:/VsCode/开源/jquery-3.5.1/jquery-3.5.1.min.js"></script>
  51. 	<script>
  52. 		$(function() {
  54. 			//登陆
  55. 			function loginOperation() {
  56. 				$.post("http://127.0.0.1:8080/login", $("#form").serialize(), responseData => {
  57. 					$("body").append(`<h1>${JSON.stringify(responseData)}</h1>`)
  58. 					$("form").hide();
  59. 				}, "json")
  61. 				return false;
  62. 			}
  64. 			$("#loginBtn").click(loginOperation);
  65. 			
  67. 		})
  68. 	</script>
  70. </html>

文件下载接口

  1. package work.linruchang.qq.htmlunitweb.controller;
  3. import cn.hutool.core.util.StrUtil;
  4. import lombok.SneakyThrows;
  5. import org.springframework.core.io.FileSystemResource;
  6. import org.springframework.http.HttpHeaders;
  7. import org.springframework.http.MediaType;
  8. import org.springframework.http.ResponseEntity;
  9. import org.springframework.stereotype.Controller;
  10. import org.springframework.web.bind.annotation.GetMapping;
  11. import org.springframework.web.bind.annotation.PostMapping;
  12. import org.springframework.web.bind.annotation.RequestMapping;
  13. import org.springframework.web.bind.annotation.ResponseBody;
  15. import javax.servlet.http.HttpServletRequest;
  16. import javax.servlet.http.HttpServletResponse;
  17. import java.net.URLEncoder;
  18. import java.util.HashMap;
  19. import java.util.Map;
  21. /**
  22.  * 作用：
  23.  *
  24.  * @author LinRuChang
  25.  * @version 1.0
  26.  * @date 2022/02/09
  27.  * @since 1.8
  28.  **/
  29. @Controller
  30. @RequestMapping
  31. @ResponseBody
  32. public class HtmlUnitController {
  34.     /**
  35.      * 下载文件测试
  36.      * http://127.0.0.1:8080/download
  37.      * @param request
  38.      * @param httpServletResponse
  39.      * @return
  40.      */
  41.     @GetMapping("download")
  42.     @SneakyThrows
  43.     public ResponseEntity login(HttpServletRequest request, HttpServletResponse httpServletResponse) {
  45.         System.out.println(request.getSession().getId() + "开始下载");
  47.         FileSystemResource fileSystemResource = new FileSystemResource("E:\\微信\\文件\\WeChat Files\\wxid_n7xzf77wr3wv22\\FileStorage\\File\\2022-02\\房东符金瑞名下楼栋需要批量处理.xlsx");
  49.         HttpHeaders headers = new HttpHeaders();
  50.         headers.add("Cache-Control", "no-cache, no-store, must-revalidate");
  51.         headers.add("Content-Disposition", StrUtil.format("attachment; filename={}", URLEncoder.encode(fileSystemResource.getFilename())));
  52.         headers.add("Pragma", "no-cache");
  53.         headers.add("Expires", "0");
  55.         return ResponseEntity.ok()
  56.                 .headers(headers)
  57.                 .contentLength(fileSystemResource.contentLength())
  58.                 .contentType(MediaType.parseMediaType("application/octet-stream"))
  59.                 .body(fileSystemResource);
  60.     }
  62. }

开始测试HtmlUnit下载功能

  1. package work.linruchang.qq;
  3. import cn.hutool.core.collection.CollUtil;
  4. import cn.hutool.core.io.IoUtil;
  5. import cn.hutool.core.lang.Console;
  6. import com.gargoylesoftware.htmlunit.*;
  7. import com.gargoylesoftware.htmlunit.html.*;
  8. import com.gargoylesoftware.htmlunit.javascript.host.event.KeyboardEvent;
  9. import lombok.SneakyThrows;
  10. import org.junit.Test;
  12. import java.io.FileOutputStream;
  13. import java.io.InputStream;
  14. import java.net.URLDecoder;
  15. import java.util.List;
  16. import java.util.logging.Level;
  18. import static java.nio.charset.StandardCharsets.UTF_8;
  20. /**
  21.  * 作用：
  22.  *
  23.  * @author LinRuChang
  24.  * @version 1.0
  25.  * @date 2022/02/08
  26.  * @since 1.8
  27.  **/
  28. public class HtmlUnitTest {
  30.     @Test
  31.     @SneakyThrows
  32.     public void test13() {
  34.         WebClient webClient = new WebClient();
  35.         webClient.setAjaxController(new NicelyResynchronizingAjaxController());
  36.         webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
  37.         webClient.getOptions().setThrowExceptionOnScriptError(false);
  38.         webClient.getOptions().setCssEnabled(true);
  39.         webClient.getOptions().setJavaScriptEnabled(true);
  40.         webClient.getOptions().setActiveXNative(false);
  42.         HtmlPage page = webClient.getPage("file:///C:/Users/Administrator/Desktop/index5.html");
  43.         //DomElement downloadBtn = page.getElementById("downloadBtn");
  44.         DomElement downloadBtn = page.getElementById("downloadBtn2");
  45.         //触发下载按钮
  46.         Page clickPage = downloadBtn.click();
  48.         //下面两句是等价
  49.         //Page enclosedPage = webClient.getWebWindows().get(webClient.getWebWindows().size() - 1).getEnclosedPage();
  50.         Page enclosedPage = clickPage.getEnclosingWindow().getEnclosedPage();
  52.         InputStream contentAsStream = enclosedPage.getWebResponse().getContentAsStream();
  53.         
  54.         //获取文件名
  55.         String responseHeaderValue = enclosedPage.getWebResponse().getResponseHeaderValue(HttpHeader.CONTENT_DISPOSITION);
  56.         String documentName = responseHeaderValue.split(";")[1].split("=")[1].trim();
  57.         documentName = URLDecoder.decode(documentName);
  58.         Console.log("文件下载成功：{}",documentName);
  59.         
  60.         //存入数据库
  61.         IoUtil.copy(contentAsStream, new FileOutputStream("C:\\Users\\Administrator\\Desktop\\图片\\"+ documentName));
  63.     }
  65. }

3.5 弹框处理

示例页面

  1. <!DOCTYPE html>
  2. <html lang="en">
  4. 	<head>
  5. 		<meta charset="UTF-8">
  6. 		<meta http-equiv="X-UA-Compatible" content="IE=edge">
  7. 		<meta name="viewport" content="width=device-width, initial-scale=1.0">
  8. 		<title>HtmlUnit测试</title>
  9. 	</head>
  11. 	<body>
  13. 		<form id="form" onclick="return false;">
  14. 			<div class="container">
  16. 				<input type="hidden" placeholder="Enter Username" name="mark" id="mark" required value="ajax手动提交">
  17. 				<label for="uname"><b>账号</b></label>
  18. 				<input type="text" placeholder="Enter Username" name="uname" id="uname" required>
  20. 				<label for="psw"><b>密码</b></label>
  21. 				<input type="password" placeholder="Enter Password" name="psw" id="psw" required>
  23. 				<button id="loginBtn" type="button">登陆</button>
  25. 			</div>
  27. 		</form>
  29. 		<form id="form2" method="post" action="http://127.0.0.1:8080/login">
  30. 			<div class="container">
  31. 				<input type="hidden" placeholder="Enter Username" name="mark" id="mark2" required value="form表单提交">
  32. 				<label for="uname"><b>账号2</b></label>
  33. 				<input type="text" placeholder="Enter Username" name="uname" id="uname2" required>
  35. 				<label for="psw"><b>密码2</b></label>
  36. 				<input type="password" placeholder="Enter Password" name="psw" id="psw2" required>
  38. 				<button id="loginBtn2" type="submit">登陆2</button>
  39. 			</div>
  40. 		</form>
  41. 		
  42. 		
  43. 		
  44. 		<a href="http://127.0.0.1:8080/download" id="downloadBtn">下载按钮（当前页面）</a>
  45. 		<br/>
  46. 		<a href="http://127.0.0.1:8080/download" id="downloadBtn2" target="_blank">下载按钮2（新页面）</a>
  48. 		<br/>
  49. 		<button id="alertBtn">弹出信息</button>
  50. 		
  51. 		<br/>
  52. 		<button id="promptBtn">提示框信息</button>
  53. 		
  54. 		<br/>
  55. 		<button id="confirmBtn">确认框信息</button>		
  56. 		
  57. 	</body>
  59. 	<script src="file:///G:/VsCode/开源/jquery-3.5.1/jquery-3.5.1.min.js"></script>
  60. 	<script>
  61. 		$(function() {
  62. 			var i = 0;
  63. 			$("#alertBtn").click(function() {
  64. 				alert("点击触发弹框信息: 第" + ++i + "次")
  65. 			})
  66. 			
  67. 			var j = 0;
  68. 			$("#promptBtn").click(function() {
  69. 				prompt("点击触发提示框信息: 第" + ++j + "次", "默认值1111")
  70. 			})
  71. 			
  72. 			
  73. 			var k = 0;
  74. 			$("#confirmBtn").click(function() {
  75. 				confirm("点击触发确认框信息: 第" + ++k + "次")
  76. 			})			
  77. 			
  78. 			
  79. 			//登陆
  80. 			function loginOperation() {
  81. 				$.post("http://127.0.0.1:8080/login", $("#form").serialize(), responseData => {
  82. 					$("body").append(`<h1>${JSON.stringify(responseData)}</h1>`)
  83. 					$("form").hide();
  84. 				}, "json")
  86. 				return false;
  87. 			}
  89. 			$("#loginBtn").click(loginOperation);
  90. 			
  92. 		})
  93. 	</script>
  95. </html>

HtmlUnit模拟用户触发弹框

  1. @Test
  2.     @SneakyThrows
  3.     public void test15() {
  5.         WebClient webClient = new WebClient();
  6.         webClient.setAjaxController(new NicelyResynchronizingAjaxController());
  7.         webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
  8.         webClient.getOptions().setThrowExceptionOnScriptError(false);
  9.         webClient.getOptions().setCssEnabled(true);
  10.         webClient.getOptions().setJavaScriptEnabled(true);
  11.         webClient.getOptions().setActiveXNative(false);
  13.         List<String> alertInfos = new ArrayList<>();
  14.         webClient.setAlertHandler(new CollectingAlertHandler(alertInfos));
  15.         
  16.         //提示框处理
  17.         final List<String> promptInfos = new ArrayList<>();
  18.         webClient.setPromptHandler(new PromptHandler() {
  19.             @Override
  20.             public String handlePrompt(Page page, String message, String defaultValue) {
  21.                 Console.log("Prompt信息：{}、{}", message,defaultValue);
  22.                 promptInfos.add(message);
  23.                 return StrUtil.blankToDefault(message,defaultValue);
  24.             }
  25.         });
  27.         //确认框消息处理
  28.         final List<String> confirmInfos = new ArrayList<>();
  29.         webClient.setConfirmHandler(new ConfirmHandler() {
  30.             @Override
  31.             public boolean handleConfirm(Page page, String message) {
  32.                 confirmInfos.add(message);
  33.                 //true确认 false取消弹框
  34.                 return true;
  35.             }
  36.         });
  39.         HtmlPage page = webClient.getPage("file:///C:/Users/Administrator/Desktop/index5.html");
  40.         DomElement alertBtn = page.getElementById("alertBtn");
  41.         page = alertBtn.click();
  42.         
  44.         DomElement promptBtn = page.getElementById("promptBtn");
  45.         page = promptBtn.click();
  46.         page = promptBtn.click();
  48.         DomElement confirmBtn = page.getElementById("confirmBtn");
  49.         page = confirmBtn.click();
  50.         page = confirmBtn.click();
  51.         page = confirmBtn.click();
  53.         Console.log("弹框信息：{}", alertInfos);
  54.         Console.log("提示框信息：{}", promptInfos);
  55.         Console.log("确认框信息：{}", confirmInfos);
  57.     }

版权说明 : 本文为转载文章, 版权归原作者所有 版权申明

原文链接 : https://blog.csdn.net/weixin_39651356/article/details/122871212

内容来源于网络，如有侵权，请联系作者删除！

JavaJavaScripthtmlunit无界面浏览器

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com