本文整理了Java中com.zyd.blog.spider.webmagic.ZhydSpider
类的一些代码示例,展示了ZhydSpider
类的具体用法。这些代码示例主要来源于Github
/Stackoverflow
/Maven
等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。ZhydSpider
类的具体详情如下:
包路径:com.zyd.blog.spider.webmagic.ZhydSpider
类名称:ZhydSpider
暂无
代码示例来源:origin: zhangyd-c/OneBlog
@Override
protected void onSuccess(Request request) {
super.onSuccess(request);
if (this.getStatus() == Spider.Status.Running && ExitWayEnum.DURATION.toString().equals(model.getExitWay())) {
if (startTime < System.currentTimeMillis()) {
this.stop();
}
}
}
代码示例来源:origin: zhangyd-c/OneBlog
public static ZhydSpider create(PageProcessor pageProcessor, BaseModel model, Long uuid) {
return new ZhydSpider(pageProcessor, model, uuid);
}
代码示例来源:origin: zhangyd-c/OneBlog
ZhydSpider spider = ZhydSpider.create(new ArticleSpiderProcessor(), model, uuid);
spider.addUrl(model.getEntryUrls())
.setScheduler(new BlockingQueueScheduler(model))
.addPipeline((resultItems, task) -> process(resultItems, virtualArticles, spider))
SimpleProxyProvider provider = SimpleProxyProvider.from(model.getProxyList().toArray(new Proxy[0]));
httpClientDownloader.setProxyProvider(provider);
spider.setDownloader(httpClientDownloader);
spider.run();
return virtualArticles;
代码示例来源:origin: zhangyd-c/OneBlog
@Override
public void stop() {
ZhydSpider spider = ZhydSpider.SPIDER_BUCKET.get(SessionUtil.getUser().getId());
if (null != spider) {
Spider.Status status = spider.getStatus();
if (status.equals(Spider.Status.Running)) {
spider.stop();
} else if (status.equals(Spider.Status.Init)) {
throw new ZhydException("[ crawl ] 爬虫正在初始化!");
} else {
throw new ZhydException("[ crawl ] 当前没有正在运行的爬虫!");
}
} else {
throw new ZhydException("[ crawl ] 当前没有正在运行的爬虫!");
}
}
内容来源于网络,如有侵权,请联系作者删除!