com.zyd.blog.spider.webmagic.ZhydSpider类的使用及代码示例

x33g5p2x  于2022-02-05 转载在 其他  
字(1.6k)|赞(0)|评价(0)|浏览(65)

本文整理了Java中com.zyd.blog.spider.webmagic.ZhydSpider类的一些代码示例,展示了ZhydSpider类的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。ZhydSpider类的具体详情如下:
包路径:com.zyd.blog.spider.webmagic.ZhydSpider
类名称:ZhydSpider

ZhydSpider介绍

暂无

代码示例

代码示例来源:origin: zhangyd-c/OneBlog

@Override
protected void onSuccess(Request request) {
  super.onSuccess(request);
  if (this.getStatus() == Spider.Status.Running && ExitWayEnum.DURATION.toString().equals(model.getExitWay())) {
    if (startTime < System.currentTimeMillis()) {
      this.stop();
    }
  }
}

代码示例来源:origin: zhangyd-c/OneBlog

public static ZhydSpider create(PageProcessor pageProcessor, BaseModel model, Long uuid) {
  return new ZhydSpider(pageProcessor, model, uuid);
}

代码示例来源:origin: zhangyd-c/OneBlog

ZhydSpider spider = ZhydSpider.create(new ArticleSpiderProcessor(), model, uuid);
spider.addUrl(model.getEntryUrls())
    .setScheduler(new BlockingQueueScheduler(model))
    .addPipeline((resultItems, task) -> process(resultItems, virtualArticles, spider))
  SimpleProxyProvider provider = SimpleProxyProvider.from(model.getProxyList().toArray(new Proxy[0]));
  httpClientDownloader.setProxyProvider(provider);
  spider.setDownloader(httpClientDownloader);
spider.run();
return virtualArticles;

代码示例来源:origin: zhangyd-c/OneBlog

@Override
public void stop() {
  ZhydSpider spider = ZhydSpider.SPIDER_BUCKET.get(SessionUtil.getUser().getId());
  if (null != spider) {
    Spider.Status status = spider.getStatus();
    if (status.equals(Spider.Status.Running)) {
      spider.stop();
    } else if (status.equals(Spider.Status.Init)) {
      throw new ZhydException("[ crawl ] 爬虫正在初始化!");
    } else {
      throw new ZhydException("[ crawl ] 当前没有正在运行的爬虫!");
    }
  } else {
    throw new ZhydException("[ crawl ] 当前没有正在运行的爬虫!");
  }
}

相关文章