如何在scrapy中捕获并处理各种异常

前言

    使用scrapy进行大型爬取任务的时候(爬取耗时以天为单位),无论主机网速多好,爬完之后总会发现scrapy日志中“item_scraped_count”不等于预先的种子数量,总有一部分种子爬取失败,失败的类型可能有如下图两种(下图为scrapy爬取结束完成时的日志):

如何在scrapy中捕获并处理各种异常

scrapy中常见的异常包括但不限于:download error(蓝色区域), http code 403/500(橙色区域)。

不管是哪种异常,我们都可以参考scrapy自带的retry中间件写法来编写自己的中间件。

正文

     使用IDE,现在scrapy项目中任意一个文件敲上以下代码:

from scrapy.downloadermiddlewares.retry import RetryMiddleware

按住ctrl键,鼠标左键点击RetryMiddleware进入该中间件所在的项目文件的位置,也可以通过查看文件的形式找到该中间件的位置,路径是:site-packages/scrapy/downloadermiddlewares/retry.RetryMiddleware

该中间件的源代码如下:

class RetryMiddleware(object):

  # IOError is raised by the HttpCompression middleware when trying to
  # decompress an empty response
  EXCEPTIONS_TO_RETRY = (defer.TimeoutError, TimeoutError, DNSLookupError,
              ConnectionRefusedError, ConnectionDone, ConnectError,
              ConnectionLost, TCPTimedOutError, ResponseFailed,
              IOError, TunnelError)

  def __init__(self, settings):
    if not settings.getbool('RETRY_ENABLED'):
      raise NotConfigured
    self.max_retry_times = settings.getint('RETRY_TIMES')
    self.retry_http_codes = set(int(x) for x in settings.getlist('RETRY_HTTP_CODES'))
    self.priority_adjust = settings.getint('RETRY_PRIORITY_ADJUST')

  @classmethod
  def from_crawler(cls, crawler):
    return cls(crawler.settings)

  def process_response(self, request, response, spider):
    if request.meta.get('dont_retry', False):
      return response
    if response.status in self.retry_http_codes:
      reason = response_status_message(response.status)
      return self._retry(request, reason, spider) or response
    return response

  def process_exception(self, request, exception, spider):
    if isinstance(exception, self.EXCEPTIONS_TO_RETRY) \
        and not request.meta.get('dont_retry', False):
      return self._retry(request, exception, spider)

  def _retry(self, request, reason, spider):
    retries = request.meta.get('retry_times', 0) + 1

    retry_times = self.max_retry_times

    if 'max_retry_times' in request.meta:
      retry_times = request.meta['max_retry_times']

    stats = spider.crawler.stats
    if retries <= retry_times:
      logger.debug("Retrying %(request)s (failed %(retries)d times): %(reason)s",
             {'request': request, 'retries': retries, 'reason': reason},
             extra={'spider': spider})
      retryreq = request.copy()
      retryreq.meta['retry_times'] = retries
      retryreq.dont_filter = True
      retryreq.priority = request.priority + self.priority_adjust

      if isinstance(reason, Exception):
        reason = global_object_name(reason.__class__)

      stats.inc_value('retry/count')
      stats.inc_value('retry/reason_count/%s' % reason)
      return retryreq
    else:
      stats.inc_value('retry/max_reached')
      logger.debug("Gave up retrying %(request)s (failed %(retries)d times): %(reason)s",
             {'request': request, 'retries': retries, 'reason': reason},
             extra={'spider': spider})

查看源码我们可以发现,对于返回http code的response,该中间件会通过process_response方法来处理,处理办法比较简单,大概是判断response.status是否在定义好的self.retry_http_codes集合中,通过向前查找,这个集合是一个列表,定义在default_settings.py文件中,定义如下:

RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408]

也就是先判断http code是否在这个集合中,如果在,就进入retry的逻辑,不在集合中就直接return response。这样就已经实现对返回http code但异常的response的处理了。

但是对另一种异常的处理方式就不一样了,刚才的异常准确的说是属于HTTP请求error(超时),而另一种异常发生的时候则是如下图这种实实在在的代码异常(不处理的话):

如何在scrapy中捕获并处理各种异常

如何在scrapy中捕获并处理各种异常

扫一扫手机访问