Downloader Middleware¶
The downloader middleware is a framework of hooks into Scrapy’s request/response processing. It’s a light, low-level system for globally altering Scrapy’s requests and responses.
Activating a downloader middleware¶
To activate a downloader middleware component, add it to the
DOWNLOADER_MIDDLEWARES
setting, which is a dict whose keys are the
middleware class paths and their values are the middleware orders.
Here’s an example:
DOWNLOADER_MIDDLEWARES = {
'myproject.middlewares.CustomDownloaderMiddleware': 543,
}
The DOWNLOADER_MIDDLEWARES
setting is merged with the
DOWNLOADER_MIDDLEWARES_BASE
setting defined in Scrapy (and not meant to
be overridden) and then sorted by order to get the final sorted list of enabled
middlewares: the first middleware is the one closer to the engine and the last
is the one closer to the downloader.
To decide which order to assign to your middleware see the
DOWNLOADER_MIDDLEWARES_BASE
setting and pick a value according to
where you want to insert the middleware. The order does matter because each
middleware performs a different action and your middleware could depend on some
previous (or subsequent) middleware being applied.
If you want to disable a built-in middleware (the ones defined in
DOWNLOADER_MIDDLEWARES_BASE
and enabled by default) you must define it
in your project’s DOWNLOADER_MIDDLEWARES
setting and assign None
as its value. For example, if you want to disable the off-site middleware:
DOWNLOADER_MIDDLEWARES = {
'myproject.middlewares.CustomDownloaderMiddleware': 543,
'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
}
Finally, keep in mind that some middlewares may need to be enabled through a particular setting. See each middleware documentation for more info.
Writing your own downloader middleware¶
Writing your own downloader middleware is easy. Each middleware component is a single Python class that defines one or more of the following methods:
-
class
scrapy.contrib.downloadermiddleware.
DownloaderMiddleware
¶ -
process_request
(request, spider)¶ This method is called for each request that goes through the download middleware.
process_request()
should return eitherNone
, aResponse
object, or aRequest
object.If it returns
None
, Scrapy will continue processing this request, executing all other middlewares until, finally, the appropriate downloader handler is called the request performed (and its response downloaded).If it returns a
Response
object, Scrapy won’t bother calling ANY other request or exception middleware, or the appropriate download function; it’ll return that Response. Response middleware is always called on every Response.If it returns a
Request
object, the returned request will be rescheduled (in the Scheduler) to be downloaded in the future. The callback of the original request will always be called. If the new request has a callback it will be called with the response downloaded, and the output of that callback will then be passed to the original callback. If the new request doesn’t have a callback, the response downloaded will be just passed to the original request callback.If it returns an
IgnoreRequest
exception, the entire request will be dropped completely and its callback never called.Parameters: - request (
Request
object) – the request being processed - spider (
BaseSpider
object) – the spider for which this request is intended
- request (
-
process_response
(request, response, spider)¶ process_response()
should return aResponse
object or raise aIgnoreRequest
exception.If it returns a
Response
(it could be the same given response, or a brand-new one), that response will continue to be processed with theprocess_response()
of the next middleware in the pipeline.If it returns an
IgnoreRequest
exception, the response will be dropped completely and its callback never called.Parameters: - request (is a
Request
object) – the request that originated the response - reponse – the response being processed
- spider (
BaseSpider
object) – the spider for which this response is intended
- request (is a
-
process_exception
(request, exception, spider)¶ Scrapy calls
process_exception()
when a download handler or aprocess_request()
(from a downloader middleware) raises an exception.process_exception()
should return eitherNone
,Response
orRequest
object.If it returns
None
, Scrapy will continue processing this exception, executing any other exception middleware, until no middleware is left and the default exception handling kicks in.If it returns a
Response
object, the response middleware kicks in, and won’t bother calling any other exception middleware.If it returns a
Request
object, the returned request is used to instruct an immediate redirection. The original request won’t finish until the redirected request is completed. This stops theprocess_exception()
middleware the same as returning Response would do.Parameters: - request (is a
Request
object) – the request that generated the exception - exception (an
Exception
object) – the raised exception - spider (
BaseSpider
object) – the spider for which this request is intended
- request (is a
-
Built-in downloader middleware reference¶
This page describes all downloader middleware components that come with Scrapy. For information on how to use them and how to write your own downloader middleware, see the downloader middleware usage guide.
For a list of the components enabled by default (and their orders) see the
DOWNLOADER_MIDDLEWARES_BASE
setting.
CookiesMiddleware¶
This middleware enables working with sites that require cookies, such as those that use sessions. It keeps track of cookies sent by web servers, and send them back on subsequent requests (from that spider), just like web browsers do.
The following settings can be used to configure the cookie middleware:
COOKIES_ENABLED¶
Default: True
Whether to enable the cookies middleware. If disabled, no cookies will be sent to web servers.
COOKIES_DEBUG¶
Default: False
If enabled, Scrapy will log all cookies sent in requests (ie. Cookie
header) and all cookies received in responses (ie. Set-Cookie
header).
Here’s an example of a log with COOKIES_DEBUG
enabled:
2011-04-06 14:35:10-0300 [diningcity] INFO: Spider opened
2011-04-06 14:35:10-0300 [diningcity] DEBUG: Sending cookies to: <GET http://www.diningcity.com/netherlands/index.html>
Cookie: clientlanguage_nl=en_EN
2011-04-06 14:35:14-0300 [diningcity] DEBUG: Received cookies from: <200 http://www.diningcity.com/netherlands/index.html>
Set-Cookie: JSESSIONID=B~FA4DC0C496C8762AE4F1A620EAB34F38; Path=/
Set-Cookie: ip_isocode=US
Set-Cookie: clientlanguage_nl=en_EN; Expires=Thu, 07-Apr-2011 21:21:34 GMT; Path=/
2011-04-06 14:49:50-0300 [diningcity] DEBUG: Crawled (200) <GET http://www.diningcity.com/netherlands/index.html> (referer: None)
[...]
DefaultHeadersMiddleware¶
-
class
scrapy.contrib.downloadermiddleware.defaultheaders.
DefaultHeadersMiddleware
¶ This middleware sets all default requests headers specified in the
DEFAULT_REQUEST_HEADERS
setting.
DownloadTimeoutMiddleware¶
-
class
scrapy.contrib.downloadermiddleware.downloadtimeout.
DownloadTimeoutMiddleware
¶ This middleware sets the download timeout for requests specified in the
DOWNLOAD_TIMEOUT
setting.
HttpAuthMiddleware¶
-
class
scrapy.contrib.downloadermiddleware.httpauth.
HttpAuthMiddleware
¶ This middleware authenticates all requests generated from certain spiders using Basic access authentication (aka. HTTP auth).
To enable HTTP authentication from certain spiders, set the
http_user
andhttp_pass
attributes of those spiders.Example:
class SomeIntranetSiteSpider(CrawlSpider): http_user = 'someuser' http_pass = 'somepass' name = 'intranet.example.com' # .. rest of the spider code omitted ...
HttpCacheMiddleware¶
-
class
scrapy.contrib.downloadermiddleware.httpcache.
HttpCacheMiddleware
¶ This middleware provides low-level cache to all HTTP requests and responses. Every request and its corresponding response are cached. When the same request is seen again, the response is returned without transferring anything from the Internet.
The HTTP cache is useful for testing spiders faster (without having to wait for downloads every time) and for trying your spider offline, when an Internet connection is not available.
Scrapy ships with two storage backends for the HTTP cache middleware:
You can change the storage backend with the
HTTPCACHE_STORAGE
setting. Or you can also implement your own backend.
File system backend (default)¶
By default, the HttpCacheMiddleware
uses a file system storage with the following structure:
Each request/response pair is stored in a different directory containing the following files:
request_body
- the plain request bodyrequest_headers
- the request headers (in raw HTTP format)response_body
- the plain response bodyresponse_headers
- the request headers (in raw HTTP format)meta
- some metadata of this cache resource in Pythonrepr()
format (grep-friendly format)pickled_meta
- the same metadata inmeta
but pickled for more efficient deserialization
The directory name is made from the request fingerprint (see
scrapy.utils.request.fingerprint
), and one level of subdirectories is
used to avoid creating too many files into the same directory (which is
inefficient in many file systems). An example directory could be:
/path/to/cache/dir/example.com/72/72811f648e718090f041317756c03adb0ada46c7
DBM storage backend¶
New in version 0.13.
A DBM storage backend is also available for the HTTP cache middleware. To use
it (instead of the default filesystem backend) set HTTPCACHE_STORAGE
to scrapy.contrib.httpcache.DbmCacheStorage
.
By default, it uses the anydbm module, but you can change it with the
HTTPCACHE_DBM_MODULE
setting.
HTTPCache middleware settings¶
The HttpCacheMiddleware
can be configured through the following
settings:
HTTPCACHE_ENABLED¶
New in version 0.11.
Default: False
Whether the HTTP cache will be enabled.
Changed in version 0.11: Before 0.11, HTTPCACHE_DIR
was used to enable cache.
HTTPCACHE_EXPIRATION_SECS¶
Default: 0
Expiration time for cached requests, in seconds.
Cached requests older than this time will be re-downloaded. If zero, cached requests will never expire.
Changed in version 0.11: Before 0.11, zero meant cached requests always expire.
HTTPCACHE_DIR¶
Default: 'httpcache'
The directory to use for storing the (low-level) HTTP cache. If empty, the HTTP cache will be disabled. If a relative path is given, is taken relative to the project data dir. For more info see: Default structure of Scrapy projects.
HTTPCACHE_IGNORE_HTTP_CODES¶
New in version 0.10.
Default: []
Don’t cache response with these HTTP codes.
HTTPCACHE_IGNORE_MISSING¶
Default: False
If enabled, requests not found in the cache will be ignored instead of downloaded.
HTTPCACHE_IGNORE_SCHEMES¶
New in version 0.10.
Default: ['file']
Don’t cache responses with these URI schemes.
HTTPCACHE_STORAGE¶
Default: 'scrapy.contrib.downloadermiddleware.httpcache.FilesystemCacheStorage'
The class which implements the cache storage backend.
HTTPCACHE_DBM_MODULE¶
New in version 0.13.
Default: 'anydbm'
The database module to use in the DBM storage backend. This setting is specific to the DBM backend.
HttpCompressionMiddleware¶
-
class
scrapy.contrib.downloadermiddleware.httpcompression.
HttpCompressionMiddleware
¶ This middleware allows compressed (gzip, deflate) traffic to be sent/received from web sites.
ChunkedTransferMiddleware¶
-
class
scrapy.contrib.downloadermiddleware.chunked.
ChunkedTransferMiddleware
¶ This middleware adds support for chunked transfer encoding
HttpProxyMiddleware¶
New in version 0.8.
-
class
scrapy.contrib.downloadermiddleware.httpproxy.
HttpProxyMiddleware
¶ This middleware sets the HTTP proxy to use for requests, by setting the
proxy
meta value toRequest
objects.Like the Python standard library modules urllib and urllib2, it obeys the following enviroment variables:
http_proxy
https_proxy
no_proxy
RedirectMiddleware¶
-
class
scrapy.contrib.downloadermiddleware.redirect.
RedirectMiddleware
¶ This middleware handles redirection of requests based on response status and meta-refresh html tag.
The urls which the request goes through (while being redirected) can be found
in the redirect_urls
Request.meta
key.
The RedirectMiddleware
can be configured through the following
settings (see the settings documentation for more info):
If Request.meta
contains the
dont_redirect
key, the request will be ignored by this middleware.
RedirectMiddleware settings¶
REDIRECT_ENABLED¶
New in version 0.13.
Default: True
Whether the Redirect middleware will be enabled.
REDIRECT_MAX_TIMES¶
Default: 20
The maximum number of redirections that will be follow for a single request.
REDIRECT_MAX_METAREFRESH_DELAY¶
Default: 100
The maximum meta-refresh delay (in seconds) to follow the redirection.
RetryMiddleware¶
-
class
scrapy.contrib.downloadermiddleware.retry.
RetryMiddleware
¶ A middlware to retry failed requests that are potentially caused by temporary problems such as a connection timeout or HTTP 500 error.
Failed pages are collected on the scraping process and rescheduled at the end, once the spider has finished crawling all regular (non failed) pages. Once there are no more failed pages to retry, this middleware sends a signal (retry_complete), so other extensions could connect to that signal.
The RetryMiddleware
can be configured through the following
settings (see the settings documentation for more info):
About HTTP errors to consider:
You may want to remove 400 from RETRY_HTTP_CODES
, if you stick to the
HTTP protocol. It’s included by default because it’s a common code used
to indicate server overload, which would be something we want to retry.
If Request.meta
contains the dont_retry
key, the request will be ignored by this middleware.
RobotsTxtMiddleware¶
-
class
scrapy.contrib.downloadermiddleware.robotstxt.
RobotsTxtMiddleware
¶ This middleware filters out requests forbidden by the robots.txt exclusion standard.
To make sure Scrapy respects robots.txt make sure the middleware is enabled and the
ROBOTSTXT_OBEY
setting is enabled.Warning
Keep in mind that, if you crawl using multiple concurrent requests per domain, Scrapy could still download some forbidden pages if they were requested before the robots.txt file was downloaded. This is a known limitation of the current robots.txt middleware and will be fixed in the future.
DownloaderStats¶
-
class
scrapy.contrib.downloadermiddleware.stats.
DownloaderStats
¶ Middleware that stores stats of all requests, responses and exceptions that pass through it.
To use this middleware you must enable the
DOWNLOADER_STATS
setting.