Frontera 0.6 documentation¶

Use cases¶

Here are few cases, external crawl frontier can be suitable for:

• URL ordering/queueing isolation from the spider (e.g. distributed cluster of spiders, need of remote management of ordering/queueing),
• URL (meta)data storage is needed (e.g. to demonstrate it’s contents somewhere),
• advanced URL ordering logic is needed, when it’s hard to maintain code within spider/fetcher.

Single process¶

Frontera is instantiated in the same process as fetcher (for example in Scrapy). To achieve that use BACKEND setting set to storage backend subclass of Backend. This run mode is suitable for small number of documents and time non-critical applications.

Distributed spiders¶

Spiders are distributed and backend isn’t. Backend is running in db worker and it’s communicating with spiders using message bus.

1. Use BACKEND in spider processes set to MessageBusBackend
2. In DB worker BACKEND should point to Backend subclasse.
3. Every spider process should have it’s own SPIDER_PARTITION_ID, starting from 0 to SPIDER_FEED_PARTITIONS.
4. Both spiders and workers should have it’s MESSAGE_BUS setting set to the message bus class of your choice, and other implementation depending settings.

This mode is suitable for applications where it’s critical to fetch documents fast, at the same time amount of them is relatively small.

Distributed spiders and backend¶

Spiders and backend are distributed. Backend is divided on two parts: strategy worker and db worker. Strategy worker instances are assigned to their own part of spider log.

1. Use BACKEND in spider processes set to MessageBusBackend
2. In DB and SW workers BACKEND should point to DistributedBackend subclasses. And selected backend have to be configured.
3. Every spider process should have it’s own SPIDER_PARTITION_ID, starting from 0 to SPIDER_FEED_PARTITIONS. Last must be accessible also to all DB worker instances.
4. Every SW worker process should have it’s own SCORING_PARTITION_ID, starting from 0 to SPIDER_LOG_PARTITIONS. Last must be accessible to all SW worker instances.
5. Both spiders and workers should have it’s MESSAGE_BUS setting set to the message bus class of your choice and selected message bus have to be configured.

Only Kafka message bus can be used in this mode out of the box and SQLAlchemy and HBase distributed backends.

This mode is suitable for broad crawling and large amount of pages.

1. Create your spider¶

Create your Scrapy project as you usually do. Enter a directory where you’d like to store your code and then run:

scrapy startproject tutorial

This will create a tutorial directory with the following contents:

tutorial/
    scrapy.cfg
    tutorial/
        __init__.py
        items.py
        pipelines.py
        settings.py
        spiders/
            __init__.py
            ...

These are basically:

• scrapy.cfg: the project configuration file
• tutorial/: the project’s python module, you’ll later import your code from here.
• tutorial/items.py: the project’s items file.
• tutorial/pipelines.py: the project’s pipelines file.
• tutorial/settings.py: the project’s settings file.
• tutorial/spiders/: a directory where you’ll later put your spiders.

2. Install Frontera¶

See Installation Guide.

3. Integrate your spider with the Frontera¶

This article about integration with Scrapy explains this step in detail.

4. Choose your backend¶

Configure frontier settings to use a built-in backend like in-memory BFS:

BACKEND = 'frontera.contrib.backends.memory.BFS'

5. Run the spider¶

Run your Scrapy spider as usual from the command line:

scrapy crawl myspider

And that’s it! You got your spider running integrated with Frontera.

What else?¶

You’ve seen a simple example of how to use Frontera with Scrapy, but this is just the surface. Frontera provides many powerful features for making frontier management easy and efficient, such as:

• Built-in support for database storage for crawled pages.
• Easy built-in integration with Scrapy and any other crawler through its API.
• Two distributed crawling modes with use of ZeroMQ or Kafka and distributed backends.
• Creating different crawling logic/policies defining your own backend.
• Plugging your own request/response altering logic using middlewares.
• Create fake sitemaps and reproduce crawling without crawler with the Graph Manager.
• Record your Scrapy crawls and use it later for frontier testing.
• Logging facility that you can hook on to for catching errors and debug your frontiers.

Prerequisites¶

Here is what services needs to be installed and configured before running Frontera:

• Python 2.7+ or 3.4+
• Scrapy

$ pip install frontera[distributed,zeromq,sql]

Get a spider example code¶

First checkout a GitHub Frontera repository:

$ git clone https://github.com/scrapinghub/frontera.git

There is a general spider example in examples/general-spider folder.

This is a general spider, it does almost nothing except extracting links from downloaded content. It also contains some settings files, please consult settings reference to get more information.

Start cluster¶

First, let’s start ZeroMQ broker.

$ python -m frontera.contrib.messagebus.zeromq.broker

You should see a log output of broker with statistics on messages transmitted.

All further commands have to be made from general-spider root directory.

Second, let’s start DB worker.

$ python -m frontera.worker.db --config frontier.workersettings

You should notice that DB is writing messages to the output. It’s ok if nothing is written in ZeroMQ sockets, because of absence of seed URLs in the system.

There are Spanish (.es zone) internet URLs from DMOZ directory in general spider repository, let’s use them as seeds to bootstrap crawling. Starting the spiders:

$ scrapy crawl general -L INFO -s FRONTERA_SETTINGS=frontier.spider_settings -s SEEDS_SOURCE=seeds_es_smp.txt -s SPIDER_PARTITION_ID=0
$ scrapy crawl general -L INFO -s FRONTERA_SETTINGS=frontier.spider_settings -s SPIDER_PARTITION_ID=1

You should end up with 2 spider processes running. Each should read it’s own Frontera config, and first one is using SEEDS_SOURCE option to read seeds to bootstrap Frontera cluster.

After some time seeds will pass the streams and will be scheduled for downloading by workers. At this moment crawler is bootstrapped. Now you can periodically check DB worker output or metadata table contents to see that there is actual activity.

Things to decide¶

• The speed you want to crawl with,
• number of spider processes (assuming that single spider process gives a maximum of 1200 pages/min),
• number of DB and Strategy worker processes.

Things to setup before you start¶

• Kafka,
• HBase (we recommend 1.0.x and higher),
• DNS Service (recommended but not required).

Things to implement before you start¶

• Crawling strategy
• Spider code

Configuring Kafka¶

Configuring HBase¶

• create a namespace crawler (see HBASE_NAMESPACE),
• make sure Snappy compression is supported natively.

Configuring Frontera¶

Every Frontera component requires it’s own configuration module, but some options are shared, so we recommend to create a common modules and import settings from it in component’s modules.

1. Create a common module and add there:

   from __future__ import absolute_import
   from frontera.settings.default_settings import MIDDLEWARES
   MAX_NEXT_REQUESTS = 512
   SPIDER_FEED_PARTITIONS = 2 # number of spider processes
   SPIDER_LOG_PARTITIONS = 2 # worker instances
   MIDDLEWARES.extend([
       'frontera.contrib.middlewares.domain.DomainMiddleware',
       'frontera.contrib.middlewares.fingerprint.DomainFingerprintMiddleware'
   ])
   
   QUEUE_HOSTNAME_PARTITIONING = True
   KAFKA_LOCATION = 'localhost:9092' # your Kafka broker host:port
   SCORING_TOPIC = 'frontier-scoring'
   URL_FINGERPRINT_FUNCTION='frontera.utils.fingerprint.hostname_local_fingerprint'
   

2. Create workers shared module:

   from __future__ import absolute_import
   from .common import *
   
   BACKEND = 'frontera.contrib.backends.hbase.HBaseBackend'
   
   MAX_NEXT_REQUESTS = 2048
   NEW_BATCH_DELAY = 3.0
   
   HBASE_THRIFT_HOST = 'localhost' # HBase Thrift server host and port
   HBASE_THRIFT_PORT = 9090
   

3. Create DB worker module:

   from __future__ import absolute_import
   from .worker import *
   
   LOGGING_CONFIG='logging-db.conf' # if needed
   

4. Create Strategy worker’s module:

   from __future__ import absolute_import
   from .worker import *
   
   CRAWLING_STRATEGY = '' # path to the crawling strategy class
   LOGGING_CONFIG='logging-sw.conf' # if needed
   

The logging can be configured according to https://docs.python.org/2/library/logging.config.html see the list of loggers.

5. Configure spiders module:

   from __future__ import absolute_import
   from .common import *
   
   BACKEND = 'frontera.contrib.backends.remote.messagebus.MessageBusBackend'
   KAFKA_GET_TIMEOUT = 0.5
   

6. Configure Scrapy settings module. It’s located in Scrapy project folder and referenced in scrapy.cfg. Let’s add there:

from scrapy.settings.default_settings import SPIDER_MIDDLEWARES, DOWNLOADER_MIDDLEWARES

FRONTERA_SETTINGS = ''  # module path to your Frontera spider config module

SCHEDULER = 'frontera.contrib.scrapy.schedulers.frontier.FronteraScheduler'
SPIDER_MIDDLEWARES.update({
    'frontera.contrib.scrapy.middlewares.schedulers.SchedulerSpiderMiddleware': 999,
    'frontera.contrib.scrapy.middlewares.seeds.file.FileSeedLoader': 1,
})
DOWNLOADER_MIDDLEWARES.update({
    'frontera.contrib.scrapy.middlewares.schedulers.SchedulerDownloaderMiddleware': 999,
})

Starting the cluster¶

First, let’s start storage worker:

# start DB worker only for batch generation
$ python -m frontera.worker.db --config [db worker config module] --no-incoming
...
# Then start next one dedicated to spider log processing
$ python -m frontera.worker.db --no-batches --config [db worker config module]

Next, let’s start strategy workers, one process per spider log partition:

$ python -m frontera.worker.strategy --config [strategy worker config] --partition-id 0
$ python -m frontera.worker.strategy --config [strategy worker config] --partition-id 1
...
$ python -m frontera.worker.strategy --config [strategy worker config] --partition-id N

You should notice that all processes are writing messages to the log. It’s ok if nothing is written in streams, because of absence of seed URLs in the system.

Let’s put our seeds in text file, one URL per line and start spiders. A single spider per spider feed partition:

$ scrapy crawl [spider] -L INFO -s SEEDS_SOURCE = 'seeds.txt' -s SPIDER_PARTITION_ID=0
...
$ scrapy crawl [spider] -L INFO -s SPIDER_PARTITION_ID=1
$ scrapy crawl [spider] -L INFO -s SPIDER_PARTITION_ID=2
...
$ scrapy crawl [spider] -L INFO -s SPIDER_PARTITION_ID=N

You should end up with N spider processes running. Usually it’s enough for a single instance to read seeds from SEEDS_SOURCE variable to pass seeds to Frontera cluster. Seeds are only read if spider queue is empty. :SPIDER_PARTITION_ID can be read from config file also.

After some time seeds will pass the streams and will be scheduled for downloading by workers. Crawler is bootstrapped.

Options¶

Each option installs dependencies needed for particular functionality.

• sql - relational database,
• graphs - Graph Manager,
• logging - color logging,
• tldextract - can be used with TLDEXTRACT_DOMAIN_INFO
• hbase - HBase distributed backend,
• zeromq - ZeroMQ message bus,
• kafka - Kafka message bus,
• distributed - workers dependencies.

Request objects¶

class frontera.core.models.Request(url, method='GET', headers=None, cookies=None, meta=None, body='')¶

A Request object represents an HTTP request, which is generated for seeds, extracted page links and next pages to crawl. Each one should be associated to a Response object when crawled.

Parameters:	url (string) – URL to send. method (string) – HTTP method to use. headers (dict) – dictionary of headers to send. cookies (dict) – dictionary of cookies to attach to this request. meta (dict) – dictionary that contains arbitrary metadata for this request, the keys must be bytes and the values must be either bytes or serializable objects such as lists, tuples, dictionaries with byte type items.

body¶

A string representing the request body.

cookies¶

Dictionary of cookies to attach to this request.

headers¶

A dictionary which contains the request headers.

meta¶

A dict that contains arbitrary metadata for this request. This dict is empty for new Requests, and is usually populated by different Frontera components (middlewares, etc). So the data contained in this dict depends on the components you have enabled. The keys are bytes and the values are either bytes or serializable objects such as lists, tuples, dictionaries with byte type items.

method¶

A string representing the HTTP method in the request. This is guaranteed to be uppercase. Example: GET, POST, PUT, etc

url¶

A string containing the URL of this request.

Response objects¶

class frontera.core.models.Response(url, status_code=200, headers=None, body='', request=None)¶

A Response object represents an HTTP response, which is usually downloaded (by the crawler) and sent back to the frontier for processing.

Parameters:	url (string) – URL of this response. status_code (int) – the HTTP status of the response. Defaults to 200. headers (dict) – dictionary of headers to send. body (str) – the response body. request (Request) – The Request object that generated this response.

body¶

A str containing the body of this Response.

headers¶

A dictionary object which contains the response headers.

meta¶

A shortcut to the Request.meta attribute of the Response.request object (ie. self.request.meta).

request¶

The Request object that generated this response.

status_code¶

An integer representing the HTTP status of the response. Example: 200, 404, 500.

url¶

A string containing the URL of the response.

Fields domain and fingerprint are added by built-in middlewares

Identifying unique objects¶

As frontier objects are shared between the crawler and the frontier, some mechanism to uniquely identify objects is needed. This method may vary depending on the frontier logic (in most cases due to the backend used).

By default, Frontera activates the fingerprint middleware to generate a unique fingerprint calculated from the Request.url and Response.url fields, which is added to the Request.meta and Response.meta fields respectively. You can use this middleware or implement your own method to manage frontier objects identification.

An example of a generated fingerprint for a Request object:

>>> request.url
'http://thehackernews.com'

>>> request.meta['fingerprint']
'198d99a8b2284701d6c147174cd69a37a7dea90f'

Adding additional data to objects¶

In most cases frontier objects can be used to represent the information needed to manage the frontier logic/policy.

Also, additional data can be stored by components using the Request.meta and Response.meta fields.

For instance the frontier domain middleware adds a domain info field for every Request.meta and Response.meta if is activated:

>>> request.url
'http://www.scrapinghub.com'

>>> request.meta['domain']
{
    "name": "scrapinghub.com",
    "netloc": "www.scrapinghub.com",
    "scheme": "http",
    "sld": "scrapinghub",
    "subdomain": "www",
    "tld": "com"
}

Activating a middleware¶

To activate a Middleware component, add it to the MIDDLEWARES setting, which is a list whose values can be class paths or instances of Middleware objects.

Here’s an example:

MIDDLEWARES = [
    'frontera.contrib.middlewares.domain.DomainMiddleware',
]

Middlewares are called in the same order they’ve been defined in the list, to decide which order to assign to your middleware pick a value according to where you want to insert it. The order does matter because each middleware performs a different action and your middleware could depend on some previous (or subsequent) middleware being applied.

Finally, keep in mind that some middlewares may need to be enabled through a particular setting. See each middleware documentation for more info.

Writing your own middleware¶

Writing your own frontier middleware is easy. Each Middleware component is a single Python class inherited from Component.

FrontierManager will communicate with all active middlewares through the methods described below.

class frontera.core.components.Middleware¶

Interface definition for a Frontier Middlewares

Methods

frontier_start()¶

Called when the frontier starts, see starting/stopping the frontier.

frontier_stop()¶

Called when the frontier stops, see starting/stopping the frontier.

add_seeds(seeds)¶

This method is called when new seeds are added to the frontier.

Parameters:	seeds (list) – A list of Request objects.
Returns:	Request object list or None

Should either return None or a list of Request objects.

If it returns None, FrontierManager won’t continue processing any other middleware and seed will never reach the Backend.

If it returns a list of Request objects, this will be passed to next middleware. This process will repeat for all active middlewares until result is finally passed to the Backend.

If you want to filter any seed, just don’t include it in the returned object list.

page_crawled(response)¶

This method is called every time a page has been crawled.

Parameters:	response (object) – The Response object for the crawled page.
Returns:	Response or None

Should either return None or a Response object.

If it returns None, FrontierManager won’t continue processing any other middleware and Backend will never be notified.

If it returns a Response object, this will be passed to next middleware. This process will repeat for all active middlewares until result is finally passed to the Backend.

If you want to filter a page, just return None.

request_error(page, error)¶

This method is called each time an error occurs when crawling a page.

Parameters:	request (object) – The crawled with error Request object. error (string) – A string identifier for the error.
Returns:	Request or None

Should either return None or a Request object.

If it returns None, FrontierManager won’t continue processing any other middleware and Backend will never be notified.

If it returns a Response object, this will be passed to next middleware. This process will repeat for all active middlewares until result is finally passed to the Backend.

If you want to filter a page error, just return None.

Class Methods

from_manager(manager)¶

Class method called from FrontierManager passing the manager itself.

Example of usage:

def from_manager(cls, manager):
    return cls(settings=manager.settings)

Built-in middleware reference¶

This page describes all Middleware components that come with Frontera. For information on how to use them and how to write your own middleware, see the middleware usage guide..

For a list of the components enabled by default (and their orders) see the MIDDLEWARES setting.

Built-in canonical URL solvers reference¶

Activating a backend¶

To activate the frontier backend component, set it through the BACKEND setting.

Here’s an example:

BACKEND = 'frontera.contrib.backends.memory.FIFO'

Keep in mind that some backends may need to be additionally configured through a particular setting. See backends documentation for more info.

Writing your own backend¶

Each backend component is a single Python class inherited from Backend or DistributedBackend and using one or all of Queue, Metadata and States.

FrontierManager will communicate with active backend through the methods described below.

class frontera.core.components.Backend¶

Interface definition for frontier backend.

Methods

frontier_start()¶

Called when the frontier starts, see starting/stopping the frontier.

Returns:	None.

frontier_stop()¶

Called when the frontier stops, see starting/stopping the frontier.

Returns:	None.

finished()¶

Quick check if crawling is finished. Called pretty often, please make sure calls are lightweight.

Returns:	boolean

add_seeds(seeds)¶

This method is called when new seeds are added to the frontier.

Parameters:	seeds (list) – A list of Request objects.
Returns:	None.

page_crawled(response)¶

This method is called every time a page has been crawled.

Parameters:	response (object) – The Response object for the crawled page.
Returns:	None.

request_error(page, error)¶

This method is called each time an error occurs when crawling a page.

Parameters:	request (object) – The crawled with error Request object. error (string) – A string identifier for the error.
Returns:	None.

get_next_requests(max_n_requests, **kwargs)¶

Returns a list of next requests to be crawled.

Parameters:	max_next_requests (int) – Maximum number of requests to be returned by this method. kwargs (dict) – A parameters from downloader component.
Returns:	list of Request objects.

Class Methods

from_manager(manager)¶

Class method called from FrontierManager passing the manager itself.

Example of usage:

def from_manager(cls, manager):
    return cls(settings=manager.settings)

Properties

queue¶

Returns:	associated Queue object

states¶

Returns:	associated States object

metadata¶

Returns:	associated Metadata object

class frontera.core.components.DistributedBackend¶

Interface definition for distributed frontier backend. Implies using in strategy worker and DB worker.

Inherits all methods of Backend, and has two more class methods, which are called during strategy and db worker instantiation.

classmethod DistributedBackend.strategy_worker(manager)¶

classmethod DistributedBackend.db_worker(manager)¶

Backend should communicate with low-level storage by means of these classes:

Built-in backend reference¶

This article describes all backend components that come bundled with Frontera.

To know the default activated Backend check the BACKEND setting.

Built-in message bus reference¶

Protocol¶

Depending on stream Frontera is using several message types to code it’s messages. Every message is a python native object serialized using msgpack or JSON. The codec module can be selected using MESSAGE_BUS_CODEC, and it’s required to export Encoder and Decoder classes.

Here are the classes needed to subclass to implement own codec:

class frontera.core.codec.BaseEncoder¶

encode_add_seeds(seeds)¶

Encodes add_seeds message

Parameters:	seeds (list) – A list of frontier Request objects
Returns:	bytes encoded message

encode_page_crawled(response)¶

Encodes a page_crawled message

Parameters:	response (object) – A frontier Response object
Returns:	bytes encoded message

encode_request_error(request, error)¶

Encodes a request_error message

Parameters:	request (object) – A frontier Request object error (string) – Error description
Returns:	bytes encoded message

encode_request(request)¶

Encodes requests for spider feed stream.

Parameters:	request (object) – Frontera Request object
Returns:	bytes encoded message

encode_update_score(request, score, schedule)¶

Encodes update_score messages for scoring log stream.

Parameters:	request (object) – Frontera Request object score (float) – score schedule (bool) – True if document needs to be scheduled for download
Returns:	bytes encoded message

encode_new_job_id(job_id)¶

Encodes changing of job_id parameter.

Parameters:	job_id (int) –
Returns:	bytes encoded message

encode_offset(partition_id, offset)¶

Encodes current spider offset in spider feed.

Parameters:	partition_id (int) – offset (int) –
Returns:	bytes encoded message

class frontera.core.codec.BaseDecoder¶

decode(buffer)¶

Decodes the message.

Parameters:	buffer (bytes) – encoded message
Returns:	tuple of message type and related objects

decode_request(buffer)¶

Decodes Request objects.

Parameters:	buffer (bytes) – serialized string
Returns:	object Request

Available codecs¶

Activating the frontier¶

The Frontera uses 2 different middlewares: SchedulerSpiderMiddleware and SchedulerDownloaderMiddleware, and it’s own scheduler FronteraScheduler.

To activate the Frontera in your Scrapy project, just add them to the SPIDER_MIDDLEWARES, DOWNLOADER_MIDDLEWARES and SCHEDULER settings:

SPIDER_MIDDLEWARES.update({
    'frontera.contrib.scrapy.middlewares.schedulers.SchedulerSpiderMiddleware': 1000,
})

DOWNLOADER_MIDDLEWARES.update({
    'frontera.contrib.scrapy.middlewares.schedulers.SchedulerDownloaderMiddleware': 1000,
})

SCHEDULER = 'frontera.contrib.scrapy.schedulers.frontier.FronteraScheduler'

Create a Frontera settings.py file and add it to your Scrapy settings:

FRONTERA_SETTINGS = 'tutorial.frontera.settings'

Another option is to put these settings right into Scrapy settings module.

Organizing files¶

When using frontier with a Scrapy project, we propose the following directory structure:

my_scrapy_project/
    my_scrapy_project/
        frontera/
            __init__.py
            settings.py
            middlewares.py
            backends.py
        spiders/
            ...
        __init__.py
        settings.py
     scrapy.cfg

These are basically:

• my_scrapy_project/frontera/settings.py: the Frontera settings file.
• my_scrapy_project/frontera/middlewares.py: the middlewares used by the Frontera.
• my_scrapy_project/frontera/backends.py: the backend(s) used by the Frontera.
• my_scrapy_project/spiders: the Scrapy spiders folder
• my_scrapy_project/settings.py: the Scrapy settings file
• scrapy.cfg: the Scrapy config file

Running the сrawl¶

Just run your Scrapy spider as usual from the command line:

scrapy crawl myspider

Frontier Scrapy settings¶

You can configure your frontier two ways:

• Using FRONTERA_SETTINGS parameter, which is a module path pointing to Frontera settings in Scrapy settings file. Defaults to None
• Define frontier settings right into Scrapy settings file.

Writing Scrapy spider¶

@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
    spider = cls(*args, **kwargs)
    spider._set_crawler(crawler)
    spider.crawler.signals.connect(spider.spider_idle, signal=signals.spider_idle)
    return spider

def spider_idle(self):
    self.log("Spider idle signal caught.")
    raise DontCloseSpider

SPIDER_MIDDLEWARES.update({
    'frontera.contrib.scrapy.middlewares.seeds.file.FileSeedLoader': 1,
})

HTTPCACHE_ENABLED = False   # Turns off disk cache, which has low hit ratio during broad crawls
REDIRECT_ENABLED = True
COOKIES_ENABLED = False
DOWNLOAD_TIMEOUT = 120
RETRY_ENABLED = False   # Retries can be handled by Frontera itself, depending on crawling strategy
DOWNLOAD_MAXSIZE = 10 * 1024 * 1024  # Maximum document size, causes OOM kills if not set
LOGSTATS_INTERVAL = 10  # Print stats every 10 secs to console

# auto throttling
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_DEBUG = False
AUTOTHROTTLE_MAX_DELAY = 3.0
AUTOTHROTTLE_START_DELAY = 0.25     # Any small enough value, it will be adjusted during operation by averaging
                                    # with response latencies.
RANDOMIZE_DOWNLOAD_DELAY = False

# concurrency
CONCURRENT_REQUESTS = 256           # Depends on many factors, and should be determined experimentally
CONCURRENT_REQUESTS_PER_DOMAIN = 10
DOWNLOAD_DELAY = 0.0

Scrapy Seed Loaders¶

Frontera has some built-in Scrapy middlewares for seed loading.

Seed loaders use the process_start_requests method to generate requests from a source that are added later to the FrontierManager.

SPIDER_MIDDLEWARES.update({
    'frontera.contrib.scrapy.middlewares.seeds.FileSeedLoader': 650
})

http://www.asite.com
http://www.anothersite.com
...

...
#http://www.acommentedsite.com
...

Designating the settings¶

When you use Frontera, you have to tell it which settings you’re using. As FrontierManager is the main entry point to Frontier usage, you can do this by using the method described in the Loading from settings section.

When using a string path pointing to a settings file for the frontier we propose the following directory structure:

my_project/
    frontier/
        __init__.py
        settings.py
        middlewares.py
        backends.py
    ...

These are basically:

• frontier/settings.py: the frontier settings file.
• frontier/middlewares.py: the middlewares used by the frontier.
• frontier/backends.py: the backend(s) used by the frontier.

How to access settings¶

Settings can be accessed through the FrontierManager.settings attribute, that is passed to Middleware.from_manager and Backend.from_manager class methods:

class MyMiddleware(Component):

    @classmethod
    def from_manager(cls, manager):
        manager = crawler.settings
        if settings.TEST_MODE:
            print "test mode is enabled!"

In other words, settings can be accessed as attributes of the Settings object.

Settings class¶

class frontera.settings.Settings(module=None, attributes=None)¶

Built-in frontier settings¶

Here’s a list of all available Frontera settings, in alphabetical order, along with their default values and the scope where they apply.

[
    'frontera.contrib.middlewares.fingerprint.UrlFingerprintMiddleware',
]

Built-in fingerprint middleware settings¶

Settings used by the UrlFingerprintMiddleware and DomainFingerprintMiddleware.

Built-in backends settings¶

{
    'MetadataModel': 'frontera.contrib.backends.sqlalchemy.models.MetadataModel',
    'StateModel': 'frontera.contrib.backends.sqlalchemy.models.StateModel',
    'QueueModel': 'frontera.contrib.backends.sqlalchemy.models.QueueModel'
}

ZeroMQ message bus settings¶

The message bus class is distributed_frontera.messagebus.zeromq.MessageBus

Kafka message bus settings¶

The message bus class is frontera.contrib.messagebus.kafkabus.MessageBus

Default settings¶

If no settings are specified, frontier will use the built-in default ones. For a complete list of default values see: Built-in settings reference. All default settings can be overridden.

Defining a Site Graph¶

Pages from a web site and its links can be easily defined as a directed graph, where each node represents a page and the edges the links between them.

Let’s use a really simple site representation with a starting page A that have links inside to tree pages B, C, D. We can represent the site with this graph:

We use a list to represent the different site pages and one tuple to define the page and its links, for the previous example:

site = [
    ('A', ['B', 'C', 'D']),
]

Note that we don’t need to define pages without links, but we can also use it as a valid representation:

site = [
    ('A', ['B', 'C', 'D']),
    ('B', []),
    ('C', []),
    ('D', []),
]

A more complex site:

Can be represented as:

site = [
    ('A', ['B', 'C', 'D']),
    ('D', ['A', 'D', 'E', 'F']),
]

Note that D is linking to itself and to his parent A.

In the same way, a page can have several parents:

site = [
    ('A', ['B', 'C', 'D']),
    ('B', ['C']),
    ('D', ['C']),
]

In order to simplify examples we’re not using urls for page representation, but of course urls are the intended use for site graphs:

site = [
    ('http://example.com', ['http://example.com/anotherpage', 'http://othersite.com']),
]

Using the Graph Manager¶

Once we have defined our site represented as a graph, we can start using it with the Graph Manager.

We must first create our graph manager:

>>> from frontera.utils import graphs
>>> g = graphs.Manager()

And add the site using the add_site method:

>>> site = [('A', ['B', 'C', 'D'])]
>>> g.add_site(site)

The manager is now initialized and ready to be used.

We can get all the pages in the graph:

>>> g.pages
[<1:A*>, <2:B>, <3:C>, <4:D>]

Asterisk represents that the page is a seed, if we want to get just the seeds of the site graph:

>>> g.seeds
[<1:A*>]

We can get individual pages using get_page, if a page does not exists None is returned

>>> g.get_page('A')
<1:A*>

>>> g.get_page('F')
None

CrawlPage objects¶

Pages are represented as a CrawlPage object:

class CrawlPage¶

A CrawlPage object represents an Graph Manager page, which is usually generated in the Graph Manager.

id¶

Autonumeric page id.

url¶

The url of the page.

status¶

Represents the HTTP code status of the page.

is_seed¶

Boolean value indicating if the page is seed or not.

links¶

List of pages the current page links to.

referers¶

List of pages that link to the current page.

In our example:

>>> p = g.get_page('A')
>>> p.id
1

>>> p.url
u'A'

>>> p.status  # defaults to 200
u'200'

>>> p.is_seed
True

>>> p.links
[<2:B>, <3:C>, <4:D>]

>>> p.referers  # No referers for A
[]

>>> g.get_page('B').referers  # referers for B
[<1:A*>]

Adding pages and Links¶

Site graphs can be also defined adding pages and links individually, the same graph from our example can be defined this way:

>>> g = graphs.Manager()
>>> a = g.add_page(url='A', is_seed=True)
>>> b = g.add_link(page=a, url='B')
>>> c = g.add_link(page=a, url='C')
>>> d = g.add_link(page=a, url='D')

add_page and add_link can be combined with add_site and used anytime:

>>> site = [('A', ['B', 'C', 'D'])]
>>> g = graphs.Manager()
>>> g.add_site(site)
>>> d = g.get_page('D')
>>> g.add_link(d, 'E')

Adding multiple sites¶

Multiple sites can be added to the manager:

>>> site1 = [('A1', ['B1', 'C1', 'D1'])]
>>> site2 = [('A2', ['B2', 'C2', 'D2'])]

>>> g = graphs.Manager()
>>> g.add_site(site1)
>>> g.add_site(site2)

>>> g.pages
[<1:A1*>, <2:B1>, <3:C1>, <4:D1>, <5:A2*>, <6:B2>, <7:C2>, <8:D2>]

>>> g.seeds
[<1:A1*>, <5:A2*>]

Or as a list of sites with add_site_list method:

>>> site_list = [
    [('A1', ['B1', 'C1', 'D1'])],
    [('A2', ['B2', 'C2', 'D2'])],
]
>>> g = graphs.Manager()
>>> g.add_site_list(site_list)

Graphs Database¶

Graph Manager uses SQLAlchemy to store and represent graphs.

By default it uses an in-memory SQLite database as a storage engine, but any databases supported by SQLAlchemy can be used.

An example using SQLite:

>>> g = graphs.Manager(engine='sqlite:///graph.db')

Changes are committed with every new add by default, graphs can be loaded later:

>>> graph = graphs.Manager(engine='sqlite:///graph.db')
>>> graph.add_site(('A', []))

>>> another_graph = graphs.Manager(engine='sqlite:///graph.db')
>>> another_graph.pages
[<1:A1*>]

A database content reset can be done using clear_content parameter:

>>> g = graphs.Manager(engine='sqlite:///graph.db', clear_content=True)

Using graphs with status codes¶

In order to recreate/simulate crawling using graphs, HTTP response codes can be defined for each page.

Example for a 404 error:

>>> g = graphs.Manager()
>>> g.add_page(url='A', status=404)

Status codes can be defined for sites in the following way using a list of tuples:

>>> site_with_status_codes = [
    ((200, "A"), ["B", "C"]),
    ((404, "B"), ["D", "E"]),
    ((500, "C"), ["F", "G"]),
]
>>> g = graphs.Manager()
>>> g.add_site(site_with_status_codes)

Default status code value is 200 for new pages.

A simple crawl faking example¶

Frontier tests can better be done using the Frontier Tester tool, but here’s an example of how fake a crawl with a frontier:

from frontera import FrontierManager, Request, Response
from frontera.utils import graphs

if __name__ == '__main__':
    # Load graph from existing database
    graph = graphs.Manager('sqlite:///graph.db')

    # Create frontier from default settings
    frontier = FrontierManager.from_settings()

    # Create and add seeds
    seeds = [Request(seed.url) for seed in graph.seeds]
    frontier.add_seeds(seeds)

    # Get next requests
    next_requets = frontier.get_next_requests()

    # Crawl pages
    while (next_requests):
        for request in next_requests:

            # Fake page crawling
            crawled_page = graph.get_page(request.url)

            # Create response
            response = Response(url=crawled_page.url, status_code=crawled_page.status)

            # Update Page
            page = frontier.page_crawled(response=response
                                         links=[link.url for link in crawled_page.links])
            # Get next requests
            next_requets = frontier.get_next_requests()

Rendering graphs¶

Graphs can be rendered to png files:

>>> g.render(filename='graph.png', label='A simple Graph')

Rendering graphs uses pydot, a Python interface to Graphviz‘s Dot language.

How to use it¶

Graph Manager can be used to test frontiers in conjunction with Frontier Tester and also with Scrapy Recordings.

Activating the recorder¶

The recorder uses 2 different middlewares: CrawlRecorderSpiderMiddleware and CrawlRecorderDownloaderMiddleware.

To activate the recording in your Scrapy project, just add them to the SPIDER_MIDDLEWARES and DOWNLOADER_MIDDLEWARES settings:

SPIDER_MIDDLEWARES.update({
    'frontera.contrib.scrapy.middlewares.recording.CrawlRecorderSpiderMiddleware': 1000,
})

DOWNLOADER_MIDDLEWARES.update({
    'frontera.contrib.scrapy.middlewares.recording.CrawlRecorderDownloaderMiddleware': 1000,
})

Choosing your storage engine¶

As Graph Manager is internally used by the recorder to store crawled pages, you can choose between different storage engines.

We can set the storage engine with the RECORDER_STORAGE_ENGINE setting:

RECORDER_STORAGE_ENGINE = 'sqlite:///my_record.db'

You can also choose to reset database tables or just reset data with this settings:

RECORDER_STORAGE_DROP_ALL_TABLES = True
RECORDER_STORAGE_CLEAR_CONTENT = True

Running the Crawl¶

Just run your Scrapy spider as usual from the command line:

scrapy crawl myspider

Once it’s finished you should have the recording available and ready for use.

In case you need to disable recording, you can do it by overriding the RECORDER_ENABLED setting:

scrapy crawl myspider -s RECORDER_ENABLED=False

Recorder settings¶

Here’s a list of all available Scrapy Recorder settings, in alphabetical order, along with their default values and the scope where they apply.

Why crawling speed is so low?¶

Search for a bottleneck.

• All requests are targeted towards a few websites.
• DNS resolution (see DNS Service article),
• strategy worker performance,
• db worker batch generation insufficiency.
• HBase response times are too high,
• Network within cluster is overloaded.

Tuning HBase¶

• Increase block cache in HBase.
• Put Thrift server on each HBase region server and spread load from SW to Thrift.
• Enable Snappy compression (see HBASE_USE_SNAPPY).

Tuning Kafka¶

• Decrease the log size to minimum and optimize the system to avoid storing in Kafka huge volumes of data. Once data was written it should be consumed as fast as possible.
• Use SSD or even RAM storage for Kafka logs,
• Enable Snappy compression for Kafka.

Flow control between various components¶

The MAX_NEXT_REQUESTS is used for controlling the batch size. In spiders config it controls how much items will be consumed per one get_next_requests call. At the same time in DB worker config it sets count of items to generate per partition. When setting these parameters keep in mind:

• DB worker and spider values have to be consistent to avoid overloading of message bus and loosing messages. In other words, DB worker have to produce slightly more than consumed by spiders, because the spider should still be able to fetch new pages even though the DB worker has not pushed a new batch yet.
• Spider consumption rate depends on many factors: internet connection latency, amount of spider parsing/scraping work, delays and auto throttling settings, usage of proxies, etc.
• Keep spider queue always full to prevent spider idling.
• General recommendation is to set DB worker value 2-4 times bigger than spiders.
• Batch size shouldn’t be big to not generate too much load on backend, and allow system quickly react on queue changes.
• Watch out warnings about lost messages.

Single process¶

The following diagram shows an architecture of the Frontera pipeline with its components (referenced by numbers) and an outline of the data flow that takes place inside the system. A brief description of the components is included below with links for more detailed information about them. The data flow is also described below.

Distributed¶

The same Frontera Manager pipeline is used in all Frontera processes when running in distributed mode.

Overall system forms a closed circle and all the components are working as daemons in infinite cycles. There is a message bus responsible for transmitting messages between components, persistent storage and fetchers (when combined with extraction these processes called spiders). There is a transport and storage layer abstractions, so one can plug it’s own transport. Distributed backend run mode has instances of three types:

• Spiders or fetchers, implemented using Scrapy (sharded).

  Responsible for resolving DNS queries, getting content from the Internet and doing link (or other data) extraction from content.

• Strategy workers (sharded).

  Run the crawling strategy code: scoring the links, deciding if link needs to be scheduled and when to stop crawling.

• DB workers (sharded).

  Store all the metadata, including scores and content, and generating new batches for downloading by spiders.

Where sharded means component consumes messages of assigned partition only, e.g. processes certain share of the stream, and replicated is when components consume stream regardless of partitioning.

Such design allows to operate online. Crawling strategy can be changed without having to stop the crawl. Also crawling strategy can be implemented as a separate module; containing logic for checking the crawling stopping condition, URL ordering, and scoring model.

Frontera is polite to web hosts by design and each host is downloaded by no more than one spider process. This is achieved by stream partitioning.

Frontera API / Manager¶

The main entry point to Frontera API is the FrontierManager object, passed to middlewares and backend through the from_manager class method. This object provides access to all Frontera core components, and is the only way for middlewares and backend to access them and hook their functionality into Frontera.

The FrontierManager is responsible for loading the installed middlewares and backend, as well as for managing the data flow around the whole frontier.

Loading from settings¶

Although FrontierManager can be initialized using parameters the most common way of doing this is using Frontera Settings.

This can be done through the from_settings class method, using either a string path:

>>> from frontera import FrontierManager
>>> frontier = FrontierManager.from_settings('my_project.frontier.settings')

or a BaseSettings object instance:

>>> from frontera import FrontierManager, Settings
>>> settings = Settings()
>>> settings.MAX_PAGES = 0
>>> frontier = FrontierManager.from_settings(settings)

It can also be initialized without parameters, in this case the frontier will use the default settings:

>>> from frontera import FrontierManager, Settings
>>> frontier = FrontierManager.from_settings()

Frontier Manager¶

class frontera.core.manager.FrontierManager(request_model, response_model, backend, middlewares=None, test_mode=False, max_requests=0, max_next_requests=0, auto_start=True, settings=None, canonicalsolver=None, db_worker=False, strategy_worker=False)¶

The FrontierManager object encapsulates the whole frontier, providing an API to interact with. It’s also responsible of loading and communicating all different frontier components.

Parameters:

request_model (object/string) – The Request object to be used by the frontier. response_model (object/string) – The Response object to be used by the frontier. backend (object/string) – The Backend object to be used by the frontier. middlewares (list) – A list of Middleware objects to be used by the frontier. test_mode (bool) – Activate/deactivate frontier test mode. max_requests (int) – Number of pages after which the frontier would stop (See Finish conditions). max_next_requests (int) – Maximum number of requests returned by get_next_requests method. auto_start (bool) – Activate/deactivate automatic frontier start (See starting/stopping the frontier). settings (object/string) – The Settings object used by the frontier. canonicalsolver (object/string) – The CanonicalSolver object to be used by frontier. db_worker (bool) – True if class is instantiated in DB worker environment strategy_worker (bool) – True if class is instantiated in strategy worker environment

Attributes

request_model¶

The Request object to be used by the frontier. Can be defined with REQUEST_MODEL setting.

response_model¶

The Response object to be used by the frontier. Can be defined with RESPONSE_MODEL setting.

backend¶

The Backend object to be used by the frontier. Can be defined with BACKEND setting.

middlewares¶

A list of Middleware objects to be used by the frontier. Can be defined with MIDDLEWARES setting.

test_mode¶

Boolean value indicating if the frontier is using frontier test mode. Can be defined with TEST_MODE setting.

max_requests¶

Number of pages after which the frontier would stop (See Finish conditions). Can be defined with MAX_REQUESTS setting.

max_next_requests¶

Maximum number of requests returned by get_next_requests method. Can be defined with MAX_NEXT_REQUESTS setting.

auto_start¶

Boolean value indicating if automatic frontier start is activated. See starting/stopping the frontier. Can be defined with AUTO_START setting.

settings¶

The Settings object used by the frontier.

iteration¶

Current frontier iteration.

n_requests¶

Number of accumulated requests returned by the frontier.

finished¶

Boolean value indicating if the frontier has finished. See Finish conditions.

API Methods

start()¶

Notifies all the components of the frontier start. Typically used for initializations (See starting/stopping the frontier).

Returns:	None.

stop()¶

Notifies all the components of the frontier stop. Typically used for finalizations (See starting/stopping the frontier).

Returns:	None.

add_seeds(seeds)¶

Adds a list of seed requests (seed URLs) as entry point for the crawl.

Parameters:	seeds (list) – A list of Request objects.
Returns:	None.

get_next_requests(max_next_requests=0, **kwargs)¶

Returns a list of next requests to be crawled. Optionally a maximum number of pages can be passed. If no value is passed, FrontierManager.max_next_requests will be used instead. (MAX_NEXT_REQUESTS setting).

Parameters:	max_next_requests (int) – Maximum number of requests to be returned by this method. kwargs (dict) – Arbitrary arguments that will be passed to backend.
Returns:	list of Request objects.

page_crawled(response)¶

Informs the frontier about the crawl result.

Parameters:	response (object) – The Response object for the crawled page.
Returns:	None.

request_error(request, error)¶

Informs the frontier about a page crawl error. An error identifier must be provided.

Parameters:	request (object) – The crawled with error Request object. error (string) – A string identifier for the error.
Returns:	None.

Class Methods

classmethod from_settings(settings=None, db_worker=False, strategy_worker=False)¶

Returns a FrontierManager instance initialized with the passed settings argument. If no settings is given, frontier default settings are used.

Starting/Stopping the frontier¶

Sometimes, frontier components need to perform initialization and finalization operations. The frontier mechanism to notify the different components of the frontier start and stop is done by the start() and stop() methods respectively.

By default auto_start frontier value is activated, this means that components will be notified once the FrontierManager object is created. If you need to have more fine control of when different components are initialized, deactivate auto_start and manually call frontier API start() and stop() methods.

Frontier iterations¶

Once frontier is running, the usual process is the one described in the data flow section.

Crawler asks the frontier for next pages using the get_next_requests() method. Each time the frontier returns a non empty list of pages (data available), is what we call a frontier iteration.

Current frontier iteration can be accessed using the iteration attribute.

Finishing the frontier¶

Crawl can be finished either by the Crawler or by the Frontera. Frontera will finish when a maximum number of pages is returned. This limit is controlled by the max_requests attribute (MAX_REQUESTS setting).

If max_requests has a value of 0 (default value) the frontier will continue indefinitely.

Once the frontier is finished, no more pages will be returned by the get_next_requests method and finished attribute will be True.

Component objects¶

class frontera.core.components.Component¶

Interface definition for a frontier component The Component object is the base class for frontier Middleware and Backend objects.

FrontierManager communicates with the active components using the hook methods listed below.

Implementations are different for Middleware and Backend objects, therefore methods are not fully described here but in their corresponding section.

Attributes

name¶

The component name

Abstract methods

frontier_start()¶

Called when the frontier starts, see starting/stopping the frontier.

frontier_stop()¶

Called when the frontier stops, see starting/stopping the frontier.

add_seeds(seeds)¶

This method is called when new seeds are added to the frontier.

Parameters:	seeds (list) – A list of Request objects.

page_crawled(response)¶

This method is called every time a page has been crawled.

Parameters:	response (object) – The Response object for the crawled page.

request_error(page, error)¶

This method is called each time an error occurs when crawling a page.

Parameters:	request (object) – The crawled with error Request object. error (string) – A string identifier for the error.

Class Methods

classmethod from_manager(manager)¶

Class method called from FrontierManager passing the manager itself.

Example of usage:

def from_manager(cls, manager):
    return cls(settings=manager.settings)

Test mode¶

In some cases while testing, frontier components need to act in a different way than they usually do (for instance domain middleware accepts non valid URLs like 'A1' or 'B1' when parsing domain urls in test mode).

Components can know if the frontier is in test mode via the boolean test_mode attribute.

Other ways of using the frontier¶

Communication with the frontier can also be done through other mechanisms such as an HTTP API or a queue system. These functionalities are not available for the time being, but hopefully will be included in future versions.

requests¶

A simple script that follow all the links from a site using Requests library.

How to run it:

python links_follower.py

general-spider¶

A simple Scrapy spider that follows all the links from the seeds. Contains configuration files for single process, distributed spider and backends run modes.

See Quick start distributed mode for how to run it.

cluster¶

Is a large scale crawling application for performing broad crawls with number of pages per host limit. It preserves each host state in HBase and uses it when schedule new requests for downloading. Designed for running in distributed backend run mode using HBase.

scrapy_recording¶

A simple script with a spider that follows all the links for a site, recording crawling results.

How to run it:

scrapy crawl recorder

scripts¶

Some sample scripts on how to use different frontier components.

Running tests¶

To run all tests go to the root directory of source code and run:

py.test

Writing tests¶

All functionality (including new features and bug fixes) must include a test case to check that it works as expected, so please include tests for your patches if you want them to get accepted sooner.

Backend testing¶

A base pytest class for Backend testing is provided: BackendTest

class tests.backends.BackendTest¶

A simple pytest base class with helper methods for Backend testing.

get_settings()¶

Returns backend settings

get_frontier()¶

Returns frontierManager object

setup_backend(method)¶

Setup method called before each test method call

teardown_backend(method)¶

Teardown method called after each test method call

Let’s say for instance that you want to test to your backend MyBackend and create a new frontier instance for each test method call, you can define a test class like this:

class TestMyBackend(backends.BackendTest):

    backend_class = 'frontera.contrib.backend.abackend.MyBackend'

    def test_one(self):
        frontier = self.get_frontier()
        ...

    def test_two(self):
        frontier = self.get_frontier()
        ...

    ...

And let’s say too that it uses a database file and you need to clean it before and after each test:

class TestMyBackend(backends.BackendTest):

    backend_class = 'frontera.contrib.backend.abackend.MyBackend'

    def setup_backend(self, method):
        self._delete_test_db()

    def teardown_backend(self, method):
        self._delete_test_db()

    def _delete_test_db(self):
        try:
            os.remove('mytestdb.db')
        except OSError:
            pass

    def test_one(self):
        frontier = self.get_frontier()
        ...

    def test_two(self):
        frontier = self.get_frontier()
        ...

    ...

Testing backend sequences¶

To test Backend crawling sequences you can use the BackendSequenceTest class.

class tests.backends.BackendSequenceTest¶

A pytest base class for testing Backend crawling sequences.

get_sequence(site_list, max_next_requests, downloader_simulator=<frontera.utils.tester.BaseDownloaderSimulator object>, frontier_tester=<class 'frontera.utils.tester.FrontierTester'>)¶

Returns an Frontera iteration sequence from a site list

Parameters:	site_list (list) – A list of sites to use as frontier seeds. max_next_requests (int) – Max next requests for the frontier.

assert_sequence(site_list, expected_sequence, max_next_requests)¶

Asserts that crawling sequence is the one expected

Parameters:	site_list (list) – A list of sites to use as frontier seeds. max_next_requests (int) – Max next requests for the frontier.

BackendSequenceTest class will run a complete crawl of the passed site graphs and return the sequence used by the backend for visiting the different pages.

Let’s say you want to test to a backend that sort pages using alphabetic order. You can define the following test:

class TestAlphabeticSortBackend(backends.BackendSequenceTest):

    backend_class = 'frontera.contrib.backend.abackend.AlphabeticSortBackend'

    SITE_LIST = [
        [
            ('C', []),
            ('B', []),
            ('A', []),
        ],
    ]

    def test_one(self):
        # Check sequence is the expected one
        self.assert_sequence(site_list=self.SITE_LIST,
                             expected_sequence=['A', 'B', 'C'],
                             max_next_requests=0)

    def test_two(self):
        # Get sequence and work with it
        sequence = self.get_sequence(site_list=SITE_LIST,
                            max_next_requests=0)
        assert len(sequence) > 2

    ...

Testing basic algorithms¶

If your backend uses any of the basic algorithms logics, you can just inherit the correponding test base class for each logic and sequences will be automatically tested for it:

from tests import backends


class TestMyBackendFIFO(backends.FIFOBackendTest):
    backend_class = 'frontera.contrib.backends.abackend.MyBackendFIFO'


class TestMyBackendLIFO(backends.LIFOBackendTest):
    backend_class = 'frontera.contrib.backends.abackend.MyBackendLIFO'


class TestMyBackendDFS(backends.DFSBackendTest):
    backend_class = 'frontera.contrib.backends.abackend.MyBackendDFS'


class TestMyBackendBFS(backends.BFSBackendTest):
    backend_class = 'frontera.contrib.backends.abackend.MyBackendBFS'


class TestMyBackendRANDOM(backends.RANDOMBackendTest):
    backend_class = 'frontera.contrib.backends.abackend.MyBackendRANDOM'

Loggers used¶

• kafka
• hbase.backend
• hbase.states
• hbase.queue
• sqlalchemy.revisiting.queue
• sqlalchemy.metadata
• sqlalchemy.states
• sqlalchemy.queue
• offset-fetcher
• messagebus-backend
• cf-server
• db-worker
• strategy-worker
• messagebus.kafka
• memory.queue
• memory.dequequeue
• memory.states
• manager.components
• manager
• frontera.contrib.scrapy.schedulers.FronteraScheduler

Creating a Frontier Tester¶

FrontierTester needs a Graph Manager and a FrontierManager instances:

>>> from frontera import FrontierManager, FrontierTester
>>> from frontera.utils import graphs
>>> graph = graphs.Manager('sqlite:///graph.db')  # Crawl fake data loading
>>> frontier = FrontierManager.from_settings()  # Create frontier from default settings
>>> tester = FrontierTester(frontier, graph)

Running a Test¶

The tester is now initialized, to run the test just call the method run:

>>> tester.run()

When run method is called the tester will:

1. Add all the seeds from the graph.
2. Ask the frontier about next pages.
3. Fake page response and inform the frontier about page crawl and its links.

Steps 1 and 2 are repeated until crawl or frontier ends.

Once the test is finished, the crawling page sequence is available as a list of frontier Request objects.

Test Parameters¶

In some test cases you may want to add all graph pages as seeds, this can be done with the parameter add_all_pages:

>>> tester.run(add_all_pages=True)

Maximum number of returned pages per get_next_requests call can be set using frontier settings, but also can be modified when creating the FrontierTester with the max_next_pages argument:

>>> tester = FrontierTester(frontier, graph, max_next_pages=10)

An example of use¶

A working example using test data from graphs and basic backends:

from frontera import FrontierManager, Settings, FrontierTester, graphs


def test_backend(backend):
    # Graph
    graph = graphs.Manager()
    graph.add_site_list(graphs.data.SITE_LIST_02)

    # Frontier
    settings = Settings()
    settings.BACKEND = backend
    settings.TEST_MODE = True
    frontier = FrontierManager.from_settings(settings)

    # Tester
    tester = FrontierTester(frontier, graph)
    tester.run(add_all_pages=True)

    # Show crawling sequence
    print '-'*40
    print frontier.backend.name
    print '-'*40
    for page in tester.sequence:
        print page.url

if __name__ == '__main__':
    test_backend('frontera.contrib.backends.memory.heapq.FIFO')
    test_backend('frontera.contrib.backends.memory.heapq.LIFO')
    test_backend('frontera.contrib.backends.memory.heapq.BFS')
    test_backend('frontera.contrib.backends.memory.heapq.DFS')

How to download efficiently in parallel?¶

Typically the design of URL ordering implies fetching many URLs from the same domain. If crawling process needs to be polite it has to preserve some delay and rate of requests. From the other side, there are downloaders which can afford downloading many URLs (say 100) at once, in parallel. So, flooding of the URLs from the same domain leads to inefficient waste of downloader connection pool resources.

Here is a short example. Imagine, we have a queue of 10K URLs from many different domains. Our task is to fetch it as fast as possible. During downloading we want to be polite and limit per host RPS. At the same time we have a prioritization which tends to group URLs from the same domain. When crawler will be requesting for batches of URLs to fetch, it will be getting hundreds of URLs from the same host. The downloader will not be able to fetch them quickly because of RPS limit and delay. Therefore, picking top URLs from the queue leeds us to the time waste, because connection pool of downloader most of the time underused.

The solution is to supply Frontera backend with hostname/ip (usually, but not necessary) usage in downloader. We have a keyword arguments in method get_next_requests for passing these stats, to the Frontera backend. Information of any kind can be passed there. This arguments are usually set outside of Frontera, and then passed to CF via FrontierManagerWrapper subclass to backend.