Examples

The best way to learn is with examples, and Scrapy is no exception. For this reason, there is an example Scrapy project named dirbot, that you can use to play and learn more about Scrapy. It contains the dmoz spider described in the tutorial.

This dirbot project is available at: https://github.com/scrapy/dirbot

It contains a README file with a detailed description of the project contents.

If you’re familiar with git, you can checkout the code. Otherwise you can download a tarball or zip file of the project by clicking on Downloads.

There is also a site for sharing code snippets such as spiders, middlewares, extensions, or scripts, called Scrapy snippets. Feel free to share any code there.