Installation guide¶
This document describes how to install Scrapy on Linux, Windows and Mac OS X.
Requirements¶
- Python 2.5, 2.6, 2.7 (3.x is not yet supported)
- Twisted 2.5.0, 8.0 or above (Windows users: you’ll need to install Zope.Interface and maybe pywin32 because of this Twisted bug)
- w3lib
- lxml or libxml2 (if using libxml2, version 2.6.28 or above is highly recommended)
- simplejson (not required if using Python 2.6 or above)
- pyopenssl (for HTTPS support. Optional, but highly recommended)
Install Python¶
First, you need to install Python, if you haven’t done so already.
Scrapy works with Python 2.5, 2.6 or 2.7, which you can get at http://www.python.org/download/
Install Scrapy¶
There are many ways to install Scrapy. Pick the one you feel more comfortable with.
- Download and install an official release (requires installing dependencies separately)
- Installing with easy_install (automatically installs dependencies)
- Installing with pip (automatically installs dependencies)
Download and install an official release¶
Download Scrapy from the Download page. Scrapy is distributed in two ways: a
source code tarball (for Unix and Mac OS X systems) and a Windows installer
(for Windows). If you downloaded the tarball, you can install it as any Python
package using setup.py:
tar zxf Scrapy-X.X.X.tar.gz
cd Scrapy-X.X.X
python setup.py install
If you downloaded the Windows installer, just run it.
Warning
In Windows, you may need to add the C:\Python25\Scripts (or
C:\Python26\Scripts) folder to the system path by adding that directory
to the PATH environment variable from the Control Panel.
Installing with easy_install¶
You can install Scrapy using setuptools‘s easy_install with:
easy_install -U Scrapy
Platform specific instructions¶
Linux¶
Ubuntu 9.10 or above¶
If you’re running Ubuntu 9.10 (or above), use the official Ubuntu Packages, which already solve all dependencies for you and are continuously updated with the latest bug fixes.
Debian or Ubuntu (9.04 or older)¶
If you’re running Debian Linux, run the following command as root:
apt-get install python-twisted python-libxml2 python-pyopenssl python-simplejson
Then:
easy_install -U w3lib
And then follow the instructions in Install Scrapy.
Arch Linux¶
If you are running Arch Linux, run the following command as root:
pacman -S twisted libxml2 pyopenssl python-simplejson
Then:
easy_install -U w3lib
And then follow the instructions in Install Scrapy.
Other Linux distros¶
The easiest way to install Scrapy on other Linux distros is through
easy_install or pip, which will automatically install Twisted, w3lib
and lxml dependencies.
See Installing with easy_install or Installing with pip
Another way would be to install dependencies, if you know the packages in your distros that meets them. See Requirements.
Mac OS X¶
The easiest way to install Scrapy on Mac is through easy_install or
pip, which will automatically install Twisted, w3lib and lxml dependencies.
Windows¶
The easiest way to install Scrapy on Windows is through easy_install or
pip, which will automatically install Twisted, w3lib and lxml dependencies.