Installation guide¶
This document describes how to install Scrapy on Linux, Windows and Mac OS X.
Requirements¶
- Python 2.5, 2.6, 2.7 (3.x is not yet supported)
- Twisted 2.5.0, 8.0 or above (Windows users: you’ll need to install Zope.Interface and maybe pywin32 because of this Twisted bug)
- lxml or libxml2 (if using libxml2, version 2.6.28 or above is highly recommended)
- simplejson (not required if using Python 2.6 or above)
- pyopenssl (for HTTPS support. Optional, but highly recommended)
Install Python¶
First, you need to install Python, if you haven’t done so already.
Scrapy works with Python 2.5, 2.6 or 2.7, which you can get at http://www.python.org/download/
Install Scrapy¶
There are many ways to install Scrapy. Pick the one you feel more comfortable with.
- Download and install an official release (requires installing dependencies separately)
- Installing with easy_install (automatically installs dependencies)
- Installing with pip (automatically installs dependencies)
Download and install an official release¶
Download Scrapy from the Download page. Scrapy is distributed in two ways: a
source code tarball (for Unix and Mac OS X systems) and a Windows installer
(for Windows). If you downloaded the tarball, you can install it as any Python
package using setup.py
:
tar zxf Scrapy-X.X.X.tar.gz
cd Scrapy-X.X.X
python setup.py install
If you downloaded the Windows installer, just run it.
Warning
In Windows, you may need to add the C:\Python25\Scripts
(or
C:\Python26\Scripts
) folder to the system path by adding that directory
to the PATH
environment variable from the Control Panel.
Installing with easy_install
¶
You can install Scrapy using setuptools‘s easy_install
with:
easy_install -U Scrapy
Platform specific instructions¶
Linux¶
Ubuntu 9.10 or above¶
If you’re running Ubuntu 9.10 (or above), use the official Ubuntu Packages, which already solve all dependencies for you and are continuously updated with the latest bug fixes.
Debian or Ubuntu (9.04 or older)¶
If you’re running Debian Linux, run the following command as root:
apt-get install python-twisted python-libxml2 python-pyopenssl python-simplejson
And then follow the instructions in Install Scrapy.
Arch Linux¶
If you are running Arch Linux, run the following command as root:
pacman -S twisted libxml2 pyopenssl python-simplejson
And then follow the instructions in Install Scrapy.
Other Linux distros¶
The easiest way to install Scrapy in other Linux distros is through
easy_install
, which will automatically install Twisted and lxml as
dependencies. See Installing with easy_install.
Another way would be to install dependencies, if you know the packages in your distros that meets them. See Requirements.
Mac OS X¶
The easiest way to install Scrapy on Mac is through easy_install
, which
will automatically install Twisted and lxml as dependencies.
Windows¶
There are two ways to install Scrapy in Windows:
- using
easy_install
orpip
- see Installing with easy_install or Installing with pip - using the Windows installer, but you need to download and install the
dependencies manually:
- Twisted for Windows - you may need to install pywin32 because of this Twisted bug
- Install Zope.Interface (required by Twisted)
- libxml2 for Windows
- PyOpenSSL for Windows
- Download the Windows installer from the Downloads page and install it.