Welcome to Localshop¶
Localshop is a PyPI server which automatically proxies and mirrors PyPI packages based upon packages requested. It also supports the uploading of local (private) packages.
Contents:
Installing¶
Download and install localshop via the following command:
pip install localshop
This should best be done in a new virtualenv. Now initialize your localshop environment by issuing the following command:
localshop init
If you are upgrading from an earlier version simply run:
localshop upgrade
And then start it via:
gunicorn localshop.wsgi:application
You will also need to start the celery daemon, it’s responsible for downloading and updating the packages from PyPI. So open another terminal, activate your virtualenv (if you have created one) and run the following command:
localshop celery worker -B -E
You can now visit http://localhost:8000/ and view all the packages in your localshop!
Note: If you prefer to start listening on a different network interface and
HTTP port, you have the pass the parameter -b
to gunicorn
. For example,
the following command starts localshop on port 7000 instead of 8000:
gunicorn localshop.wsgi:application -b 0.0.0.0:7000
The next step is to give access to various hosts to use the shop. This is done via the webinterface (menu -> permissions -> cidr). Each ip address listed there will be able to download and upload packages. If you are unsure about ips configuration, but still want to use authentication, specify “0.0.0.0/0” as the unique cidr configuration. It will enable for any ip address.
Docker alternative¶
Install docker and docker-compose and then run:
cp docker.conf.py{.example,}
docker-compose build
docker-compose run localshop syncdb
docker-compose run localshop createsuperuser
docker-compose up
You should be able to see localshop running in http://docker-host:8000.
How it works¶
Packages which are requested and are unknown are looked up on pypi via the xmlrpc interface. At the moment the client downloads one of the files which is not yet mirror’ed a 302 redirect is issued to the correct file (on pypi). At that point the worker starts downloading the package and stores it in ~/.localshop/files so that the next time the package is request it is available within your own shop!
Uploading local/private packages¶
To upload your own packages to your shop you need to modify/create a .pypirc file. See the following example:
[distutils]
index-servers =
local
[local]
username: myusername
password: mysecret
repository: http://localhost:8000/repo/default/
To upload a custom package issue the following command in your package:
python setup.py upload -r local
It should now be available via the webinterace
Using the shop for package installation¶
To install packages with pip from your localshop add -i flag, e.g.:
pip install -i http://localhost:8000/repo/default/ localshop
or edit/create a ~/.pip/pip.conf file following this template:
[global]
index-url = http://<access_key>:<secret_key>@localhost:8000/repo/default/
Then just use pip install as you are used to do. You can replace access_key and secret_key by a valid username and password.
Credentials for authentication¶
If you don’t want to use your Django username/password to authenticate uploads and downloads you can easily create one of the random credentials localshop can create for you.
Go to the Credentials section and click on create. Use the access key
as the username and the secret key as the password when uloading packages.
A ~/.pypirc
could look like this:
[distutils]
index-servers =
local
[local]
username: 4baf221849c84a20b77a6f2d539c3e8a
password: 200984e70f0c463b994388c4da26ec3f
repository: http://localhost:8000/simple/
pip allows you do use those values in the index URL during download, e.g.:
pip install -i http://<access_key>:<secret_key>@localhost:8000/repo/default/ localshop
So for example:
pip install -i http://4baf221849c84a20b77a6f2d539c3e8a:200984e70f0c463b994388c4da26ec3f@localhost:8000/repo/default/ localshop
Warning
Please be aware that those credentials are transmitted unencrypted over http unless you setup your localshop instance to run on a server that serves pages via https.
In case you ever think a credential has been compromised you can disable it or delete it on the credential page.
Adding users¶
You can add users using the Django admin backend at /admin
. After adding
the user you need to assign the user to a team. Only teams can be assigned to
repositories.
Settings¶
There are a few settings to set in ~/.localshop/localshop.conf.py
that
change the behaviour of the localshop.
LOCALSHOP_DELETE_FILES
¶
default: | False |
---|
If set to True
files will be cleaned up after deleting a package or
release from the localshop.
LOCALSHOP_DISTRIBUTION_STORAGE
¶
default: | 'storages.backends.overwrite.OverwriteStorage' |
---|
The dotted import path of a Django storage class to be used when uploading a release file or retrieving it from PyPI.
LOCALSHOP_HTTP_PROXY
¶
default: | None |
---|
Proxy configuration used for Internet access. Expects a dictionary configured as mentioned by http://docs.python-requests.org/en/latest/user/advanced/#proxies
LOCALSHOP_ISOLATED
¶
default: | False |
---|
If set to True
Localshop never will try to redirect the client to PyPI.
This is useful for environments where the client has no Internet connection.
Note
If you set LOCALSHOP_ISOLATED
to True
, client request can be delayed
for a long time because the package must be downloaded from Internet before
it is served. You may want to set pip environment variable
PIP_DEFAULT_TIMEOUT
to a big value. Ex: 300
LOCALSHOP_USE_PROXIED_IP
¶
default: | False |
---|
If set to True
Localshop will use the X-Forwarded-For header to validate
the client IP address. Use this when Localshop is running behind a reverse
proxy such as Nginx or Apache and you want to use IP-based permissions.
LOCALSHOP_RELEASE_OVERWRITE
¶
default: | True |
---|
If set to False
, users will be preveneted from overwriting already existing
release files. Can be used to encourage developers to bump versions rather than
overwriting. This is PyPI’s behaviour.
LOCALSHOP_VERSIONING_TYPE
¶
default: | None |
---|
If set to False
, no versioning “style” will be enforced.
If you want to validated versions you can choose any Versio available backends.
IMPORTANT the value of this config must be a full path of the wanted class e.g. versio.version_scheme.Pep440VersionScheme.
- Simple3VersionScheme which supports 3 numerical part versions (A.B.C where A, B, and C are integers)
- Simple4VersionScheme which supports 4 numerical part versions (A.B.C.D where A, B, C, and D are integers)
- Pep440VersionScheme which supports PEP 440 versions (N[.N]+[{a|b|c|rc}N][.postN][.- devN][+local])
- PerlVersionScheme which supports 2 numerical part versions where the second part is at least two digits A.BB where A and B - are integers and B is zero padded on the left. For example: 1.02, 1.34, 1.567)
Contributing¶
Want to contribute with Localshop? Great! We really appreciate your help. But before digging into your new fluffy-next-millionaine-feature code keep in mind that you MUST follow this guide to get your pull requests approved.
Get started¶
- Fork the project and follow the installation instructions [1].
- Your code MUST contain tests. This is a requirement and no pull request will be approved if it lacks tests. Even if your’re making a small bug fix we want to ensure that it will not introduce any another bug.
- Help to keep the documentation up-to-date is really appreaciated. Always check if your’re making changes that make the documentation obsolete and update it.
- Squash your commits before making a pull request whenever possible. This will avoid history pollution with middle commits that breaks things. Your pull request should be a single commit with all your changes.
- Open a pull request. Usually, the target branch at the main repository
will be
develop
, but if your’re sending a bugfix to avoid the extinction of human race, maybe you want to target themaster
branch.
Tip
Use a meaningful and convincing pull request description. Feel free to use emojis to give us a clue of what kind changes your’re making. The Style guide contains some of our preferred ones.
Running Tests¶
To run all tests, simply use tox:
pip install tox tox -e py27 # use `tox -e py27 -r` to rebuild the virtual environment
To run a specific test, pass the test module filename as an argument:
tox -e py27 tests/apps/packages/test_models.py
If you would like tox to run tests for all supported python versions, you should first install pyenv.
After installing pyenv you should download all required python versions (see .python-version), and then simply run tox without the -e argument; e.g:
# install pyenv curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash # install all required python versions cat .python-version | xargs -I{} pyenv install {} # run all tests tox
tox automatically creates an environment and installs dependencies on each run. When actively developing, you may want to reuse your existing environment to quickly rerun the tests. In this case first install the test dependencies and then run py.test directly. E.g :
# recommended: set up a virtual environment localshop $ virtualenv -a . -r requirements.txt localshop # install the requirements (localshop) localshop $ pip install -r requirements.txt (localshop) localshop $ pip install -e .[test] # run the tests for the module you are working on (localshop) localshop (develop)$ py.test tests/apps/packages/test_models.py ================================= test session starts ================================= ... ============================== 3 passed in 0.92 seconds ===============================
Style guide¶
Follow the PEP8. Try to keep the line length to 79 but don’t make it a big a deal.
Make sure that your code does not raises any Pylint errors or warnings.
Always group the imports in 3 blocks: native libraries, third party libraries and project imports.
Keep the import block alphabetically ordered. If you use Sublime Text, you can do this by selecting the import block and hitting
F9
Avoid polluting the current namespace with lots of imports. If you find yourself in a situation of importing a lot of symbols from the same package, consider import the package itself.
Wrong way:
from django.core.exceptions import (ImproperlyConfigured, AppRegistryNotReady, FieldError, DisallowedHost, DisallowedRedirect, DjangoRuntimeWarning)
Preferred way:
from django.core import exceptions as djexc
Commit messages¶
Limit the first line to 72 characters or less
Always use English
- Consider starting the commit message with an applicable emoji:
:lipstick:
when improving the format/structure of the code:fire:
when removing code or files:bug:
when fixing a bug:beetle:
when fixing a bug:book:
when writing docs:green_heart:
when fixing the CI build:white_check_mark:
when adding tests:x:
when commiting code with failed tests:arrow_up:
when upgrading dependencies:arrow_down:
when downgrading dependencies
Footnotes
[1] | Installing |
Changelog¶
0.10.0 (in development)¶
- Add support for multiple indexes (repositories). Indexes can be managed via the web interface
- Implement teams
- Create separate user access keys and make the existing credentials only work in combination with repositories.
- Update interface to bootstrap 3
- Add support for Python 3.3 and 3.4
- Update django-storages to django-storages-redux
- The LOCALSHOP_PYPI_URL is removed and now configurable per repository
- Add button to package detail dashboard template to mirror a package release file.
0.9.2¶
- BUGFIX: Invalid template path in config. Fixes #149
0.9.1¶
- BUGFIX: Validate package name before save.
0.9.0¶
- Added optional package version checking using the Versio library.
0.8.3¶
- Fixed a bug added by the in the #137 fix. Package upload was not working correctly.
0.8.2¶
- Fixed bug #139. XMLRPC endpoints were broken.
0.8.1¶
- Fixed bug #137. Twine replaces underscores with hyphen in package names. This caused localshop to create a new package instead of using the existing.
0.8.0¶
- Changed the simple_detail view so that it creates the Package model asynchronously.
0.7.0¶
- Localshop no longer uses the PyPI XmlRPC API. It was replaced with the the JSON API.
- Fixed issue #134
0.6.1¶
- Localshop return a redirect when the MEDIA_URL is set.
0.6.0¶
- Upgraded to Django 1.7
- Dropped Python 2.6 support
- Fixed bug #117 (Localshop return a 404 when uploading a package from a python version >=2.7.9 or >=3.4.2
- Fixed bug #116 (Localshop returns a 500 error when trying to download a package with a missing file in disk)
0.5.0¶
- Allow localshop to be initialised without any interaction, useful for
- automating installation Fix downloading packages with mismatched underscore vs dashes in the package names.
- Switch to pytest Include various fixes (see pull-requests / contributors)
0.4.1 (2013-01-23)¶
- Fix a bug which resulted in throwing 404’s for packages which have a dot in the name.
0.4.0 (2013-01-07)¶
- Use django-userena for authentication
- Use django-configurations and now uses the optional custom settings at ~/.localshop.py. Logan is now also not used anymore.
- Add additional separate credentials for uploading and downloading from the pypi instance (using access/secret keys).
- Implement support for ‘pip search’
- Many other improvements! (thanks Jannis Leidel)
0.3.0¶
- Use Django 1.4 and restructure the app layout
- Other minor bugfixes
0.2.2¶
- Don’t display the download_url or home_page url if they are unknown
- Add correct rel-tag to the urls
0.2.1¶
- Use the correct LOGIN_URL setting.
- Add download_url and home_page url to the simple detail page. note that this is currently not mirrored yet.
0.2.0¶
- Implement configurable access control / ip check
0.1.3¶
- Fix distutils interface
- Add missing requirement (docutils)
0.1.2¶
- Refactor client validation for /simple and download
0.1.1¶
- Add setting to list ip addresses which have access to download packages, LOCALSHOP_ALLOWED_REMOTE_IPS.
0.1.0¶
- Initial release