Welcome

Welcome to Open Library developer documentation.

This place is intended for developers of Open Library project.

If you are new to Open Library development, you can get started by Setting up a dev instance.

Table of Contents

Setting up a dev instance

Open Library supports dev instance based on Vagrant. This document will step you though the installation process.

The vagrant setup uses Ubuntu 14.04 LTS operating system. Make sure you have at least 1GB of RAM in the virtual machine.

Getting the source

Open Library uses git for version control and the code repository is hosted on github.

You can get the source code from there using:

$ git clone git://github.com/internetarchive/openlibrary.git
$ cd openlibrary

In case you don’t have git installed already, you can install it on Ubuntu using:

$ sudo apt-get install git-core

and on Mac OS X using:

$ brew install git

Starting the dev-instance

The Open Library dev-instance can be started using:

$ vagrant up

This will setup a virtual machine with Ubuntu 14.04, installs all dependencies, setup database and loads sample data.

Once, the virtual machine is up, you’ll be able to access the website at:

http://0.0.0.0:8080/

An admin user with the following credentials is created as part of the installation.

<pre> username: openlibrary password: openlibrary </pre>

Known Issues

It is known that the following issues exist:

  • Stats on the home page is not working
  • /admin is failing
  • subject search is not working

Getting Started with Development

Working with JavaScript

Introduction

The Open Library website makes heavy use of jQuery libraries.

Except jquery and jquery-ui, all other third-party javascript libraries are combined and included as vendor.js.

All the custom javascripts are combined and includes as all.js.

vendor.js and third party libraries

All third-party javascripts are added in the vendor/js directory in the repository and static/build/vendor.js is generated by combining these javascripts. The files included in static/build/vendor.js are specified in a shell script static/js/vendor.jsh.

To include a new third-party library:

  • Add that library in vendor/js in the repository

  • Add an entry in static/js/vendor.jsh

  • Generate vendor.js by running make

    $ make js

  • Commit vendor.jsh and the library added to the repository

all.js and custom javascripts

All the custom javascript files are put in the repo at openlibrary/plugins/openlibrary/js. All these javascript files are combined to generate build/js/all.js.

The order in which these files are included is determined by the sort order of the names. In general, it is a bad idea to depend on the order of files loaded.

If you make any changes to any of the javascript files, run make js to regenerate build/js/all.js.

Working with CSS

All stylesheets are in static/css. They are combined to generate build/css/all.css, which is included in all the web pages.

It’s a good idea to break CSS into multiple logical files, instead of putting it in one monolithic file.

If you make changes to any CSS, run make css to regenerate build/css/all.css.

Architecture of the dev instance

This page contains details on how the dev instance is designed, how the various pieces run and interact with each other.

  • Services used
  • How they interact
  • How are they managed
  • Logging
  • Config files

Design Notes

Contains design and implementation details of complex parts of the system.

Mostly about specific pieces like lists (not overall)

Dev Instance

Installing Dependencies
Installing CouchDB

On Linux, installing CouchDB involves installing so many dependencies, some of them are incompatible with ubuntu/debian installations. Fortunately, a binary distribution with all the binaries is provided by CouchOne.

http://www.couchone.com/get

The problem with that distribution is that it is interactive and it won’t be possible to automate the installation of that. To overcome this problem, the binaries after running the interactive installation are taken and bundled. The ol install script, downloads the bundle, unpacks it and runs a script to updates the path in some scripts and config files.

The created bundle couchdb-1.0.1-linux-binaries.tgz is uploaded to http://www.archive.org/details/ol_vendor.

On Max OS X, installing couchdb is somewhat easier than installing on Linux. However to make the installations on both mac and linux similar, binaries are taken from CouchDBX app.

Unlike Linux, where bin and etc are available in the top-level, mac binaries have them at 2 level deep. To make both distributions identical, a bin/couchdb script is added and etc is symlinked from couchdb_1.0.1/etc. Also the mac couchdb script expects the base directory should be the working directory. The above bin/couchdb script, takes of this too.

Lists

This document explain how lists feature is implemented.

Feature Flags

Feature flags is a way to have features under development live on production and visible only to admins/beta-users.

The idea of Feature Flags came from Flicker. They manage their development on a single branch using feature flags.

http://code.flickr.com/blog/2009/12/02/flipping-out/

Using Feature Flags

Feature flags are used in templates and in controller classes.

To make some part of the template visible only if a feature-flag is enabled:

$if "lists" in ctx.features:
    <h3>Lists</h3>
    $for list in page.get_lists():
        ...

To enable a url only if a feature flag is enabled:

class home(delegate.page):
    path = "/"

    def is_enabled(self):
        return "home-v2" in web.ctx.features

    def GET(self):
        return render_template("home")
Setting Feature Flags

In Open Library, the feature flags are specified in the openlibrary.yml file as follows:

features:
    merge-authors: enabled
    lists: admin
    lending_v2:
        filter: usergroup
        usergroup: beta-users

The value of a feature flag is called a filter. A filter can be specified either as its name or as a dict containing its name and parameters. For example, the following 2 example mean the same.

features:
    lists: admin

features:
    lists:
        filter: admin

Available filters are:

enabled

Enabled for all users.

disabled

Disabled for all users.

loggedin

Enabled only for logged-in users.

admin

Enabled for admin users.

usergroup

Enabled for the users part of the specified usergroup.

lending_v2:
    filter: usergroup
    usergroup: beta-users

queryparam

Enabled only if the url has a specified query parameter.

debug:
    filter: queryparam
    name: debug
    value: true

API Documentation

This is the API documentation generated from openlibrary source code.

Appendices

Configuration files

List of all the relevant config files, their syntaxes and what the control.

setup.py commands

Most openlibrary operations are modeled as setup.py commands.

bootstrap

Bootstraps the dev instance.

$ python setup.py bootstrap

Bootstrap command is explained in bootstrap.

start

Starts all the OL services.:

$ python setup.py start

This command starts supervisord using configuration from conf/supervisor/linux.ini or conf/supervisor/macosx.ini based on the platform.

Logs of the services are written to var/log.

shell

Start a bash shell with the env required for running all OL scripts.

$ python setup.py shell

This starts a bash shell with conf/bashrc as rc file.

test

Runs all the test cases.

$ python setup.py test

Behind the scenes it runs scripts/runtests.sh.

build_sphinx

Builds the sphinx docs.

$ python setup.py build_sphinx

The docs will be generated in build/sphinx/html/.

The custom setup.py commands are implemented in setup_commands.py.

To know more about how to add custom setuptools commands, see:

http://tarekziade.wordpress.com/2007/09/30/extending-setuptools-adding-a-new-command/

Database Migrations

Ocationally, new tables get added to openlibrary database and some existing tables get altered. Scripts are provided to migrate the existing dev instances to new schema.

To migrate an existing dev instance:

$ python setup.py shell
$ python scripts/migrate_db.py

This will look the current database schema and identifies its version and upgrades it to the latest version.

Dependencies

The system requires a lot of 3rd party programs as well as a whole bunch of python libraries. We have an installation script which you can run that will setup the entire thing for you but in order to run this script, the following packages have to be installed first.

Git
For getting the source repository
PostgreSQL 8.2 or later (psql)
This is where we store our data
Python 2.5 or later (python)
The application is written in python
Java Runtime (tested with openjdk-6-jre)
The indexer (solr) is a Java application
Python virtualenv (python-virtualenv)
Necessary to create “virtual” installations of Python so that we can install packages without touching your system distribution. More details at the virtualenv PyPI page.
Apache Solr (installed by the installation script)
The search engine.

On Linux, you will also have to install the following dev packages python-dev, libpq-dev and libxslt-dev.

Some Python packages are also required, but they will be automatically installed by the installation script.

Indices and tables