Environment¶

We recommend highly to use Debian based configuration as they are the only one having been tested for now. The following configuration takes into account what we think might be the best configuration available with a good cache system.

You can use a docker image we built and fork it for your own use. As of April 11th, 2016, the docker image does not use Redis-based cache but filesystem based cache.

The environment we propose contains a flask.ext.nemo instance, for control purposes. Disabling it is documented.

Nginx, Supervisor, GUnicorn¶

The following configuration file for supervisor is enough for running the whole nginx, supervisord and gunicorn trio. In general, servers configuration should always - in production - have a trio made of an HTTP/Reverse proxy server such as nginx, a process control system such as supervisor and a WSGI http server such as gunicorn or uWSGI. This configuration takes the road of gunicorn, but feel free to test and benchmark any combination to know what works best on your own server(s).

On the software is read, run :

service supervisor stop
service nginx stop

apt-get install zlib1g-dev libxslt1-dev libxml2-dev python3 python3-dev python3-pip build-essential nginx supervisor
apt-get install python-setuptools

easy_install pip
pip2.7 install supervisor-stdout
easy_install3 --upgrade pip

mkdir /var/capitains-server
cd /var/capitains-server

virtualenv venv
pip install Nautilus
pip install gunicorn
pip install flask_nemo

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

[supervisord]
nodaemon = true

[program:nginx]
command = /usr/sbin/nginx
startsecs = 5
stdout_events_enabled = true
stderr_events_enabled = true

[program:app-gunicorn]
# See explanation for this line
command=/usr/local/bin/gunicorn app:app -w 4 --threads 2 -b 127.0.0.1:5000 --log-level=debug --pythonpath /usr/bin/python3
directory=/code
stdout_events_enabled = true
stderr_events_enabled = true

[eventlistener:stdout]
command = supervisor_stdout
buffer_size = 100
events = PROCESS_LOG
result_handler = supervisor_stdout:event_handler

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

daemon off;
error_log /dev/stdout info;
worker_processes 1;

# user nobody nogroup;
pid /tmp/nginx.pid;

events {
    worker_connections 1024;
    accept_mutex off;
}

http {
    include mime.types;
    default_type application/octet-stream;
    access_log /dev/stdout combined;
    sendfile on;

    upstream app_server {
        # For a TCP configuration:
        server 127.0.0.1:5000 fail_timeout=0;
    }

    server {
        listen 80 default;
        client_max_body_size 4G;
        server_name _;

        keepalive_timeout 5;

        # path for static files
        root /opt/app/static;

        location / {
            # checks for static file, if not found proxy to app
            try_files $uri @proxy_to_app;
        }

        location @proxy_to_app {
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $http_host;
            proxy_redirect off;

            proxy_pass   http://app_server;
        }

    }
}

Flask Application Configuration¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

 # -*- coding: utf-8 -*-

 from flask import Flask, request
 from flask.ext.nemo import Nemo
 from capitains_nautilus.flask_ext import FlaskNautilus
 from werkzeug.contrib.cache import FileSystemCache
 from flask_cache import Cache

 app = Flask("Nautilus")
 nautilus_cache = FileSystemCache("/var/capitains-cache")
 nautilus = FlaskNautilus(
     app=app,
     prefix="/api/cts",
     name="nautilus",
     # Add here paths to all CapiTainS repository you have locally
     resources=["/var/capitains-data/canonical-latinLit-master"],
     parser_cache=nautilus_cache,
     http_cache=Cache(config={'CACHE_TYPE': "simple"})
 )
 # We set up Nemo
 # This part can be removed
 nemo = Nemo(
     app=app,
     name="nemo",
     base_url="",
     api_url="/api/cts",
     endpoint=nautilus.retriever
 )
 # We register its routes
 nemo.register_routes()
 # We register its filters
 nemo.register_filters()

 # Removes this line for production
 app.debug = True

 if __name__ == "__main__":
     app.run(debug=True, host='0.0.0.0')

Library Structure¶

Resolvers¶

Resolver provides a system to retrieve a text file and an inventory from local resources for example.

Retriever¶

Errors¶

exception capitains_nautilus.errors.CTSError[source]¶

Bases: BaseException

CODE = None¶

exception capitains_nautilus.errors.InvalidContext[source]¶

Bases: capitains_nautilus.errors.CTSError

Invalid value for context parameter in GetPassage or GetPassagePlus request

CODE = 5¶

exception capitains_nautilus.errors.InvalidLevel[source]¶

Bases: capitains_nautilus.errors.CTSError

Invalid value for level parameter in GetValidReff request

CODE = 4¶

exception capitains_nautilus.errors.InvalidURN[source]¶

Bases: capitains_nautilus.errors.CTSError

Syntactically valid URN refers in invalid value

CODE = 3¶

exception capitains_nautilus.errors.InvalidURNSyntax[source]¶

Bases: capitains_nautilus.errors.CTSError

Invalid URN syntax

CODE = 2¶

exception capitains_nautilus.errors.MissingParameter[source]¶

Bases: capitains_nautilus.errors.CTSError

Request missing one or more required parameters

CODE = 1¶

exception capitains_nautilus.errors.UnknownResource[source]¶

Bases: capitains_nautilus.errors.CTSError

Resource requested is not found

CODE = 6¶

Cache¶

class capitains_nautilus.cache.BaseCache(default_timeout=300)[source]¶

Bases: object

Based on the werkzeug.contrib.cache.BaseCache object. Provides a wrapper for other cache system in the future.

Parameters:	default_timeout – the default timeout (in seconds) that is used if no timeout is specified on set(). A timeout of 0 indicates that the cache never expires.

add(key, value, timeout=None)[source]¶

Works like set() but does not overwrite the values of already existing keys. :param key: the key to set :param value: the value for the key :param timeout: the cache timeout for the key in seconds (if not

specified, it uses the default timeout). A timeout of 0 idicates that the cache never expires.

Returns:	Same as set(), but also False for already existing keys.
Return type:	boolean

clear()[source]¶

Clears the cache. Keep in mind that not all caches support completely clearing the cache. :returns: Whether the cache has been cleared. :rtype: boolean

dec(key, delta=1)[source]¶

Decrements the value of a key by delta. If the key does not yet exist it is initialized with -delta. For supporting caches this is an atomic operation. :param key: the key to increment. :param delta: the delta to subtract. :returns: The new value or None for backend errors.

delete(key)[source]¶

Delete key from the cache. :param key: the key to delete. :returns: Whether the key existed and has been deleted. :rtype: boolean

delete_many(*keys)[source]¶

Deletes multiple keys at once. :param keys: The function accepts multiple keys as positional

arguments.

Returns:	Whether all given keys have been deleted.
Return type:	boolean

get(key)[source]¶

Look up key in the cache and return the value for it. :param key: the key to be looked up. :returns: The value if it exists and is readable, else None.

get_dict(*keys)[source]¶

Like get_many() but return a dict::

d = cache.get_dict(“foo”, “bar”) foo = d[“foo”] bar = d[“bar”]

Parameters:	keys – The function accepts multiple keys as positional arguments.

get_many(*keys)[source]¶

Returns a list of values for the given keys. For each key a item in the list is created:

foo, bar = cache.get_many("foo", "bar")

Has the same error handling as get(). :param keys: The function accepts multiple keys as positional

arguments.

has(key)[source]¶

Checks if a key exists in the cache without returning it. This is a cheap operation that bypasses loading the actual data on the backend. This method is optional and may not be implemented on all caches. :param key: the key to check

inc(key, delta=1)[source]¶

Increments the value of a key by delta. If the key does not yet exist it is initialized with delta. For supporting caches this is an atomic operation. :param key: the key to increment. :param delta: the delta to add. :returns: The new value or None for backend errors.

set(key, value, timeout=None)[source]¶

Add a new key/value to the cache (overwrites value, if key already exists in the cache). :param key: the key to set :param value: the value for the key :param timeout: the cache timeout for the key in seconds (if not

specified, it uses the default timeout). A timeout of 0 idicates that the cache never expires.

Returns:	True if key has been updated, False for backend errors. Pickling errors, however, will raise a subclass of pickle.PickleError.
Return type:	boolean

set_many(mapping, timeout=None)[source]¶

Sets multiple keys and values from a mapping. :param mapping: a mapping with the keys/values to set. :param timeout: the cache timeout for the key in seconds (if not

specified, it uses the default timeout). A timeout of 0 idicates that the cache never expires.

Returns:	Whether all given keys have been set.
Return type:	boolean

Flask Extension¶

class capitains_nautilus.flask_ext.FlaskNautilus(prefix='', app=None, name=None, resources=None, parser_cache=None, compresser=True, http_cache=None, pagination=False, access_Control_Allow_Origin=None, access_Control_Allow_Methods=None, logger=None, auto_parse=True)[source]¶

Bases: object

Initiate the class

Parameters:

prefix – Prefix on which to install the extension app – Application on which to register name – Name to use for the blueprint resources (list(str)) – List of directory to feed the inventory logger (logging) – Logging handler. parser_cache (BaseCache) – Cache object http_cache – HTTP Cache should be a FlaskCache object auto_parse – Parses on first execution the resources given to build inventory. Not recommended for production

Variables:

ROUTES – List of triple length tuples Access_Control_Allow_Methods – Dictionary with route name and allowed methods over CORS Access_Control_Allow_Origin – Dictionary with route name and allowed host over CORS or “*” LoggingHandler – Logging handler to be set for the blueprint logger – Logging handler retriever – CapiTainS Retriever

Access_Control_Allow_Methods = {'r_dispatcher': 'OPTIONS, GET'}¶

Access_Control_Allow_Origin = '*'¶

LoggingHandler¶

alias of StreamHandler

ROUTES = [('/', 'r_dispatcher', ['GET'])]¶

init_app(app, compresser=False)[source]¶

Initiate the extension on the application

Parameters:	app – Flask Application
Returns:	Blueprint for Flask Nautilus registered in app
Return type:	Blueprint

init_blueprint()[source]¶

Properly generates the blueprint, registering routes and filters and connecting the app and the blueprint

Returns:	Blueprint of the extension
Return type:	Blueprint

r_dispatcher()[source]¶

Actual main route of CTS APIs. Transfer typical requests through the ?request=REQUESTNAME route

Returns:	Response

setLogger(logger)[source]¶

Set up the Logger for the application

Parameters:	logger – logging.Logger object
Returns:	Logger instance

view(function_name)[source]¶

Builds response according to a function name

Parameters:	function_name – Route name / function name
Returns:	Function

capitains_nautilus.flask_ext.FlaskNautilusManager(nautilus, app=None)[source]¶

Provides a manager for flask scripts to perform specific maintenance operations

Parameters:	nautilus – Nautilus Extension Instance app – Flask Application
Returns:	Sub-Manager
Return type:	Manager

Import with

class capitains_nautilus.flask_ext.WerkzeugCacheWrapper(instance=None, *args, **kwargs)[source]¶

Bases: capitains_nautilus.cache.BaseCache

Werkzeug Cache Wrapper for Nautilus Base Cache object

Parameters:	instance – Werkzeug Cache instance

add(key, value, timeout=None)[source]¶

clear()[source]¶

delete(key)[source]¶

get(key)[source]¶

set(key, value, timeout=None)[source]¶