Basic Auth¶

If enabled in configuration, using a Basic Auth token will associate a unique user identifier to an username/password combination.

Authorization: Basic <basic_token>

The token shall be built using this formula base64("username:password").

Empty passwords are accepted, and usernames can be anything (custom, UUID, etc.)

If the token has an invalid format, or if Basic Auth is not enabled, this will result in a 401 error response.

OAuth Bearer token¶

If the configured authentication policy uses OAuth2 bearer tokens, authentication shall be done using this header:

Authorization: Bearer <oauth_token>

The policy will verify the provided OAuth2 bearer token on a remote server.

Firefox Accounts¶

In order to enable authentication with Firefox Accounts, install and configure mozilla-services/cliquet-fxa.

GET /{collection}¶

Requires authentication

Returns all records of the current user for this collection.

The returned value is a JSON mapping containing:

• data: the list of records, with exhaustive fields;
• permissions: optional a json dict containing the permissions for the collection of records.

A Total-Records response header indicates the total number of records of the collection.

A Last-Modified response header provides a human-readable (rounded to second) of the current collection timestamp.

For cache and concurrency control, an ETag response header gives the value that consumers can provide in subsequent requests using If-Match and If-None-Match headers (see section about timestamps).

Request:

GET /articles HTTP/1.1
Accept: application/json
Authorization: Basic bWF0Og==
Host: localhost:8000

Response:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Backoff, Retry-After, Alert, Content-Length, ETag, Next-Page, Total-Records, Last-Modified
Content-Length: 436
Content-Type: application/json; charset=UTF-8
Date: Tue, 28 Apr 2015 12:08:11 GMT
Last-Modified: Mon, 12 Apr 2015 11:12:07 GMT
ETag: "1430222877724"
Total-Records: 2

{
    "data": [
        {
            "id": "dc86afa9-a839-4ce1-ae02-3d538b75496f",
            "last_modified": 1430222877724,
            "title": "MoCo",
            "url": "https://mozilla.com",
        },
        {
            "id": "23160c47-27a5-41f6-9164-21d46141804d",
            "last_modified": 1430140411480,
            "title": "MoFo",
            "url": "https://mozilla.org",
        }
    ]
}

GET /articles?_since=1437035923844 HTTP/1.1
Accept: application/json
Authorization: Basic bWF0Og==
Host: localhost:8000

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Backoff, Retry-After, Alert, Content-Length, ETag, Next-Page, Total-Records, Last-Modified
Content-Length: 436
Content-Type: application/json; charset=UTF-8
Date: Tue, 28 Apr 2015 12:08:11 GMT
Last-Modified: Mon, 12 Apr 2015 11:12:07 GMT
ETag: "1430222877724"
Total-Records: 2

{
    "data": [
        {
            "id": "dc86afa9-a839-4ce1-ae02-3d538b75496f",
            "last_modified": 1430222877724,
            "title": "MoCo",
            "url": "https://mozilla.com",
        },
        {
            "id": "23160c47-27a5-41f6-9164-21d46141804d",
            "last_modified": 1430140411480,
            "title": "MoFo",
            "url": "https://mozilla.org",
        },
        {
            "id": "11130c47-37a5-41f6-9112-32d46141804f",
            "deleted": true,
            "last_modified": 1430140411480
        }
    ]
}

POST /{collection}¶

Requires authentication

Used to create a record in the collection. The POST body is a JSON mapping containing:

• data: the values of the resource schema fields;
• permissions: optional a json dict containing the permissions for the record to be created.

The POST response body is a JSON mapping containing:

• data: the newly created record, if all posted values are valid;
• permissions: optional a json dict containing the permissions for the requested resource.

If the request header If-Match is provided, and if the record has changed meanwhile, a 412 Precondition failed error is returned.

Request:

POST /articles HTTP/1.1
Accept: application/json
Authorization: Basic bWF0Og==
Content-Type: application/json; charset=utf-8
Host: localhost:8000

{
    "data": {
        "title": "Wikipedia FR",
        "url": "http://fr.wikipedia.org"
    }
}

Response:

HTTP/1.1 201 Created
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Backoff, Retry-After, Alert, Content-Length
Content-Length: 422
Content-Type: application/json; charset=UTF-8
Date: Tue, 28 Apr 2015 12:35:02 GMT

{
    "data": {
        "id": "cd30c031-c208-4fb9-ad65-1582d2a7ad5e",
        "last_modified": 1430224502529,
        "title": "Wikipedia FR",
        "url": "http://fr.wikipedia.org"
    }
}

DELETE /{collection}¶

Requires authentication

Delete multiple records. Disabled by default, see Configuration.

The DELETE response is a JSON mapping containing:

• data: list of records that were deleted, without schema fields.

It supports the same filtering capabilities as GET.

If the request header If-Match is provided, and if the collection has changed meanwhile, a 412 Precondition failed error is returned.

Request:

DELETE /articles HTTP/1.1
Accept: application/json
Authorization: Basic bWF0Og==
Host: localhost:8000

Response:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Backoff, Retry-After, Alert, Content-Length
Content-Length: 193
Content-Type: application/json; charset=UTF-8
Date: Tue, 28 Apr 2015 12:38:36 GMT

{
    "data": [
        {
            "deleted": true,
            "id": "cd30c031-c208-4fb9-ad65-1582d2a7ad5e",
            "last_modified": 1430224716097
        },
        {
            "deleted": true,
            "id": "dc86afa9-a839-4ce1-ae02-3d538b75496f",
            "last_modified": 1430224716098
        }
    ]
}

GET /{collection}/<id>¶

Requires authentication

Returns a specific record by its id. The GET response body is a JSON mapping containing:

• data: the record with exhaustive schema fields;
• permissions: optional a json dict containing the permissions for the requested record.

If the request header If-None-Match is provided, and if the record has not changed meanwhile, a 304 Not Modified is returned.

Request:

GET /articles/d10405bf-8161-46a1-ac93-a1893d160e62 HTTP/1.1
Accept: application/json
Authorization: Basic bWF0Og==
Host: localhost:8000

Response:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Backoff, Retry-After, Alert, Content-Length, ETag, Last-Modified
Content-Length: 438
Content-Type: application/json; charset=UTF-8
Date: Tue, 28 Apr 2015 12:42:42 GMT
ETag: "1430224945242"

{
    "data": {
        "id": "d10405bf-8161-46a1-ac93-a1893d160e62",
        "last_modified": 1430224945242,
        "title": "No backend",
        "url": "http://nobackend.org"
    }
}

DELETE /{collection}/<id>¶

Requires authentication

Delete a specific record by its id.

The DELETE response is the record that was deleted. The DELETE response is a JSON mapping containing:

• data: the record that was deleted, without schema fields.

If the record is missing (or already deleted), a 404 Not Found is returned. The consumer might decide to ignore it.

If the request header If-Match is provided, and if the record has changed meanwhile, a 412 Precondition failed error is returned.

PUT /{collection}/<id>¶

Requires authentication

Create or replace a record with its id. The PUT body is a JSON mapping containing:

• data: the values of the resource schema fields;
• permissions: optional a json dict containing the permissions for the record to be created.

The PUT response body is a JSON mapping containing:

• data: the newly created/updated record, if all posted values are valid;
• permissions: optional the newly created permissions dict, containing the permissions for the created record.

Validation and conflicts behaviour is similar to creating records (POST).

If the request header If-Match is provided, and if the record has changed meanwhile, a 412 Precondition failed error is returned.

Request:

PUT /articles/d10405bf-8161-46a1-ac93-a1893d160e62 HTTP/1.1
Accept: application/json
Authorization: Basic bWF0Og==
Content-Type: application/json; charset=utf-8
Host: localhost:8000

{
    "data": {
        "title": "Static apps",
        "url": "http://www.staticapps.org"
    }
}

Response:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Backoff, Retry-After, Alert, Content-Length
Content-Length: 439
Content-Type: application/json; charset=UTF-8
Date: Tue, 28 Apr 2015 12:46:36 GMT
ETag: "1430225196396"

{
    "data": {
        "id": "d10405bf-8161-46a1-ac93-a1893d160e62",
        "last_modified": 1430225196396,
        "title": "Static apps",
        "url": "http://www.staticapps.org"
    }
}

PATCH /{collection}/<id>¶

Requires authentication

Modify a specific record by its id. The PATCH body is a JSON mapping containing:

• data: a subset of the resource schema fields;
• permissions: optional a json dict containing the permissions for the record to be modified.

The PATCH response body is a JSON mapping containing:

• data: the modified record (full by default);
• permissions: optional the newly created permissions dict, containing the permissions for the modified record.

If a request header Response-Behavior is set to light, only the fields whose value was changed are returned. If set to diff, only the fields whose value became different than the one provided are returned.

Request:

PATCH /articles/d10405bf-8161-46a1-ac93-a1893d160e62 HTTP/1.1
Accept: application/json
Authorization: Basic bWF0Og==
Content-Type: application/json; charset=utf-8
Host: localhost:8000

{
    "data": {
        "title": "No Backend"
    }
}

Response:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Backoff, Retry-After, Alert, Content-Length
Content-Length: 439
Content-Type: application/json; charset=UTF-8
Date: Tue, 28 Apr 2015 12:46:36 GMT
ETag: "1430225196396"

{
    "data": {
        "id": "d10405bf-8161-46a1-ac93-a1893d160e62",
        "last_modified": 1430225196396,
        "title": "No Backend",
        "url": "http://nobackend.org"
    }
}

If the record is missing (or already deleted), a 404 Not Found error is returned. The consumer might decide to ignore it.

If the request header If-Match is provided, and if the record has changed meanwhile, a 412 Precondition failed error is returned.

Protected resources¶

All of the described endpoints can be either protected or not. Protecting an enpoint means that only principals which have been granted access will be able to issue requests successfully.

In the case of a protected resource, body is a JSON mapping containing a permissions key in addition to the data key. Permissions can also be replaced and modified independantly from data.

On a request, permissions is a json dict containing the permissions for the record to be modified. It has the following signature:

'permissions': {'{permission}': [{list_of_principals}]}

{permission} is a placeholder for the permission name (e.g. read, write, create) and {list_of_principals} should be replaced by an actual list of principals.

permissions is also added to JSON mapping response bodies, and contains the modified version of the permissions in case of a modification, or the list of permissions in case of a read operation.

POST /batch¶

Requires authentication

The POST body is a mapping, with the following attributes:

• requests: the list of requests
• defaults: (optional) default requests values in common for all requests

Each request is a JSON mapping, with the following attribute:

• method: HTTP verb
• path: URI
• body: a mapping
• headers: (optional), otherwise take those of batch request

{
  "defaults": {
    "method" : "POST",
    "path" : "/v0/articles",
    "headers" : {
      ...
    }
  },
  "requests": [
    {
      "body" : {
        "title": "MoFo",
        "url" : "http://mozilla.org",
        "added_by": "FxOS",
      }
    },
    {
      "body" : {
        "title": "MoCo",
        "url" : "http://mozilla.com"
        "added_by": "FxOS",
      }
    },
    {
      "method" : "PATCH",
      "path" : "/articles/409",
      "body" : {
        "read_position" : 3477
      }
    }
  ]
}

The response body is a list of all responses:

{
  "responses": [
    {
      "path" : "/articles/409",
      "status": 200,
      "body" : {
        "id": 409,
        "url": "...",
        ...
        "read_position" : 3477
      },
      "headers": {
        ...
      }
    },
    {
      "status": 201,
      "path" : "/articles",
      "body" : {
        "id": 411,
        "title": "MoFo",
        "url" : "http://mozilla.org",
        ...
      },
    },
    {
      "status": 201,
      "path" : "/articles",
      "body" : {
        "id": 412,
        "title": "MoCo",
        "url" : "http://mozilla.com",
        ...
      },
    },
  ]
}

GET /¶

The returned value is a JSON mapping containing:

• hello: the name of the service (e.g. "reading list")

• version: complete version ("X.Y.Z")

• commit: the HEAD git revision number when run from a git repository.

• url: absolute URI (without a trailing slash) of the API (can be used by client to build URIs)

• eos: date of end of support in ISO 8601 format ("yyyy-mm-dd", undefined if unknown)

• documentation: The URL to the service documentation. (this document!)

• settings: a mapping with the values of relevant public settings for clients

  • cliquet.batch_max_requests: Number of requests that can be made in a batch request.

• userid: The connected perso user id. The field is not present when no Authorization header is provided.

GET /__heartbeat__¶

Return the status of each service the application depends on. The returned value is a JSON mapping containing:

• storage true if operational
• cache true if operational
• oauth true if operational, or null if not enabled

Return 200 if the connection with each service is working properly and 503 if something doesn’t work.

Cache control¶

In order to check that the client version has not changed, a If-None-Match request header can be used. If the response is 304 Not Modified then the cached version if still good.

Concurrency control¶

In order to prevent race conditions, like overwriting changes occured in the interim for example, a If-Match request header can be used. If the response is 412 Precondition failed then the resource has changed meanwhile.

The client can then choose to:

• overwrite by repeating the request without If-Match;
• reconcile the resource by fetching, merging and repeating the request.

Backoff header on heavy load¶

A Backoff header will be added to the success responses (>=200 and <400) when the server is under heavy load. It provides the client with a number of seconds during which it should avoid doing unnecessary requests.

Backoff: 30

Retry-After indicators¶

A Retry-After header will be added to error responses (>=500), telling the client how many seconds it should wait before trying again.

Retry-After: 30

Protocol description¶

Every response is JSON.

If the HTTP status is not OK (<200 or >=400), the response contains a JSON mapping, with the following attributes:

• code: matches the HTTP status code (e.g 400)
• errno: stable application-level error number (e.g. 109)
• error: string description of error type (e.g. "Bad request")
• message: context information (e.g. "Invalid request parameters")
• info: online resource (e.g. URL to error details)
• details: additional details (e.g. list of validation errors)

Example response

{
    "code": 412,
    "errno": 114,
    "error": "Precondition Failed",
    "message": "Resource was modified meanwhile",
    "info": "https://server/docs/api.html#errors",
}

Refer yourself to the ref:set of errors codes <errors>.

Precondition errors¶

As detailed in the timestamps section, it is possible to add concurrency control using ETag request headers.

When a concurrency error occurs, a 412 Precondition Failed error response is returned.

Additional information about the record currently stored on the server will be provided in the details field:

{
    "code": 412,
    "errno": 114,
    "error":"Precondition Failed"
    "message": "Resource was modified meanwhile",
    "details": {
        "existing": {
            "last_modified": 1436434441550,
            "id": "00dd028f-16f7-4755-ab0d-e0dc0cb5da92",
            "title": "Original title"
        }
    },
}

Conflict errors¶

When a record violates unicity constraints, a 409 Conflict error response is returned.

Additional information about conflicting record and field name will be provided in the details field.

{
    "code": 409,
    "errno": 122,
    "error": "Conflict",
    "message": "Conflict of field url on record eyjafjallajokull"
    "info": "https://server/docs/api.html#errors",
    "details": {
        "field": "url",
        "record": {
            "id": "eyjafjallajokull",
            "last_modified": 1430140411480,
            "url": "http://mozilla.org"
        }
    }
}

Validation errors¶

When multiple validation errors occur on a request, the first one is presented in the message.

The full list of validation errors is provided in the details field.

{
    "code": 400,
    "errno": 109,
    "error": "Bad Request",
    "message": "Invalid posted data",
    "info": "https://server/docs/api.html#errors",
    "details": [
        {
            "description": "42 is not a string: {'name': ''}",
            "location": "body",
            "name": "name"
        }
    ]
}

Supported Python versions¶

Cliquet supports Python 2.7, Python 3.4 and PyPy.

Distribute & Pip¶

Installing Cliquet with pip:

pip install cliquet

For PostgreSQL and monitoring support:

pip install cliquet[postgresql,monitoring]

If everything is under control python-wise, jump to the next chapter. Otherwise please find more details below.

Python 3.4¶

sudo apt-get install python3.4-dev

brew install python3.4

Cryptography libraries¶

apt-get install libffi-dev libssl-dev

apt-get install libffi-devel openssl-devel

brew install libffi openssl pkg-config

Install Redis¶

apt-get install redis-server

yum install redis

brew install redis

brew services restart redis

Install PostgreSQL¶

sudo apt-get install libpq-dev

pip install cliquet[postgresql]

sudo apt-get install postgresql-9.4

sudo -u postgres psql -c "ALTER USER postgres PASSWORD 'postgres';"

sudo -u postgres psql -c "ALTER ROLE postgres SET TIMEZONE TO 'UTC';"
sudo -u postgres psql -c "CREATE DATABASE testdb ENCODING 'UTF-8';"

sudo apt-get install docker.io

postgres=$(sudo docker run -d -p 5432:5432 postgres)

psql -h localhost -U postgres -W
#> CREATE DATABASE "testdb";

sudo docker commit $postgres cliquet-empty

cliquet=$(sudo docker run -d -p 5432:5432 cliquet-empty)

...

sudo docker stop $cliquet

Environment variables¶

In order to ease deployment or testing strategies, Cliquet reads settings from environment variables, in addition to .ini files.

For example, cliquet.storage_backend is read from environment variable CLIQUET_STORAGE_BACKEND if defined, else from application .ini, else from internal defaults.

Project info¶

cliquet.project_name = project
cliquet.project_docs = https://project.rtfd.org/
# cliquet.project_version = 1.0

Feature settings¶

# Limit number of batch operations per request
# cliquet.batch_max_requests = 25

# Force pagination *(recommended)*
# cliquet.paginate_by = 200

# Custom record id generator class
# cliquet.id_generator = cliquet.storage.generators.UUID4

Disabling endpoints¶

It is possible to deactivate specific resources operations, directly in the settings.

To do so, a setting key must be defined for the disabled resources endpoints:

'cliquet.{endpoint_type}_{resource_name}_{method}_enabled'

Where: - endpoint_type is either collection or record; - resource_name is the name of the resource (by default, Cliquet uses

the name of the class);

• method is the http method (in lower case): For instance put.

For instance, to disable the PUT on records for the Mushrooms resource, the following setting should be declared in the .ini file:

# Disable article collection DELETE endpoint
cliquet.collection_article_delete_enabled = false

# Disable mushroom record PATCH endpoint
cliquet.record_mushroom_patch_enabled = false

Deployment¶

# cliquet.backoff = 10
cliquet.retry_after_seconds = 30

# cliquet.http_scheme = https
# cliquet.http_host = production.server:7777

# cliquet.eos = 2015-01-22
# cliquet.eos_message = "Client is too old"
# cliquet.eos_url = http://website/info-shutdown.html

cliquet.logging_renderer = cliquet.logs.MozillaHekaRenderer

[loggers]
keys = root

[handlers]
keys = console

[formatters]
keys = heka

[logger_root]
level = INFO
handlers = console
formatter = heka

[handler_console]
class = StreamHandler
args = (sys.stdout,)
level = NOTSET

[formatter_heka]
format = %(message)s

cliquet.statsd_url = udp://localhost:8125
# cliquet.statsd_prefix = cliquet.project_name

cliquet.newrelic_config = /location/of/newrelic.ini
cliquet.newrelic_env = prod

Storage¶

cliquet.storage_backend = cliquet.storage.redis
cliquet.storage_url = redis://localhost:6379/1

# Safety limit while fetching from storage
# cliquet.storage_max_fetch_size = 10000

# Control number of pooled connections
# cliquet.storage_pool_size = 50

See storage backend documentation for more details.

Cache¶

cliquet.cache_backend = cliquet.cache.redis
cliquet.cache_url = redis://localhost:6379/0

# Control number of pooled connections
# cliquet.storage_pool_size = 50

See cache backend documentation for more details.

It is possible to add cache control headers for each resource. The client (or proxy) will use them to cache the resource responses for a certain amount of time.

If set to 0 then the resource becomes uncacheable (no-cache).

cliquet.mushroom_cache_expires_seconds = 3600

Basically, this will add both Cache-Control: max-age=3600 and Expire: <server datetime + 1H> response headers to the GET responses.

Authentication¶

Since user identification is hashed in storage, a secret key is required in configuration:

# cliquet.userid_hmac_secret = b4c96a8692291d88fe5a97dd91846eb4

multiauth.policies = basicauth pyramid_persona ipauth

multiauth.policy.ipauth.use = pyramid_ipauth.IPAuthentictionPolicy
multiauth.policy.ipauth.ipaddrs = 192.168.0.*
multiauth.policy.ipauth.userid = LAN-user
multiauth.policy.ipauth.principals = trusted

multiauth.policies = basicauth

multiauth.policies = basicauth
multiauth.policy.basicauth.use = myproject.authn.BasicAuthPolicy

cliquet.bucket_create_principals = system.Authenticated

Application profiling¶

It is possible to profile the application while its running. This is especially useful when trying to find slowness in the application.

Enable middlewares as described here.

Update the configuration file with the following values:

cliquet.profiler_enabled = true
cliquet.profiler_dir = /tmp/profiling

Run a load test (for example):

SERVER_URL=http://localhost:8000 make bench -e

Render execution graphs using GraphViz:

sudo apt-get install graphviz

pip install gprof2dot
gprof2dot -f pstats POST.v1.batch.000176ms.1427458675.prof | dot -Tpng -o output.png

Enable middleware¶

In order to enable Cliquet middleware, wrap the application in the project main function:

def main(global_config, **settings):
    config = Configurator(settings=settings)
    cliquet.initialize(config, __version__)
    app = config.make_wsgi_app()
    return cliquet.install_middlewares(app, settings)

Initialization sequence¶

In order to control what part of Cliquet should be run during application startup, or add custom initialization steps from configuration, it is possible to change the initialization_sequence setting.

cliquet.initialization_sequence = cliquet.initialization.setup_json_serializer
                                  cliquet.initialization.setup_logging
                                  cliquet.initialization.setup_storage
                                  cliquet.initialization.setup_cache
                                  cliquet.initialization.setup_requests_scheme
                                  cliquet.initialization.setup_version_redirection
                                  cliquet.initialization.setup_deprecation
                                  cliquet.initialization.setup_authentication
                                  cliquet.initialization.setup_backoff
                                  cliquet.initialization.setup_stats

Full example¶

import colander

from cliquet import resource
from cliquet import schema
from cliquet import utils


class BookmarkSchema(resource.ResourceSchema):
    url = schema.URL()
    title = colander.SchemaNode(colander.String())
    favorite = colander.SchemaNode(colander.Boolean(), missing=False)
    device = colander.SchemaNode(colander.String(), missing='')

    class Options:
        readonly_fields = ('device',)
        unique_fields = ('url',)


@resource.register()
class Bookmark(resource.BaseResource):
    mapping = BookmarkSchema()

    def process_record(self, new, old=None):
        if new['device'] != old['device']:
            new['device'] = self.request.headers.get('User-Agent')

        return new

See the ReadingList and Kinto projects source code for real use cases.

Resource Schema¶

Override the base schema to add extra fields using the Colander API.

class Movie(ResourceSchema):
    director = colander.SchemaNode(colander.String())
    year = colander.SchemaNode(colander.Int(),
                               validator=colander.Range(min=1850))
    genre = colander.SchemaNode(colander.String(),
                                validator=colander.OneOf(['Sci-Fi', 'Comedy']))

class cliquet.schema.ResourceSchema(*arg, **kw)¶

Base resource schema, with Cliquet specific built-in options.

class Options¶

Resource schema options.

This is meant to be overriden for changing values:

class Product(ResourceSchema):
    reference = colander.SchemaNode(colander.String())

    class Options:
        unique_fields = ('reference',)

unique_fields = ()¶

Fields that must have unique values for the user collection. During records creation and modification, a conflict error will be raised if unicity is about to be violated.

readonly_fields = ()¶

Fields that cannot be updated. Values for fields will have to be provided either during record creation, through default values using missing attribute or implementing a custom logic in cliquet.resource.BaseResource.process_record().

preserve_unknown = False¶

Define if unknown fields should be preserved or not.

For example, in order to define a schema-less resource, in other words a resource that will accept any form of record, the following schema definition is enough:

class SchemaLess(ResourceSchema):
    class Options:
        preserve_unknown = True

ResourceSchema.is_readonly(field)¶

Return True if specified field name is read-only.

Parameters:	field (str) – the field name in the schema
Returns:	True if the specified field is read-only, False otherwise.
Return type:	bool

class cliquet.schema.PermissionsSchema(*args, **kwargs)¶

A permission mapping defines ACEs.

It has permission names as keys and principals as values.

{
    "write": ["fxa:af3e077eb9f5444a949ad65aa86e82ff"],
    "groups:create": ["fxa:70a9335eecfe440fa445ba752a750f3d"]
}

class cliquet.schema.TimeStamp(*arg, **kw)¶

Basic integer schema field that can be set to current server timestamp in milliseconds if no value is provided.

class Book(ResourceSchema):
    added_on = TimeStamp()
    read_on = TimeStamp(auto_now=False, missing=-1)

schema_type¶

alias of Integer

title = 'Epoch timestamp'¶

Default field title.

auto_now = True¶

Set to current server timestamp (milliseconds) if not provided.

missing = None¶

Default field value if not provided in record.

class cliquet.schema.URL(*arg, **kw)¶

String field representing a URL, with max length of 2048. This is basically a shortcut for string field with ~colander:colander.url.

class BookmarkSchema(ResourceSchema):
    url = URL()

schema_type¶

alias of String

Resource class¶

In order to customize the resource URLs or behaviour on record processing, the resource class can be extended:

class cliquet.resource.BaseResource(request, context=None)¶

Base resource class providing every endpoint.

default_viewset¶

Default cliquet.viewset.ViewSet class to use when the resource is registered.

alias of ViewSet

default_collection¶

Default cliquet.collection.Collection class to use for interacting the :module:`cliquet.storage` and :module:`cliquet.permission` backends.

alias of Collection

mapping = <cliquet.schema.ResourceSchema object at 140532609059024 (named )>¶

Schema to validate records.

get_parent_id(request)¶

Return the parent_id of the resource with regards to the current request.

Parameters:	request – The request used to create the resource.
Return type:	str

is_known_field(field)¶

Return True if field is defined in the resource mapping.

Parameters:	field (str) – Field name
Return type:	bool

collection_get()¶

Collection GET endpoint: retrieve multiple records.

Raises:	HTTPNotModified if If-None-Match header is provided and collection not modified in the interim.
Raises:	HTTPPreconditionFailed if If-Match header is provided and collection modified in the iterim.
Raises:	HTTPBadRequest if filters or sorting are invalid.

collection_post()¶

Collection POST endpoint: create a record.

If the new record conflicts against a unique field constraint, the posted record is ignored, and the existing record is returned, with a 200 status.

Raises:	HTTPPreconditionFailed if If-Match header is provided and collection modified in the iterim.

See also

Add custom behaviour by overriding cliquet.resource.BaseResource.process_record()

collection_delete()¶

Collection DELETE endpoint: delete multiple records.

Raises:	HTTPPreconditionFailed if If-Match header is provided and collection modified in the iterim.
Raises:	HTTPBadRequest if filters are invalid.

get()¶

Record GET endpoint: retrieve a record.

Raises:	HTTPNotFound if the record is not found.
Raises:	HTTPNotModified if If-None-Match header is provided and record not modified in the interim.
Raises:	HTTPPreconditionFailed if If-Match header is provided and record modified in the iterim.

put()¶

Record PUT endpoint: create or replace the provided record and return it.

Raises:	HTTPPreconditionFailed if If-Match header is provided and record modified in the iterim.

Note

If If-None-Match: * request header is provided, the PUT will succeed only if no record exists with this id.

See also

Add custom behaviour by overriding cliquet.resource.BaseResource.process_record().

patch()¶

Record PATCH endpoint: modify a record and return its new version.

If a request header Response-Behavior is set to light, only the fields whose value was changed are returned. If set to diff, only the fields whose value became different than the one provided are returned.

Raises:	HTTPNotFound if the record is not found.
Raises:	HTTPPreconditionFailed if If-Match header is provided and record modified in the iterim.

See also

Add custom behaviour by overriding cliquet.resource.BaseResource.apply_changes() or cliquet.resource.BaseResource.process_record().

delete()¶

Record DELETE endpoint: delete a record and return it.

Raises:	HTTPNotFound if the record is not found.
Raises:	HTTPPreconditionFailed if If-Match header is provided and record modified in the iterim.

process_record(new, old=None)¶

Hook for processing records before they reach storage, to introduce specific logics on fields for example.

def process_record(self, new, old=None):
    version = old['version'] if old else 0
    new['version'] = version + 1
    return new

Or add extra validation based on request:

from cliquet.errors import raise_invalid

def process_record(self, new, old=None):
    if new['browser'] not in request.headers['User-Agent']:
        raise_invalid(self.request, name='browser', error='Wrong')
    return new

Parameters:	new (dict) – the validated record to be created or updated. old (dict) – the old record to be updated, None for creation endpoints.
Returns:	the processed record.
Return type:	dict

apply_changes(record, changes)¶

Merge changes into record fields.

Note

This is used in the context of PATCH only.

Override this to control field changes at record level, for example:

def apply_changes(self, record, changes):
    # Ignore value change if inferior
    if record['position'] > changes.get('position', -1):
        changes.pop('position', None)
    return super(MyResource, self).apply_changes(record, changes)

Raises:	HTTPBadRequest if result does not comply with resource schema.
Returns:	the new record with changes applied.
Return type:	dict

from cliquet import resource


class TrackedCollection(resource.Collection):
    def create_record(self, record, parent_id=None, unique_fields=None):
        record = super(TrackedCollection, self).create_record(record,
                                                              parent_id,
                                                              unique_fields)
        trackid = index.track(record)
        record['trackid'] = trackid
        return record


class Payment(resource.BaseResource):
    def __init__(request):
        super(Payment, self).__init__(request)
        self.collection = TrackedCollection(
            storage=self.collection.storage,
            id_generator=self.collection.id_generator,
            collection_id=self.collection.collection_id,
            parent_id=self.collection.parent_id,
            auth=self.collection.auth)

Custom record ids¶

By default, records ids are UUID4 <http://en.wikipedia.org/wiki/Universally_unique_identifier>_.

A custom record ID generator can be set globally in Configuration, or at the resource level:

from cliquet import resource
from cliquet import utils
from cliquet.storage import generators


class MsecId(generators.Generator):
    def __call__(self):
        return '%s' % utils.msec_time()


@resource.register()
class Mushroom(resource.BaseResource):
    def __init__(request):
        super(Mushroom, self).__init__(request)
        self.collection.id_generator = MsecId()

Custom Usage¶

from cliquet import resource

def view(request):
    registry = request.registry

    flowers = resource.Collection(storage=registry.storage,
                                  name='app:flowers')

    flowers.create_record({'name': 'Jonquille', 'size': 30})
    flowers.create_record({'name': 'Amapola', 'size': 18})

    min_size = resource.Filter('size', 20, resource.COMPARISON.MIN)
    records, total = flowers.get_records(filters=[min_size])

    flowers.delete_record(records[0])

from cliquet import resource, DEFAULT_SETTINGS
from pyramid import Configurator


config_local = Configurator(settings=DEFAULT_SETTINGS)
config_local.add_settings({
    'cliquet.storage_backend': 'cliquet.storage.postgresql'
    'cliquet.storage_url': 'postgres://user:pass@db.server.lan:5432/dbname'
})
local = resource.Collection(storage=config_local.registry.storage,
                            parent_id='browsing',
                            name='history')

config_remote = Configurator(settings=DEFAULT_SETTINGS)
config_remote.add_settings({
    'cliquet.storage_backend': 'kinto.storage',
    'cliquet.storage_url': 'https://cloud-storage.services.mozilla.com'
})
remote = resource.Collection(storage=config_remote.registry.storage,
                             parent_id='browsing',
                             name='history',
                             auth='Basic bWF0Og==')

records, total = in remote.get_records():
for record in records:
    local.create_record(record)

Configuring a viewset¶

To use Cliquet in a basic fashion, there is not need to understand how viewsets work in full detail, but it might be useful to know how to extend the defaults.

Default viewset can be extended by passing viewset arguments to the resource.register class decorator:

from cliquet import resource


@resource.register(collection_methods=('GET',))
class Resource(resource.BaseResource):
    mapping = BookmarkSchema()

Subclassing a viewset¶

In case this isn’t enough to update the default properties, the default ViewSet class can be subclassed in a more specific viewset, and then be passed during the registration phase:

from cliquet import resource


class MyViewSet(resource.ViewSet):

    def get_service_name(self, endpoint_type, resource):
        """Returns the name of the service, depending a given type and
        resource.
        """
        # Get the resource name from an akwards location.
        return name


@resource.register(viewset=MyViewSet())
class Resource(resource.BaseResource):
    mapping = BookmarkSchema()

ViewSet class¶

In order to customize the resource URLs or permissions, the viewset class can be extended:

class cliquet.resource.ViewSet(**kwargs)¶

The default ViewSet object.

A viewset contains all the information needed to register any resource in the Cornice registry.

It provides the same features as cornice.resource(), except that it is much more flexible and extensible.

update(**kwargs)¶

Update viewset attributes with provided values.

get_view_arguments(endpoint_type, resource, method)¶

Return the Pyramid/Cornice view arguments for the given endpoint type and method.

Parameters:	endpoint_type (str) – either “collection” or “record”. resource – the resource object. method (str) – the HTTP method.

get_record_schema(resource, method)¶

Return the Cornice schema for the given method.

get_view(endpoint_type, method)¶

Return the view method name located on the resource object, for the given type and method.

• For collections, this will be “collection_{method|lower}
• For records, this will be “{method|lower}.

get_name(resource)¶

Returns the name of the resource.

get_service_name(endpoint_type, resource)¶

Returns the name of the service, depending a given type and resource.

is_endpoint_enabled(endpoint_type, resource_name, method, settings)¶

Returns if the given endpoint is enabled or not.

Uses the settings to tell so.

Backends¶

cliquet.storage_backend = kinto.storage
cliquet.storage_url = https://cloud-storage.services.mozilla.com

API¶

Implementing a custom storage backend consists in implementating the following interface:

class cliquet.storage.Filter(field, value, operator)¶

Filtering properties.

field¶

Alias for field number 0

operator¶

Alias for field number 2

value¶

Alias for field number 1

class cliquet.storage.Sort(field, direction)¶

Sorting properties.

direction¶

Alias for field number 1

field¶

Alias for field number 0

class cliquet.storage.StorageBase¶

Storage abstraction used by resource views.

It is meant to be instantiated at application startup. Any operation may raise a HTTPServiceUnavailable error if an error occurs with the underlying service.

Configuration can be changed to choose which storage backend will persist the objects.

Raises:	HTTPServiceUnavailable

initialize_schema()¶

Create every necessary objects (like tables or indices) in the backend.

This is excuted when the cliquet migrate command is ran.

flush(auth=None)¶

Remove every object from this storage.

ping(request)¶

Test that storage is operationnal.

Parameters:	request (Request) – current request object
Returns:	True is everything is ok, False otherwise.
Return type:	bool

collection_timestamp(collection_id, parent_id, auth=None)¶

Get the highest timestamp of every objects in this collection_id for this parent_id.

Note

This should take deleted objects into account.

Parameters:	collection_id (str) – the collection id. parent_id (str) – the collection parent.
Returns:	the latest timestamp of the collection.
Return type:	int

create(collection_id, parent_id, object, id_generator=None, unique_fields=None, id_field='id', modified_field='last_modified', auth=None)¶

Create the specified object in this collection_id for this parent_id. Assign the id to the object, using the attribute cliquet.resource.BaseResource.id_field.

Note

This will update the collection timestamp.

Raises:	cliquet.storage.exceptions.UnicityError
Parameters:	collection_id (str) – the collection id. parent_id (str) – the collection parent. object (dict) – the object to create.
Returns:	the newly created object.
Return type:	dict

get(collection_id, parent_id, object_id, id_field='id', modified_field='last_modified', auth=None)¶

Retrieve the object with specified object_id, or raise error if not found.

Raises:	cliquet.storage.exceptions.RecordNotFoundError
Parameters:	collection_id (str) – the collection id. parent_id (str) – the collection parent. object_id (str) – unique identifier of the object
Returns:	the object object.
Return type:	dict

update(collection_id, parent_id, object_id, object, unique_fields=None, id_field='id', modified_field='last_modified', auth=None)¶

Overwrite the object with the specified object_id.

If the specified id is not found, the object is created with the specified id.

Note

This will update the collection timestamp.

Raises:	cliquet.storage.exceptions.UnicityError
Parameters:	collection_id (str) – the collection id. parent_id (str) – the collection parent. object_id (str) – unique identifier of the object object (dict) – the object to update or create.
Returns:	the updated object.
Return type:	dict

delete(collection_id, parent_id, object_id, with_deleted=True, id_field='id', modified_field='last_modified', deleted_field='deleted', auth=None)¶

Delete the object with specified object_id, and raise error if not found.

Deleted objects must be removed from the database, but their ids and timestamps of deletion must be tracked for synchronization purposes. (See cliquet.storage.StorageBase.get_all())

Note

This will update the collection timestamp.

Raises:	cliquet.storage.exceptions.RecordNotFoundError
Parameters:	collection_id (str) – the collection id. parent_id (str) – the collection parent. object_id (str) – unique identifier of the object with_deleted (bool) – track deleted record with a tombstone
Returns:	the deleted object, with minimal set of attributes.
Return type:	dict

delete_all(collection_id, parent_id, filters=None, with_deleted=True, id_field='id', modified_field='last_modified', deleted_field='deleted', auth=None)¶

Delete all objects in this collection_id for this parent_id.

Parameters:	collection_id (str) – the collection id. parent_id (str) – the collection parent. filters (list of cliquet.storage.Filter) – Optionnally filter the objects to delete. with_deleted (bool) – track deleted records with a tombstone
Returns:	the list of deleted objects, with minimal set of attributes.
Return type:	list of dict

purge_deleted(collection_id, parent_id, before=None, id_field='id', modified_field='last_modified', auth=None)¶

Delete all deleted object tombstones in this collection_id for this parent_id.

Parameters:	collection_id (str) – the collection id. parent_id (str) – the collection parent. before (int) – Optionnal timestamp to limit deletion (exclusive)
Returns:	The number of deleted objects.
Return type:	int

get_all(collection_id, parent_id, filters=None, sorting=None, pagination_rules=None, limit=None, include_deleted=False, id_field='id', modified_field='last_modified', deleted_field='deleted', auth=None)¶

Retrieve all objects in this collection_id for this parent_id.

Parameters:	collection_id (str) – the collection id. parent_id (str) – the collection parent. filters (list of cliquet.storage.Filter) – Optionally filter the objects by their attribute. Each filter in this list is a tuple of a field, a value and a comparison (see cliquet.utils.COMPARISON). All filters are combined using AND. sorting (list of cliquet.storage.Sort) – Optionnally sort the objects by attribute. Each sort instruction in this list refers to a field and a direction (negative means descending). All sort instructions are cumulative. pagination_rules (list of list of cliquet.storage.Filter) – Optionnally paginate the list of objects. This list of rules aims to reduce the set of objects to the current page. A rule is a list of filters (see filters parameter), and all rules are combined using OR. limit (int) – Optionnally limit the number of objects to be retrieved. include_deleted (bool) – Optionnally include the deleted objects that match the filters.
Returns:	the limited list of objects, and the total number of matching objects in the collection (deleted ones excluded).
Return type:	tuple (list, integer)

Store custom data¶

Storage can be used to store arbitrary data.

data = {'subscribed': datetime.now()}
user_id = request.authenticated_userid

storage = request.registry.storage
storage.create(collection_id='__custom', parent_id='', record=data)

See the collection class to manipulate collections of records.

PostgreSQL¶

class cliquet.cache.postgresql.PostgreSQL(**kwargs)¶

Cache backend using PostgreSQL.

Enable in configuration:

cliquet.cache_backend = cliquet.cache.postgresql

Database location URI can be customized:

cliquet.cache_url = postgres://user:pass@db.server.lan:5432/dbname

Alternatively, username and password could also rely on system user ident or even specified in ~/.pgpass (see PostgreSQL documentation).

Note

Some tables and indices are created when cliquet migrate is run. This requires some privileges on the database, or some error will be raised.

Alternatively, the schema can be initialized outside the python application, using the SQL file located in cliquet/cache/postgresql/schema.sql. This allows to distinguish schema manipulation privileges from schema usage.

A threaded connection pool is enabled by default:

cliquet.cache_pool_size = 10

Note

Using a dedicated connection pool is still recommended to allow load balancing, replication or limit the number of connections used in a multi-process deployment.

Noindex:

Redis¶

class cliquet.cache.redis.Redis(*args, **kwargs)¶

Cache backend implementation using Redis.

Enable in configuration:

cliquet.cache_backend = cliquet.cache.redis

(Optional) Instance location URI can be customized:

cliquet.cache_url = redis://localhost:6379/1

A threaded connection pool is enabled by default:

cliquet.cache_pool_size = 50

Noindex:

Memory¶

class cliquet.cache.memory.Memory(*args, **kwargs)¶

Cache backend implementation in local thread memory.

Enable in configuration:

cliquet.cache_backend = cliquet.cache.memory

Noindex:

API¶

Implementing a custom cache backend consists in implementating the following interface:

class cliquet.cache.CacheBase(*args, **kwargs)¶

initialize_schema()¶

Create every necessary objects (like tables or indices) in the backend.

This is excuted when the cliquet migrate command is ran.

flush()¶

Delete every values.

ping(request)¶

Test that cache backend is operationnal.

Parameters:	request (Request) – current request object
Returns:	True is everything is ok, False otherwise.
Return type:	bool

ttl(key)¶

Obtain the expiration value of the specified key.

Parameters:	key (str) – key
Returns:	number of seconds or negative if no TTL.
Return type:	float

expire(key, ttl)¶

Set the expiration value ttl for the specified key.

Parameters:	key (str) – key ttl (float) – number of seconds

set(key, value, ttl=None)¶

Store a value with the specified key. If ttl is provided, set an expiration value.

Parameters:	key (str) – key value (str) – value to store ttl (float) – expire after number of seconds

get(key)¶

Obtain the value of the specified key.

Parameters:	key (str) – key
Returns:	the stored value or None if missing.
Return type:	str

delete(key)¶

Delete the value of the specified key.

Parameters:	key (str) – key

Glossary¶

Authorization isn’t complicated, but requires the introduction of a few terms so that explanations are easier to follow:

Object:

The data that is stored into Cliquet. objects usually match the resources you defined; For one resource there are two objects: resource’s collection and resource’s records.

Principal:

An entity that can be authenticated. principals can be individual people, computers, services, or any group of such things.

Permission:

An action that can be authorized or denied. read, write, create are permissions.

Access Control Entity (ACE):

An association of a principal, an object and a permission. For instance, (Alexis, article, write).

Access Control List (ACL):

A list of Access Control Entities (ACE).

Overview¶

By default, the resources defined by Cliquet are public, and records are isolated by user. But it is also possible to define protected resources, which will required the user to have access to the requested resource.

from cliquet import authorization
from cliquet import resource


@resource.register(factory=authorization.RouteFactory)
class Toadstool(resource.ProtectedResource):
    mapping = MushroomSchema()

In this example, a route factory is registered. Route factories are explained in more details below.

A protected resource, in addition to the data property of request / responses, takes a permissions property which contains the list of principals that are allowed to access or modify the current object.

During the creation of the object, the permissions property is stored in the permission backend, and upon access, it checks the current principal has access the the object, with the correct permission.

Route factory¶

The route factory decides which permission is required to access one resource or another. Here is a summary of the permissions that are defined by the default route factory Cliquet defines:

Route factories are best described in the pyramid documentation

class cliquet.authorization.RouteFactory(request)¶

Authorization policy¶

Upon access, the authorization policy is asked if any of the current list of principals has access to the current resource. By default, the authorization policy Cliquet checks in the permission backend for the current object.

It is possible to extend this behavior, for instance if there is an inheritance tree between the defined resources (some ACEs should give access to its child objects).

In case the application should define its own inheritance tree, it should also define its own authorization policy.

To do so, subclass the default AuthorizationPolicy and add a specific get_bound_permission method.

from cliquet import authorization
from pyramid.security import IAuthorizationPolicy
from zope.interface import implementer

@implementer(IAuthorizationPolicy)
class AuthorizationPolicy(authorization.AuthorizationPolicy):
    def get_bound_permissions(self, *args, **kwargs):
    """Callable that takes an object ID and a permission and returns
    a list of tuples (<object id>, <permission>)."""
        return build_permissions_set(*args, **kwargs)

class cliquet.authorization.AuthorizationPolicy¶

Permissions backend¶

The ACLs are stored in a permission backend. Currently, permission backends exists for Redis and PostgreSQL, as well as a in memory one. It is of course possible to add you own permission backend, if you whish to store your permissions related data in a different database.

class cliquet.permission.PermissionBase(*args, **kwargs)¶

initialize_schema()¶

Create every necessary objects (like tables or indices) in the backend.

This is excuted with the cliquet migrate command.

flush()¶

Delete all data stored in the permission backend.

add_user_principal(user_id, principal)¶

Add an additional principal to a user.

Parameters:	user_id (str) – The user_id to add the principal to. principal (str) – The principal to add.

remove_user_principal(user_id, principal)¶

Remove an additional principal from a user.

Parameters:	user_id (str) – The user_id to remove the principal to. principal (str) – The principal to remove.

user_principals(user_id)¶

Return the set of additionnal principals given to a user.

Parameters:	user_id (str) – The user_id to get the list of groups for.
Returns:	The list of group principals the user is in.
Return type:	set

add_principal_to_ace(object_id, permission, principal)¶

Add a principal to an Access Control Entry.

Parameters:	object_id (str) – The object to add the permission principal to. permission (str) – The permission to add the principal to. principal (str) – The principal to add to the ACE.

remove_principal_from_ace(object_id, permission, principal)¶

Remove a principal to an Access Control Entry.

Parameters:	object_id (str) – The object to remove the permission principal to. permission (str) – The permission that should be removed. principal (str) – The principal to remove to the ACE.

object_permission_principals(object_id, permission)¶

Return the set of principals of a bound permission (unbound permission + object id).

Parameters:	object_id (str) – The object_id the permission is set to. permission (str) – The permission to query.
Returns:	The list of user principals
Return type:	set

principals_accessible_objects(principals, permission, object_id_match=None, get_bound_permissions=None)¶

Return the list of objects id where the specified principals have the specified permission.

Parameters:	principal (list) – List of user principals permission (str) – The permission to query. object_id_match (str) – Filter object ids based on a pattern (e.g. '*articles*'). get_bound_permissions (function) – The methods to call in order to generate the list of permission to verify against. (ie: if you can write, you can read)
Returns:	The list of object ids
Return type:	set

object_permission_authorized_principals(object_id, permission, get_bound_permissions=None)¶

Return the full set of authorized principals for a given permission + object (bound permission).

Parameters:	object_id (str) – The object_id the permission is set to. permission (str) – The permission to query. get_bound_permissions (function) – The methods to call in order to generate the list of permission to verify against. (ie: if you can write, you can read)
Returns:	The list of user principals
Return type:	set

check_permission(object_id, permission, principals, get_bound_permissions=None)¶

Test if a principal set have got a permission on an object.

Parameters:	object_id (str) – The identifier of the object concerned by the permission. permission (str) – The permission to test. principals (set) – A set of user principals to test the permission against. get_bound_permissions (function) – The method to call in order to generate the set of permission to verify against. (ie: if you can write, you can read)

ping(request)¶

Test the permission backend is operationnal.

Parameters:	request (Request) – current request object
Returns:	True is everything is ok, False otherwise.
Return type:	bool

object_permissions(object_id, permissions=None)¶

Return the set of principals for each object permission.

Parameters:	object_id (str) – The object_id the permission is set to. permissions (list) – List of permissions to retrieve. If not define will try to find them all.
Returns:	The dictionnary with the list of user principals for each object permissions
Return type:	dict

replace_object_permissions(object_id, permissions)¶

Replace given object permissions.

Parameters:	object_id (str) – The object to replace permissions to. permissions (str) – The permissions dict to replace.

delete_object_permissions(*object_id_list)¶

Delete all listed object permissions.

Parameters:	object_id (str) – Remove given objects permissions.