Code Highlighting API

This example demonstrates creating a REST API using a Resource with some form validation on the input. We’re going to provide a simple wrapper around the awesome pygments library, to create the Web API for a simple pastebin.

Note

A live sandbox instance of this API is available at http://rest.ep.io/pygments/

You can browse the API using a web browser, or from the command line:

curl -X GET http://rest.ep.io/pygments/ -H 'Accept: text/plain'

URL configuration

We’ll need two resources:

• A resource which represents the root of the API.
• A resource which represents an instance of a highlighted snippet.

urls.py

from django.conf.urls.defaults import patterns, url
from pygments_api.views import PygmentsRoot, PygmentsInstance

urlpatterns = patterns('',
    url(r'^$', PygmentsRoot.as_view(), name='pygments-root'),
    url(r'^([a-zA-Z0-9-]+)/$', PygmentsInstance.as_view(), name='pygments-instance'),
)

Form validation

We’ll now add a form to specify what input fields are required when creating a new highlighted code snippet. This will include:

• The code text itself.
• An optional title for the code.
• A flag to determine if line numbers should be included.
• Which programming language to interpret the code snippet as.
• Which output style to use for the highlighting.

forms.py

from django import forms

from pygments.lexers import get_all_lexers
from pygments.styles import get_all_styles

LEXER_CHOICES = sorted([(item[1][0], item[0]) for item in get_all_lexers()])
STYLE_CHOICES = sorted((item, item) for item in list(get_all_styles()))

class PygmentsForm(forms.Form):
    """A simple form with some of the most important pygments settings.
    The code to be highlighted can be specified either in a text field, or by URL.
    We do some additional form validation to ensure clients see helpful error responses."""

    code = forms.CharField(widget=forms.Textarea,
                           label='Code Text',
                           max_length=1000000,
                           help_text='(Copy and paste the code text here.)')
    title = forms.CharField(required=False,
                            help_text='(Optional)',
                            max_length=100)
    linenos = forms.BooleanField(label='Show Line Numbers',
                                 required=False)
    lexer = forms.ChoiceField(choices=LEXER_CHOICES,
                              initial='python')
    style = forms.ChoiceField(choices=STYLE_CHOICES,
                              initial='friendly')

Creating the resources

We’ll need to define 3 resource handling methods on our resources.

• PygmentsRoot.get() method, which lists all the existing snippets.
• PygmentsRoot.post() method, which creates new snippets.
• PygmentsInstance.get() method, which returns existing snippets.

And set a number of attributes on our resources.

• Set the allowed_methods and anon_allowed_methods attributes on both resources allowing for full unauthenticated access.
• Set the form attribute on the PygmentsRoot resource, to give us input validation when we create snippets.
• Set the emitters attribute on the PygmentsInstance resource, so that

views.py

from __future__ import with_statement  # for python 2.5
from django.conf import settings

from djangorestframework.resources import FormResource
from djangorestframework.response import Response
from djangorestframework.renderers import BaseRenderer
from djangorestframework.reverse import reverse
from djangorestframework.views import View
from djangorestframework import status

from pygments.formatters import HtmlFormatter
from pygments.lexers import get_lexer_by_name
from pygments import highlight

from forms import PygmentsForm

import os
import uuid
import operator

# We need somewhere to store the code snippets that we highlight
HIGHLIGHTED_CODE_DIR = os.path.join(settings.MEDIA_ROOT, 'pygments')
MAX_FILES = 10

if not os.path.exists(HIGHLIGHTED_CODE_DIR):
    os.makedirs(HIGHLIGHTED_CODE_DIR)


def list_dir_sorted_by_ctime(dir):
    """
    Return a list of files sorted by creation time
    """
    filepaths = [os.path.join(dir, file)
                 for file in os.listdir(dir)
                 if not file.startswith('.')]
    ctimes = [(path, os.path.getctime(path)) for path in filepaths]
    ctimes = sorted(ctimes, key=operator.itemgetter(1), reverse=False)
    return [filepath for filepath, ctime in  ctimes]


def remove_oldest_files(dir, max_files):
    """
    Remove the oldest files in a directory 'dir', leaving at most 'max_files' remaining.
    We use this to limit the number of resources in the sandbox.
    """
    [os.remove(path) for path in list_dir_sorted_by_ctime(dir)[max_files:]]


class HTMLRenderer(BaseRenderer):
    """
    Basic renderer which just returns the content without any further serialization.
    """
    media_type = 'text/html'


class PygmentsRoot(View):
    """
    This example demonstrates a simple RESTful Web API around the awesome pygments library.
    This top level resource is used to create highlighted code snippets, and to list all the existing code snippets.
    """
    form = PygmentsForm

    def get(self, request):
        """
        Return a list of all currently existing snippets.
        """
        unique_ids = [os.path.split(f)[1] for f in list_dir_sorted_by_ctime(HIGHLIGHTED_CODE_DIR)]
        return [reverse('pygments-instance', request=request, args=[unique_id]) for unique_id in unique_ids]

    def post(self, request):
        """
        Create a new highlighed snippet and return it's location.
        For the purposes of the sandbox example, also ensure we delete the oldest snippets if we have > MAX_FILES.
        """
        unique_id = str(uuid.uuid1())
        pathname = os.path.join(HIGHLIGHTED_CODE_DIR, unique_id)

        lexer = get_lexer_by_name(self.CONTENT['lexer'])
        linenos = 'table' if self.CONTENT['linenos'] else False
        options = {'title': self.CONTENT['title']} if self.CONTENT['title'] else {}
        formatter = HtmlFormatter(style=self.CONTENT['style'], linenos=linenos, full=True, **options)

        with open(pathname, 'w') as outfile:
            highlight(self.CONTENT['code'], lexer, formatter, outfile)

        remove_oldest_files(HIGHLIGHTED_CODE_DIR, MAX_FILES)

        return Response(status.HTTP_201_CREATED, headers={'Location': reverse('pygments-instance', request=request, args=[unique_id])})


class PygmentsInstance(View):
    """
    Simply return the stored highlighted HTML file with the correct mime type.
    This Resource only renders HTML and uses a standard HTML renderer rather than the renderers.DocumentingHTMLRenderer class.
    """
    renderers = (HTMLRenderer,)

    def get(self, request, unique_id):
        """
        Return the highlighted snippet.
        """
        pathname = os.path.join(HIGHLIGHTED_CODE_DIR, unique_id)
        if not os.path.exists(pathname):
            return Response(status.HTTP_404_NOT_FOUND)
        return open(pathname, 'r').read()

    def delete(self, request, unique_id):
        """
        Delete the highlighted snippet.
        """
        pathname = os.path.join(HIGHLIGHTED_CODE_DIR, unique_id)
        if not os.path.exists(pathname):
            return Response(status.HTTP_404_NOT_FOUND)
        return os.remove(pathname)

Completed

And we’re done. We now have an API that is:

• Browseable. The API supports media types for both programmatic and human access, and can be accessed either via a browser or from the command line.
• Self describing. The API serves as it’s own documentation.
• Well connected. The API can be accessed fully by traversal from the initial URL. Clients never need to construct URLs themselves.

Our API also supports multiple media types for both input and output, and applies sensible input validation in all cases.

For example if we make a POST request using form input:

bash: curl -X POST --data 'code=print "hello, world!"' --data 'style=foobar' -H 'X-Requested-With: XMLHttpRequest' http://rest.ep.io/pygments/
{"detail": {"style": ["Select a valid choice. foobar is not one of the available choices."], "lexer": ["This field is required."]}}

Or if we make the same request using JSON:

bash: curl -X POST --data-binary '{"code":"print \"hello, world!\"", "style":"foobar"}' -H 'Content-Type: application/json' -H 'X-Requested-With: XMLHttpRequest' http://rest.ep.io/pygments/
{"detail": {"style": ["Select a valid choice. foobar is not one of the available choices."], "lexer": ["This field is required."]}}