PREMIS Event Service Documentation¶

PREMIS Event Service¶

PREMIS Event Service is a Django application for managing PREMIS Events in a structured, centralized, and searchable manner.

$ pip install docker-compose

$ git clone https://github.com/unt-libraries/django-premis-event-service.git
$ cd django-premis-event-service

# start the app
$ docker-compose up -d

# optional: add a superuser in order to login to the admin interface
$ docker-compose run --rm app ./manage.py createsuperuser

# stop the app
$ docker-compose stop

# remove the app container
$ docker-compose rm app

# rebuild the app container
$ docker-compose build app

# start the app
$ docker-compose up -d

$ docker-compose run --rm app tox

$ docker-compose run --rm app py.test

Events¶

A standard PREMIS event encoded as XML looks something like the following:

<premis:event xmlns:premis="info:lc/xmlns/premis-v2">
    <premis:eventType>
        http://purl.org/net/meta/vocabularies/preservationEvents/#MigrateSuccess
    </premis:eventType>
    <premis:linkingAgentIdentifier>
        <premis:linkingAgentIdentifierValue>
            http://metaarchive.org/agent/metaMigrateSuccess
        </premis:linkingAgentIdentifierValue>
        <premis:linkingAgentIdentifierType>
            http://purl.org/net/meta/vocabularies/identifier-qualifiers/#URL
        </premis:linkingAgentIdentifierType>
    </premis:linkingAgentIdentifier>
    <premis:eventIdentifier>
        <premis:eventIdentifierType>
            http://purl.org/net/meta/vocabularies/identifier-qualifiers/#UUID
        </premis:eventIdentifierType>
        <premis:eventIdentifierValue>
            e8ee3b1a8c9e4a5daf0a1e0446383d90
        </premis:eventIdentifierValue>
    </premis:eventIdentifier>
    <premis:eventDetail>
        Verification of data at /data3/meta-r1-003_dropbox/meta106w
    </premis:eventDetail>
    <premis:eventOutcomeInformation>
        <premis:eventOutcomeDetail>
            Checking content after cache server migration
        </premis:eventOutcomeDetail>
        <premis:eventOutcome>
            http://purl.org/net/meta/vocabularies/eventOutcomes/#success
        </premis:eventOutcome>
    </premis:eventOutcomeInformation>
    <premis:eventDateTime>
        2011-01-25 16:39:49
    </premis:eventDateTime>
    <premis:linkingObjectIdentifier>
        <premis:linkingObjectIdentifierType>
            http://purl.org/net/meta/vocabularies/identifier-qualifiers/#ARK
        </premis:linkingObjectIdentifierType>
        <premis:linkingObjectIdentifierValue>
            ark:/67531/meta106w
        </premis:linkingObjectIdentifierValue>
        <premis:linkingObjectRole/>
    </premis:linkingObjectIdentifier>
</premis:event>

This is a lot at first glance, but the pieces are more or less logical. The relevant things that a given PREMIS event record keeps track of are the following:

• Event Identifier - This is a unique identifier assigned to every event when it is entered into the system. This is what is used to reference given event.
• Event Type - This is an arbitrary value to categorize the kind of event we’re logging. Examples might include fixity checking, virus scanning or replication.
• Event Time - This is a timestamp for when the event itself occurred.
• Event Added - This is a timestamp for when the event was logged.
• Event Outcome - This is the simple description of the outcome. Usually something like “pass” or “fail”.
• Outcome Details - A more detailed record of the outcome. Perhaps output from a secondary program might go here.
• Agent - This is the identifier for the agent that initiated the event. An agent can be anything, from a person, to an institution, to a program. The PREMIS event service will also allow you to track agent entries as well.
• Linked Objects - These are identifiers for relevant objects that the event is associated with. If your system uses object identifiers, you could put those identifiers here when an event pertains to them.

It is important to note that most of the values that you use in a given PREMIS event record are arbitrary. You decide on your own values and vocabularies, and use what makes sense to you. It doesn’t enforce any sort of constraints as far as that goes. The service is responsible for indexing all PREMIS events sent to it and providing retrieval for them. Basic retrieval is on a per-identifier basis, but it is plausible to assume that you may wish to request events based on date added, agent used, event type, event outcome, or a combination of these factors.

Agents¶

The PREMIS metadata specification defines a separate spec for agents that looks like the following:

<?xml version="1.0"?>
<premis:agent xmlns:premis="info:lc/xmlns/premis-v2">
    <premis:agentIdentifier>
        <premis:agentIdentifierValue>
            MigrateSuccess
        </premis:agentIdentifierValue>
        <premis:agentIdentifierType>
            FDsys:agent
        </premis:agentIdentifierType>
    </premis:agentIdentifier>
    <premis:agentName>
        http://institution.edu/agent/metaMigrateSuccess
    </premis:agentName>
    <premis:agentType>softw</premis:agentType>
</premis:agent>

As you can see from the above example, the agent’s identifier above corresponds with the agent in the event example. You are able to create and register agents through the administrative panel on the PREMIS microservice; see the Administration section to learn how.

Dependencies¶

• Python 2.7.x
• Django 1.8
• libxml2-dev libxslt-dev
• Django Admin - django.contrib.admin

Important security warning¶

This application does not attempt to authenticate requests or differentiate between clients in any way – even for write and edit operations via the API. Do not simply expose the application to the public in your server configuration. Instead, use a network firewall to whitelist the server to authorized clients, or use a web server configuration directive (such as Apache’s <LimitExcept GET>) to set up who is allowed to POST/PUT/DELETE events.

Install¶

1. Install the package.

   $ pip install django-premis-event-service
   

2. Add premis_event_service to your INSTALLED_APPS. Be sure to add django.contrib.admin if it is not already present.

   INSTALLED_APPS = (
       'django.contrib.admin',
       # ...
       'premis_event_service',
   )
   

4. Include the URLs.

   urlpatterns = [
       url(r'', include('premis_event_service.urls'))
       # ...
       url(r'^admin/', include(admin.site.urls)),
   ]
   

5. Migrate the database.

   $ python manage.py migrate
   

6. Continue to Administration to begin setting up Agents.

Mandatory Configuration¶

1. Update your INSTALLED_APPS setting as follows:

   INSTALLED_APPS = (
       ...
       'django.contrib.humanize',
       'premis_event_service',
   )
   

2. Make sure you have a TEMPLATE_CONTEXT_PROCESSORS setting defined containing at least the entries shown below:

   TEMPLATE_CONTEXT_PROCESSORS = (
       'django.contrib.auth.context_processors.auth',
       'django.core.context_processors.debug',
       'django.core.context_processors.i18n',
       'django.core.context_processors.media',
       'django.core.context_processors.request',
   )
   

3. In your MIDDLEWARE_CLASSES setting, remove or comment out the CsrfViewMiddleware entry:

   MIDDLEWARE_CLASSES = (
       ...
       #'django.middleware.csrf.CsrfViewMiddleware',
       ...
   )
   

4. Add a MAINTENANCE_MSG setting at the bottom of the file:

   MAINTENANCE_MSG = ''  # Message to show during maintenance
   

Customizing the Controlled Vocabulary¶

EVENT_OUTCOME_CHOICES = (
    ('', 'None'),
    ('http://purl.org/net/untl/vocabularies/eventOutcomes/#success', 'Success'),
    ('http://purl.org/net/untl/vocabularies/eventOutcomes/#failure', 'Failure'),
)
EVENT_TYPE_CHOICES = (
    ('', 'None'),
    ('http://id.loc.gov/vocabulary/preservation/eventType/fix', 'Fixity Check'),
    ('http://id.loc.gov/vocabulary/preservation/eventType/rep', 'Replication'),
    ('http://id.loc.gov/vocabulary/preservation/eventType/ing', 'Ingestion'),
    ('http://id.loc.gov/vocabulary/preservation/eventType/mig', 'Migration'),
)

Manage User Accounts¶

At this point, you might only have one user account in your system (which is the superuser account created when you ran python manage.py syncdb during installation).

To manage or create other user accounts, do the following:

1. Visit the Django admin interface (http://[host]/admin/) in a web browser.
2. Log in using your superuser account (if you haven’t already).
3. Click Users. This takes you to the list of Users.
4. Click the Add user button near the top-right corner of the page.
5. Fill and submit the form.

Keep in mind that any account needing the ability to also administer user accounts using the admin interface will need to be given “superuser” status.

Create an Agent¶

Every event stored in the Event Service must be associated with an Agent. Agents merely represent entities that produce events. In many cases these are software processes (e.g. a web application or a script), but an agent can also be a person, an institution, or anything else.

To create a new agent (or to manage existing ones), do the following:

1. Visit the Django admin interface (http://[host]/admin/) in a web browser.
2. Log in using your superuser account (if you haven’t already).
3. Click Agents. This takes you to the list of Agents, which will be empty at first.
4. Click the Add agent button near the top-right corner of the page.
5. Fill and submit the form.

Create as many agents as you have a need for.

Events¶

Agents¶

Introduction¶

The PREMIS Event Service uses REST to handle the message passing between client and server. To better provide a standard set of conventions for this, we have elected to follow the AtomPub protocol for POSTing and GETing events from the system. The base unit for AtomPub is the Atom “entry” tag, which is what gets sent back and forth. The actual PREMIS metadata is embedded in the entry’s “content” tag. There is a lot more to AtomPub than that, but for the purpose of this document, it is helpful to just view the Atom entry as an “envelope” for the PREMIS XML.

API URL Structure¶

APIs for communicating with the Event Service programmatically are located under the /APP/ URL tree:

Example¶

Imagine that we have just completed a mass server-to-server data copy, and as part of that migrated data we have a directory called object_123/ which contains a collection of files that represents a migrated digital object. This digital object conveniently enough, has the localidentifier (for our system) of object_123.

We have a script validate_object that we can run on our objects to make certain that the files match a previously stored fixity digest and are intact after this migration. In this case, we wish to log an event of the validation in order to properly track our actions. To begin with, we run the validate_object script on our directory and wait for it to run.

Let’s say that it runs and comes back with an error: Validation of object_123/ failed Details: Generated sum for object_123/data/pic_002.tif does not match stored value. Obviously, we have to deal with the problem at some point, but right now we just want to log an event that will accurately reflect the results of the script. So, we create a PREMIS event XML tree:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

<premis:event xmlns:premis="info:lc/xmlns/premis-v2">
    <premis:eventType>
        validateObject
    </premis:eventType>
    <premis:linkingAgentIdentifier>
        <premis:linkingAgentIdentifierValue>
            validateObjectScript
        </premis:linkingAgentIdentifierValue>
        <premis:linkingAgentIdentifierType>
            Program
        </premis:linkingAgentIdentifierType>
    </premis:linkingAgentIdentifier>
    <premis:eventIdentifier>
        <premis:eventIdentifierType>
            TEMP
        </premis:eventIdentifierType>
        <premis:eventIdentifierValue>
            TEMP
        </premis:eventIdentifierValue>
    </premis:eventIdentifier>
    <premis:eventDetail>Validation of object
        object_123
    </premis:eventDetail>
    <premis:eventOutcomeInformation>
        <premis:eventOutcomeDetail>
            Generated sum for object_123/data/pic_002.tif does not match stored value
        </premis:eventOutcomeDetail>
        <premis:eventOutcome>
            Failure
        </premis:eventOutcome>
    </premis:eventOutcomeInformation>
    <premis:eventDateTime>
        2011-01-27 16:39:49
    </premis:eventDateTime>
    <premis:linkingObjectIdentifier>
        <premis:linkingObjectIdentifierType>
            Local Identifier
        </premis:linkingObjectIdentifierType>
        <premis:linkingObjectIdentifierValue>
            object_123
        </premis:linkingObjectIdentifierValue>
        <premis:linkingObjectRole />
    </premis:linkingObjectIdentifier>
</premis:event>

As you can see, the values chosen for the tags in the PREMIS event XML are arbitrary, and it is the responsibility of the user to select something that makes sense in the context of their organization. One thing to note is that the values for the eventIdentifierType and eventIdentifierValue will be overwritten, because the Event Service manages the event identifiers, and assigns new ones upon ingest.

Now, in order to send the event to the Event Service, it must be wrapped in an Atom entry, so the following Atom wrapper XML tree is created:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

<entry xmlns="http://www.w3.org/2005/Atom">
    <title>PREMIS event entry for object_123</title>
    <id>PREMIS event entry for object_123</id>
    <updated>2011-­‐01-­‐27T16:40:30Z</updated>
    <author>
        <name>Object Verification Script</name>
    </author>
    <content type="application/xml">
        <premis:event xmlns:premis="http://www.loc.gov/standards/premis/v1">
            ...
        </premis:event>
    </content>
</entry>

(With the previously-generated PREMIS XML going inside of the “content” tag.)

Now that the entry is generated and wrapped in a valid Atom document, it is ready for upload. In order to do this, we POST the Atom XML to the /APP/event/ URL.

When the Event Service receives the POST, it reads the content and parses the XML. If it finds a valid XML PREMIS event document, it will assign the event an identifier, index the values and save them, and then generate a return document, also wrapped in an Atom entry. It will look something like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

<entry xmlns="http://www.w3.org/2005/Atom">
    <title>bfa2cf2c2a4f11e089b3005056935974</title>
    <id>bfa2cf2c2a4f11e089b3-005056935974</id>
    <updated>2011-01-27T16:40:30Z</updated>
    <author>
        <name>Object Verification Script</name>
    </author>
    <content type="application/xml">
        <premis:event xmlns:premis="http://www.loc.gov/standards/premis/v1">
            <premis:eventType>validateObject</premis:eventType>
            <premis:linkingAgentIdentifier>
                <premis:linkingAgentIdentifierValue>
                    validateObjectScript
                </premis:linkingAgentIdentifierValue>
                <premis:linkingAgentIdentifierType>
                    Program
                </premis:linkingAgentIdentifierType>
            </premis:linkingAgentIdentifier>
            <premis:eventIdentifier>
                <premis:eventIdentifierType>
                    UUID
                </premis:eventIdentifierType>
                <premis:eventIdentifierValue>
                    bfa2cf2c2a4f11e089b3-005056935974
                </premis:eventIdentifierValue>
            </premis:eventIdentifier>
            ...
        </premis:event>
    </content>
</entry>

As you can see, the identifier has been changed to a UUID, which, in this case, is bfa2cf2c2a4f11e089b3-­‐005056935974. This identifier is unique and will be what the microservice will use to refer to that individual event in the future.

If the POST is successful, the updated record will be returned, along with a status of “200”. If the status is something else, there was an error, and the event cannot be considered to have been reliably recorded.

Later, when we (or, perhaps, another script) wish to review the event to find out what went wrong with the file validation, we would access it by sending an HTTP GET request to /APP/event/bfa2cf2c2a4f11e089b3-005056935974, which would return an Atom entry containing the final event record, which we could then analyze and use for whatever purposes desired.

Project Structure¶

The PREMIS Event Service is structured as a common Python project, providing a Python package named premis_event_service which is a Django app:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

premis_event_service/ # The Django app itself. Technically a Python package.
    admin.py          ## Customizes the Django admin interface
    coda/             ## Some supporting modules we need for some tasks
        anvl.py
        bagatom.py
        __init__.py   ### Marks this directory as a valid Python package
        util.py
    forms.py          ## Form processing code
    helpy.py
    __init__.py       ## Marks this directory as a valid Python package
    models.py         ## Database model definitions
    populator.py
    presentation.py
    settings.py       ## Wrapper for project settings file with custom defaults
    templates/        ## HTML templates
    urls.py           ## URL routing patterns
    views.py          ## View generation code

If you’re not sure where to look for something, urls.py is usually the best place to start. There you’ll find a list of every URL pattern handled by the application, along with its corresponding view (found in views.py) and arguments.

Models¶

Models define the data objects Django keeps in its database. The PREMIS Event Service defines these three:

• Event - Represents an event.
• Agent - Represents an agent you’ve defined using the Django admin interface.
• LinkObject - Contains an identifier for an object in your preservation workflow. Exists for the purpose of relating multiple events that pertain to the same object.

See premis_event_service/models.py for the full definitions to these models.

Views¶

View are functions (or sometimes classes) that Django calls upon to generate the result of a request. Usually this just means rendering some HTML from a template and serving it, but sometimes this involves form processing and API interactions as well. Django decides which view to run based on what’s defined in urls.py.

See premis_event_service/views.py for the full source code to all the views provided by the Event Service.