Download SuperTagging¶

There are a couple ways for you to get Django-SuperTagging,

1. Clone the git repository from GitHub
2. Use pip to install it from PyPI

pip install supertagging

Dependencies¶

• simplejson (Required)
• freebase (Optional)

Add SuperTagging to your project¶

Add to INSTALLED_APPS

INSTALLED_APPS = (
    ...
    supertagging,
    ...
)

Run syncdb:

>>> ./manage.py syncdb

Create basic settings¶

In your settings.py file:

SUPERTAGGING_SETTINGS = {
    'ENABLED': True,
    'DEBUG': True,
}

Setting up OpenCalais API¶

Go to OpenCalais‘s website and register for an api key, and in your settings.py file, alter SUPERTAGGING_SETTINGS:

SUPERTAGGING_SETTINGS = {
    'ENABLED': True,
    'DEBUG': True,
    'OPEN_CALAIS': {
        'API_KEY': 'YOUR_API_KEY',
    }
}

Setting up models to be tagged¶

You will need to decide which models and which fields in those models you will want SuperTagging to mark for tagging:

SUPERTAGGING_SETTINGS = {
    'ENABLED': True,
    'DEBUG': True,
    'OPEN_CALAIS': {
        'API_KEY': 'YOUR_API_KEY',
    },
    'WATCHED_FIELDS': {
        'stories.story': {
            'fields':[
                {'name': 'body'},
            ],
        },
    },
}

The code above tells SuperTagging to tag the body field of model stories.story. We can specify any number of fields and models as well.

SUPERTAGGING_SETTINGS = {
    'ENABLED': True,
    'DEBUG': True,
    'OPEN_CALAIS': {
        'API_KEY': 'YOUR_API_KEY',
    },
    'WATCHED_FIELDS': {
        'stories.story': {
            'fields':[
                    {'name': 'body'},
                    {'name': 'tease'},
                    {'name': 'kicker'},
            ],
        },
        'media.image': {
            'fields':[
                    {'name': 'caption'},
                    {'name': 'description'},
            ],
        }
    },
}

View WATCHED_FIELDS for more information.

Set up automatic processing¶

Finally, add:

SUPERTAGGING_SETTINGS = {
    'ENABLED': True,
    'DEBUG': True,
    'OPEN_CALAIS': {
        'API_KEY': 'YOUR_API_KEY',
    },
    'WATCHED_FIELDS': {
        'stories.story': {
            'fields':[
                {'name': 'body'},
            ],
        },
    },
    'AUTO_PROCESS': True,
}

Post save and post delete signals will be connected to the models specified in WATCHED_FIELDS. Visit Settings to view more details about the SuperTagging settings

View the complete list of Settings

Conclusion¶

That is all that is needed to get SuperTagging to start tagging your data. Upon saving a instance of one of the models specified in SUPERTAGGING_MODULES, the field(s) data will be sent to OpenCalais for processing.

Next step: View the Real World Example section of how The Washington Times has SuperTagging setup.

Our current SuperTagging settings¶

This first group of settings are the general settings.

SUPERTAGGING_DEBUG = False
SUPERTAGGING_ENABLED = True
SUPERTAGGING_AUTO_PROCESS = True
SUPERTAGGING_CALAIS_API_KEY = '...'
SUPERTAGGING_USE_QUEUE = True

Explanation:

We have USE_QUEUE set to True. Together with AUTO_PROCESS set to True, the objects are saved into the SuperTagQueue model for later processing. We run the management command provided by SuperTagging every 5 minutes.

This next group is what we call the processing settings for SuperTagging.

SUPERTAGGING_PROCESS_RELATIONS = True
SUPERTAGGING_PROCESS_TOPICS = True
SUPERTAGGING_RESOLVE_PROPERTY_KEYS = True
SUPERTAGGING_ONLY_NON_TAGGED_OBJECTS = False

SUPERTAGGING_MIN_RELEVANCE = 200

SUPERTAGGING_USE_FREEBASE = True

Explanation:

We process pretty much all OpenCalais gives us, that includes the Events/Facts (relations) and Topics, which are just tags but with no meta data.

We try to convert Calais ID’s to tag names (SUPERTAGGING_RESOLVE_PROPERTY_KEYS = True)

We tag all objects every time they are saved (SUPERTAGGING_ONLY_NON_TAGGED_OBJECTS = False). This is so we can efficiently markup up the content without worry of data being stale.

We only accept tags that have at least 200 relevance

We use Freebase to disambiguate the tag names.

Next group is the markup settings

SUPERTAGGING_MARKUP = True
SUPERTAGGING_MARKUP_EXCLUDES = ['his', 'her', 'he', 'she', 'him',]
SUPERTAGGING_MARKUP_CONTENT_CACHE_TIMEOUT = 3600
SUPERTAGGING_MARKUP_FIELD_SUFFIX = "tagged"

Explanation:

We markup all the objects that get processed by OpenCalais, below is an example of content after it has been marked up.

<p><a href="/topics/charles-b-rangel/">Mr. Rangel</a>, the longtime top Democrat on the House
<a href="/topics/ways-and-means-committee/">Ways and Means Committee</a> who stepped down under
pressure in March, has been under investigation by the panel for two years. At issue is a
plethora of subjects, including <a href="/topics/charles-b-rangel/">Mr. Rangel</a>'s ownership
of several rent-controlled apartments in New York; his failure to report $75,000 in earnings
on tax returns; and use of his official position to raise money for the
<a href="/topics/charles-b-rangel-center/">Charles B. Rangel Center</a> for 

<a href="/topics/public-service/">Public Service</a> at
<a href="/topics/city-college-of-new-york/">City College of New York</a>.</p>

We exclude some markup values OpenCalais returns. OpenCalais returns exactly where, in the content sent over, the reference to the tags is. This will not just be the exact tag name but also references to the tag such as his, her, she. We found that there was just too many links in the content and needed a way to limit it. So combinations of SUPERTAGGING_MIN_RELEVANCE setting and SUPERTAGGING_MARKUP_EXCLUDES provided just that.

One of the things we did not like from the old tagging service, was that the links were added to the content directly. With SuperTagging the content field of your instance is never changed when your using the markup feature. Instead, another attribute is added to your instance during processing. SUPERTAGGING_MARKUP_FIELD_SUFFIX tells what the prefix for that field will be. For example, if we wanted to markup a field named ‘content’, after SuperTagging is done processing, another attribute will be available in that model called ‘tagged_content’. This attribute will contain the content in a marked up state. This way, if we ever decided to change the way we markup our data, or change the location of where the links in the content go, we don’t need to change the instance directly.

Vist the Markup page for more information

This next set of settings we use is to exclude the types (entities) and relation types (events/facts)

SUPERTAGGING_TAG_TYPE_EXCLUSIONS = [
    'Anniversary',
    'City',
    #'Company',
    'Continent',
    #'Country',
    'Currency',
    'EmailAddress',
    'EntertainmentAwardEvent',
    'Facility',
    'FaxNumber',
    #'Holiday',
    'IndustryTerm',
    'MarketIndex',
    'MedicalCondition',
    'MedicalTreatment',
    'Movie',
    'MusicAlbum',
    'MusicGroup',
    'NaturalFeature',
    'OperatingSystem',
    #'Organization',
    #'Person',
    'PhoneNumber',
    'PoliticalEvent',
    'Position',
    'Product',
    'ProgrammingLanguage',
    'ProvinceOrState',
    'PublishedMedium',
    #'RadioProgram',
    'RadioStation',
    'Region',
    #'SportsEvent',
    #'SportsGame',
    #'SportsLeague',
    'Technology',
    #'TVShow',
    'TVStation',
    'URL',
]

SUPERTAGGING_REL_TYPE_EXCLUSIONS = [
   #'Acquisition',
   'Alliance',
   'AnalystEarningsEstimate',
   'AnalystRecommendation',
   #'Arrest',
   #'Bankruptcy',
   'BonusSharesIssuance',
   #'BusinessRelation',
   'Buybacks',
   'CompanyAccountingChange',
   #'CompanyAffiliates',
   'CompanyCompetitor',
   'CompanyCustomer',
   'CompanyEarningsAnnouncement',
   'CompanyEarningsGuidance',
   #'CompanyEmployeesNumber',
   'CompanyExpansion',
   'CompanyForceMajeure',
   #'CompanyFounded',
   'CompanyInvestment',
   'CompanyLaborIssues',
   'CompanyLayoffs',
   'CompanyLegalIssues',
   'CompanyListingChange',
   'CompanyLocation',
   'CompanyMeeting',
   'CompanyNameChange',
   'CompanyProduct',
   'CompanyReorganization',
   'CompanyRestatement',
   'CompanyTechnology',
   #'CompanyTicker',
   'CompanyUsingProduct',
   'ConferenceCall',
   #'Conviction',
   'CreditRating',
   'DebtFinancing',
   'DelayedFiling',
   'DiplomaticRelations',
   'Dividend',
   'EmploymentChange',
   #'EmploymentRelation',
   'EnvironmentalIssue',
   'Extinction',
   #'FamilyRelation',
   'FDAPhase',
   'Indictment',
   'IPO',
   'JointVenture',
   'ManMadeDisaster',
   'Merger',
   'MovieRelease',
   'MusicAlbumRelease',
   'NaturalDisaster',
   'PatentFiling',
   'PatentIssuance',
   #'PersonAttributes',
   #'PersonCareer',
   'PersonCommunication',
   #'PersonEducation',
   'PersonEmailAddress',
   'PersonRelation',
   'PersonTravel',
   'PoliticalEndorsement',
   #'PoliticalRelationship',
   'PollsResult',
   'ProductIssues',
   'ProductRecall',
   'ProductRelease',
   #'Quotation',
   'SecondaryIssuance',
   'StockSplit',
   'Trial',
   'VotingResult',
]

The ones that are commented out are the ones we do want. So so make sense of these to lists we only want the following:

• Tag Types (Entities)

  • Company
  • Country
  • Holiday
  • Organization
  • Person
  • RadioProgram
  • SportsEvent
  • SportsGame
  • SportsLeague
  • TVShow

• Relation Types (Events/Facts)

  • Acquisition
  • Arrest
  • Bankruptcy
  • BusinessRelation
  • CompanyAffiliates
  • CompanyEmployeesNumber
  • CompanyFounded
  • CompanyTicker
  • Conviction
  • EmploymentRelation
  • FamilyRelation
  • PersonAttributes
  • PersonCareer
  • PersonEducation
  • PoliticalRelationship
  • Quotation

This last section is the models we tag.

SUPERTAGGING_MODULES = {
   'stories.story':
        {'fields':(
            {'name': 'body',
             'process_type':'TEXT/HTML'},),
        },
    'massmedia.image':
        {'fields':(
            {'name': 'caption'},)
        },
}

Explanation:

This we very basic, since we only tag one field for both our stories and images. One thing that will change in the future for The Washington Times, will be that we will only tag stories with particular set of origins. For example, we currently have the following origins for our stories:

STORY_ORIGIN_CHOICES = (
    (0, 'Unknown'),
    (1, 'Admin'),
    (2, 'SaxoTech Editorial'),
    (3, 'SaxoTech Online'),
    (4, 'AP News'),
    (5, 'Bernini'),
    (6, 'AP NetNews'),
)

Some of these origins are not used anymore, and we will want to limit which ones we do tag, so eventually the SUPERTAGGING_MODULES will look like this:

SUPERTAGGING_MODULES = {
   'stories.story':
        {'fields':(
            {'name': 'body',
             'process_type':'TEXT/HTML'},),
         'match_kwargs': {'origin__in': [1,2,3,5]}
        },
    'massmedia.image':
        {'fields':(
            {'name': 'caption'},)
        },
}

If this was our current setup, we would not tag stories that had an origin of (0) Unknown, (4) AP News or (6) AP NetNews.

Showcasing what SuperTagging does¶

TODO

Running the process¶

Last thing is to run the cron job every 5 minutes to process the Queued objects.

*/5 *   *   *   *    /path/to/virtualenv/bin/python /path/to/project/manage.py st_process_queue>/dev/null 2>&1

SuperTag¶

SuperTagRelation¶

SuperTaggedItem¶

Generic relation to a SuperTag

SuperTaggedRelationItem¶

SuperTagProcessQueue¶

Holds a generic relation to an object to be processed at a later time, this model is only used when USE_QUEUE is set to True

Rendering Items¶

SuperTag, SuperTaggedItem, SuperTagRelation and SuperTaggedRelationItem have a render method in order to correctly display its contents.

Setup¶

In the settings you will need to have

SUPERTAGGING_SETTINGS = {

    # ... Other settings

    'MARKUP': {
        'ENABLED': True,
    }
}

How It Works¶

When SuperTagging loads up and markup is enabled, it will add an additional attribute for every field specified in WATCHED_FIELDS.

SUPERTAGGING_SETTINGS = {
    'ENABLED': True,
    'WATCHED_FIELDS': {
        'stories.story':
            {'fields':[
                {'name': 'body',
                 'markup_handler': 'MyCustomHandler'}]},
        'media.image':
            {'fields':[
                {'name': 'caption'}]},
        'blog.entry':
            {'fields':[
                {'name': 'content'},
                {'name': 'tease',
                 'markup': False}]}
    },

    # ... Other settings

    'MARKUP': {
        'ENABLED': True,
        'FIELD_SUFFIX': "tagged",
    },
}

Lets take the code sample above as an example. We notice that markup is enabled and the prefix for the markup fields is tagged. The first module is a story model, and the field named body is marked to be tagged. It also specifies a custom markup handler, which we’ll get to a bit later. The next model is a image model and the caption field is marked for tagging. The last model is an entry model and it has 2 fields marked for tagging, content and tease, but tease is not to be marked up.

After SuperTagging is done loading you will end up with three additional attributes for the three different models.

• Story model: body__tagged
• Image model: caption__tagged
• Entry model: content__tagged

Notice that the a tease__tagged does not exist for the Entry model because the markup flag for that field is False.

Markup handler¶

Each field will be assigned a MarkupHandler object, which can be found in supertagging/markup.py file. This module does all the markup processing for you on the fly. If an error occurs, since the original content is never touched, the original content is returned.

You can create your own custom handler as well.

from supertagging.markup import MarkupHandler

class MyCustomHandler(MarkupHandler):
    def handle(self, instance):
        # DO YOUR CUSTOM MARKUP HERE
        return "MARKED UP CONTENT"

The handle method needs to return a string of the marked up content.

If you want a create a custom handler but use the default markup method, your code might look something like this:

from supertagging.markup import MarkupHandler, markup_content

class MyCustomHandler(MarkupHandler):
    def handle(self, instance):
        # DO SOMETHING HERE
        return markup_content(instance, self.field)

Markup Template¶

markup.html

This template is used to render the tags in a marked up state. Below is the default html rendered.

<a href="#">{{ actual_value }}</a>

Context

• actual_value - the value of the tag, this might be the same as the tag name or a reference to the tag, IE: ‘his’, ‘her’ etc.
• tag - a SuperTag instance

Caching¶

There is a build-in cache for the markup, since every time we call this new attribute, a couple database calls need to happen to retrieve all the tags and its meta data for an instance.

You can change the default timeout for this cache by changing the following setting

SUPERTAGGING_MARKUP_CONTENT_CACHE_TIMEOUT = 3600

Gotchas¶

In some cases, after enabling markup and successfully tagging an instance the markup does not show up. Two things might cause this, 1 is the cache has not expired and 2 the markup did not validate.

Markup validation happens when the markup field is called and the data retrieved does not match what the instance has stored. This usually means that the instance was edited and the field that gets tagged was changed and it has not been re-processed by OpenCalais.