Documentation¶

Parsing a feed from a remote URL¶

>>> import feedparser
>>> d = feedparser.parse('http://feedparser.org/docs/examples/atom10.xml')
>>> d['feed']['title']
u'Sample Feed'

The following example assumes you are on Windows, and that you have saved a feed at c:incomingatom10.xml.

Parsing a feed from a local file¶

>>> import feedparser
>>> d = feedparser.parse(r'c:\\incoming\\atom10.xml')
>>> d['feed']['title']
u'Sample Feed'

Universal Feed Parser can also parse a feed in memory.

Parsing a feed from a string¶

>>> import feedparser
>>> rawdata = """<rss version="2.0">
<channel>
<title>Sample Feed</title>
</channel>
</rss>"""
>>> d = feedparser.parse(rawdata)
>>> d['feed']['title']
u'Sample Feed'

Values are returned as Python Unicode strings (except when they’re not – see advanced.encoding for all the gory details).

• Elsewhere
• Introduction to :program:`Python Unicode strings <http://docs.python.org/tut/node5.html#SECTION005130000000000000000>`_

filename=”common-rss-elements.html”

<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0">
    <channel>
        <title>Sample Feed</title>
        <description>For documentation &lt;em&gt;only&lt;/em&gt;</description>
        <link>http://example.org/</link>
        <pubDate>Sat, 07 Sep 2002 0:00:01 GMT</pubDate>
        <!-- other elements omitted from this example -->
        <item>
            <title>First entry title</title>
            <link>http://example.org/entry/3</link>
            <description>Watch out for &lt;span style="background-image:
            url(javascript:window.location='http://example.org/')"&gt;nasty
            tricks&lt;/span&gt;</description>
            <pubDate>Sat, 07 Sep 2002 0:00:01 GMT</pubDate>
            <guid>http://example.org/entry/3</guid>
            <!-- other elements omitted from this example -->
        </item>
    </channel>
</rss>

Accessing Common Channel Elements¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/rss20.xml>`_')
>>> d.feed.title
u'Sample Feed'
>>> d.feed.link
u'http://example.org/'
>>> d.feed.description
u'For documentation <em>only</em>'
>>> d.feed.date
u'Sat, 07 Sep 2002 0:00:01 GMT'
>>> d.feed.date_parsed
(2002, 9, 7, 0, 0, 1, 5, 250, 0)

The items are available in d.entries, which is a list. You access items in the list in the same order in which they appear in the original feed, so the first item is available in d.entries[0].

Accessing Common Item Elements¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/rss20.xml>`_')
>>> d.entries[0].title``u'First item title'``
>>> d.entries[0].link``u'http://example.org/item/1'``
>>> d.entries[0].description``u'Watch out for <span>nasty tricks</span>'``
>>> d.entries[0].date``u'Thu, 05 Sep 2002 0:00:01 GMT'``
>>> d.entries[0].date_parsed``(2002, 9, 5, 0, 0, 1, 3, 248, 0)``
>>> d.entries[0].id``u'http://example.org/guid/1'

filename=”common-atom-elements.html”

This sample Atom feed is at ` <http://feedparser.org/docs/examples/atom10.xml>`_.<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"
xml:base="http://example.org/"
xml:lang="en">
<title type="text">Sample Feed</title>
<subtitle type="html">
For documentation &lt;em&gt;only&lt;/em&gt;
</subtitle>
<link rel="alternate" href="/"/>
<link rel="self"
type="application/atom+xml"
href="http://www.example.org/atom10.xml"/>
<rights type="html">
&lt;p>Copyright 2005, Mark Pilgrim&lt;/p>&lt;
</rights>
<id>tag:feedparser.org,2005-11-09:/docs/examples/atom10.xml</id>
<generator
uri="http://example.org/generator/"
version="4.0">
Sample Toolkit
</generator>
<updated>2005-11-09T11:56:34Z</updated>
<entry>
<title>First entry title</title>
<link rel="alternate"
href="/entry/3"/>
<link rel="related"
type="text/html"
href="http://search.example.com/"/>
<link rel="via"
type="text/html"
href="http://toby.example.com/examples/atom10"/>
<link rel="enclosure"
type="video/mpeg4"
href="http://www.example.com/movie.mp4"
length="42301"/>
<id>tag:feedparser.org,2005-11-09:/docs/examples/atom10.xml:3</id>
<published>2005-11-09T00:23:47Z</published>
<updated>2005-11-09T11:56:34Z</updated>
<summary type="text/plain" mode="escaped">Watch out for nasty tricks</summary>
<content type="application/xhtml+xml" mode="xml"
xml:base="http://example.org/entry/3" xml:lang="en-US">
<div xmlns="http://www.w3.org/1999/xhtml">Watch out for
<span style="background: url(javascript:window.location='http://example.org/')">
nasty tricks</span></div>
</content>
</entry>
</feed>

Accessing Common Feed Elements¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/atom10.xml>`_')
>>> d.feed.title
``u'Sample feed'``
>>> d.feed.link
``u'http://example.org/'``
>>> d.feed.subtitle
``u'For documentation <em>only</em>'``
>>> d.feed.updated
``u'2005-11-09T11:56:34Z'``
>>> d.feed.updated_parsed
``(2005, 11, 9, 11, 56, 34, 2, 313, 0)``
>>> d.feed.id
``u'tag:feedparser.org,2005-11-09:/docs/examples/atom10.xml'

Entries are available in d.entries, which is a list. You access entries in the order in which they appear in the original feed, so the first entry is d.entries[0].

Accessing Common Entry Elements¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/atom10.xml>`_')
>>> d.entries[0].title
``u'First entry title'``
>>> d.entries[0].link
``u'http://example.org/entry/3``
>>> d.entries[0].id
``u'tag:feedparser.org,2005-11-09:/docs/examples/atom10.xml:3'``
>>> d.entries[0].published
``u'2005-11-09T00:23:47Z'``
>>> d.entries[0].published_parsed
``(2005, 11, 9, 0, 23, 47, 2, 313, 0)``
>>> d.entries[0].updated
``u'2005-11-09T11:56:34Z'``
>>> d.entries[0].updated_parsed
``(2005, 11, 9, 11, 56, 34, 2, 313, 0)``
>>> d.entries[0].summary
``u'Watch out for nasty tricks'``
>>> d.entries[0].content
``[{'type': u'application/xhtml+xml',
'base': u'http://example.org/entry/3',
'language': u'en-US',
'value': u'<div>Watch out for <span>nasty tricks</span></div>'}]

Because Atom entries can have more than one content element, d.entries[0].content is a list of dictionaries. Each dictionary contains metadata about a single content element. The two most important values in the dictionary are the content type, in d.entries[0].content[0].type, and the actual content value, in d.entries[0].content[0].value.

You can get this level of detail on other Atom elements too.

filename=”atom-detail.html”

Detailed Information on Feed Elements¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/atom10.xml>`_')
>>> d.feed.title_detail
``{'type': u'text/plain',
'base': u'http://example.org/',
'language': u'en',
'value': u'Sample Feed'}``
>>> d.feed.subtitle_detail
``{'type': u'text/html',
'base': u'http://example.org/',
'language': u'en',
'value': u'For documentation <em>only</em>'}``
>>> d.feed.rights_detail
``{'type': u'text/html',
'base': u'http://example.org/',
'language': u'en',
'value': u'<p>Copyright 2004, Mark Pilgrim</p>'}``
>>> d.entries[0].title_detail
``{'type': 'text/plain',
'base': u'http://example.org/',
'language': u'en',
'value': u'First entry title'}``
>>> d.entries[0].summary_detail
``{'type': u'text/plain',
'base': u'http://example.org/',
'language': u'en',
'value': u'Watch out for nasty tricks'}``
>>> len(d.entries[0].content)
``1``
>>> d.entries[0].content[0]
``{'type': u'application/xhtml+xml',
'base': u'http://example.org/entry/3',
'language': u'en-US'
'value': u'<div>Watch out for <span> nasty tricks</span></div>'}

filename=”uncommon-rss.html”

These elements are less common, but are useful for niche applications and may be present in any RSS feed.

Accessing feed image¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/rss20.xml>`_')
>>> d.feed.image
``{'title': u'Example banner',
'href': u'http://example.org/banner.png',
'width': 80,
'height': 15,
'link': u'http://example.org/'}

Feeds and entries can be assigned to multiple categories, and in some versions of RSS, categories can be associated with a domain. Both are free-form strings. For historical reasons, Universal Feed Parser makes multiple categories available as a list of tuples, rather than a list of dictionaries.

Accessing multiple categories¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/rss20.xml>`_')
>>> d.feed.categories
``[(u'Syndic8', u'1024'),
(u'dmoz', 'Top/Society/People/Personal_Homepages/P/')]

Each item in an RSS feed can have an enclosure, a delightful misnomer that is simply a link to an external file (usually a music or video file, but any type of file can be “enclosed”). Once rare, this element has recently gained popularity due to the rise of podcasting. Some clients (such as Apple’s iTunes) may automatically download enclosures; others (such as the web-based Bloglines) may simply render each enclosure as a link.

The RSS specification states that there can be at most one enclosure per item. However, Atom entries may contain more than one enclosure per entry, so Universal Feed Parser captures all of them and makes them available as a list.

Accessing enclosures¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/rss20.xml>`_')
>>> e = d.entries[0]
>>> len(e.enclosures)
``1``
>>> e.enclosures[0]
``{'type': u'audio/mpeg',
'length': u'1069871',
'href': u'http://example.org/audio/demo.mp3'}

No one is quite sure what a cloud is.

Accessing feed cloud¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/rss20.xml>`_')
>>> d.feed.cloud
``{'domain': u'rpc.example.com',
'port': u'80',
'path': u'/RPC2',
'registerprocedure': u'pingMe',
'protocol': u'soap'}

filename=”uncommon-atom.html”

These elements are less common, but are useful for niche applications and may be present in any Atom feed.

Accessing contributors¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/atom10.xml>`_')
>>> e = d.entries[0]
>>> len(e.contributors)
``2``
>>> e.contributors[0]
``{'name': u'Joe',
'href': u'http://example.org/joe/',
'email': u'joe@example.org'}``
>>> e.contributors[1]
``{'name': u'Sam',
'href': u'http://example.org/sam/',
'email': u'sam@example.org'}

Besides an alternate link, each Atom feed or entry can have an arbitrary number of other links. Each link is distinguished by its type attribute, which is a MIME-style content type, and its rel attribute.

Accessing multiple links¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/atom10.xml>`_')
>>> e = d.entries[0]
>>> len(e.links)
``4``
>>> e.links[0]
``{'rel': u'alternate',
'type': u'text/html',
'href': u'http://example.org/entry/3'}``
>>> e.links[1]
``{'rel': u'related',
'type': u'text/html',
'href': u'http://search.example.com/'}``
>>> e.links[2]
``{'rel': u'via',
'type': u'text/html',
'href': u'http://toby.example.com/examples/atom10'}``
>>> e.links[3]
``{'rel': u'enclosure',
'type': u'video/mpeg4',
'href': u'http://www.example.com/movie.mp4',
'length': u'42301'}

filename=”basic-existence.html”

Feeds in the real world may be missing elements, even elements that are required by the specification. You should always test for the existence of an element before getting its value. Never assume an element is present.

Testing if elements are present¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/atom10.xml>`_')
>>> d.feed.has_key('title')
``True``
>>> d.feed.has_key('ttl')
``False``
>>> d.feed.get('title', 'No title')
``u'Sample feed'``
>>> d.feed.get('ttl', 60)``60

filename=”advanced.html”

History of Date Formats¶

Here is a brief history of feed date formats:

• CDF states that all date values must conform to ISO 8601:1988. ISO 8601:1988 is not a freely available specification, but a brief (non-normative) description of the date formats it describes is available here: ISO 8601:1988 Date/Time Representations.
• RSS 0.90 has no date elements.
• Netscape RSS 0.91 does not specify a date format, but examples within the specification show RFC 822-style dates with 4-digit years.
• Userland RSS 0.91 states, All date-times in RSS conform to the Date and Time Specification of RFC 822. RFC 822 mandates 2-digit years; it does not allow 4-digit years.
• RSS 1.0 states that all date elements must conform to W3CDTF, which is a profile of ISO 8601:1988.
• RSS 2.0 states, All date-times in RSS conform to the Date and Time Specification of RFC 822, with the exception that the year may be expressed with two characters or four characters (four preferred).
• Atom 0.3 states that all date elements must conform to W3CDTF.
• Atom 1.0 states that all date elements “MUST conform to the date-time production in RFC 3339. In addition, an uppercase T character MUST be used to separate date and time, and an uppercase Z character MUST be present in the absence of a numeric time zone offset.”

Recognized Date Formats¶

Here is a representative list of the formats that Universal Feed Parser can recognize in any date element:

• valid W3CDTF (numeric timezone)2003-12-31T10:14:55-08:00(2003, 12, 31, 18, 14, 55, 2, 365, 0) |
• valid W3CDTF (UTC timezone)2003-12-31T10:14:55Z(2003, 12, 31, 10, 14, 55, 2, 365, 0) |
• valid W3CDTF (yyyy)2003(2003, 1, 1, 0, 0, 0, 2, 1, 0) |
• valid W3CDTF (yyyy-mm)2003-12(2003, 12, 1, 0, 0, 0, 0, 335, 0) |
• valid W3CDTF (yyyy-mm-dd)2003-12-31(2003, 12, 31, 0, 0, 0, 2, 365, 0) |
• valid ISO 8601 (yyyymmdd)20031231(2003, 12, 31, 0, 0, 0, 2, 365, 0) |
• valid ISO 8601 (-yy-mm)-03-12(2003, 12, 1, 0, 0, 0, 0, 335, 0) |
• valid ISO 8601 (-yymm)-0312(2003, 12, 1, 0, 0, 0, 0, 335, 0) |
• valid ISO 8601 (-yy-mm-dd)-03-12-31(2003, 12, 31, 0, 0, 0, 2, 365, 0) |
• valid ISO 8601 (yymmdd)031231(2003, 12, 31, 0, 0, 0, 2, 365, 0) |
• valid ISO 8601 (yyyy-o)2003-335(2003, 12, 1, 0, 0, 0, 0, 335, 0) |
• valid ISO 8601 (yyo)03335(2003, 12, 1, 0, 0, 0, 0, 335, 0) |
• valid asctimeSun Jan 4 16:29:06 PST 2004(2004, 1, 5, 0, 29, 6, 0, 5, 0) | |
• bogus RFC 822 (invalid day/month) | Thu, 31 Jun 2004 19:48:21 | GMT(2004, 7, 1, 19, 48, 21, 3, 183, 0) |
• bogus RFC 822 (invalid month)Mon, 26 January 2004 16:31:00 EST(2004, 1, 26, 21, 31, 0, 0, 26, 0) |
• bogus RFC 822 (invalid timezone)Mon, 26 Jan 2004 16:31:00 ET(2004, 1, 26, 21, 31, 0, 0, 26, 0) |
• bogus W3CDTF (invalid hour)2003-12-31T25:14:55Z(2004, 1, 1, 1, 14, 55, 3, 1, 0) |
• bogus W3CDTF (invalid minute)2003-12-31T10:61:55Z(2003, 12, 31, 11, 1, 55, 2, 365, 0) |
• bogus W3CDTF (invalid second)2003-12-31T10:14:61Z(2003, 12, 31, 10, 15, 1, 2, 365, 0) |
• bogus (MSSQL)2004-07-08 23:56:58.0(2004, 7, 8, 14, 56, 58, 3, 190, 0) |
• bogus (MSSQL-ish, without fractional second)2004-07-08 23:56:58(2004, 7, 8, 14, 56, 58, 3, 190, 0) |
• bogus (Korean)2004-05-25 오 11:23:17(2004, 5, 25, 14, 23, 17, 1, 146, 0) |
• bogus (Greek)Κυρ, 11 Ιούλ 2004 12:00:00 EST(2004, 7, 11, 17, 0, 0, 6, 193, 0) |
• bogus (Hungarian)július-13T9:15-05:00(2004, 7, 13, 14, 15, 0, 1, 195, 0) |

Universal Feed Parser recognizes all character-based timezone abbreviations defined in RFC 822. In addition, Universal Feed Parser recognizes the following invalid timezones:

• AT is treated as AST
• ET is treated as EST
• CT is treated as CST
• MT is treated as MST
• PT is treated as PST

Supporting Additional Date Formats¶

Universal Feed Parser supports many different date formats, but there are probably many more in the wild that are still unsupported. If you find other date formats, you can support them by registering them with registerDateHandler. It takes a single argument, a callback function. The callback function should take a single argument, a string, and return a single value, a 9-tuple Python date in UTC.

Registering a third-party date handler

::

import feedparser import re

_my_date_pattern = re.compile(\r’(\d{,2})/(\d{,2})/(\d{4}) (\d{,2}):(\d{2}):(\d{2})’)

def myDateHandler(aDateString):

“”“parse a UTC date in MM/DD/YYYY HH:MM:SS format”“” month, day, year, hour, minute, second = \ _my_date_pattern.search(aDateString).groups() return (int(year), int(month), int(day), \ int(hour), int(minute), int(second), 0, 0, 0)

feedparser.registerDateHandler(myDateHandler) d = feedparser.parse(...)

Your newly-registered date handler will be tried before all the other date handlers built into Universal Feed Parser. (More specifically, all date handlers are tried in “last in, first out” order; i.e. the last handler to be registered is the first one tried, and so on in reverse order of registration.)

If your date handler returns None, or anything other than a Python 9-tuple date, or raises an exception of any kind, the error will be silently ignored and the other registered date handlers will be tried in order. If no date handlers succeed, then the date is not parsed, and the *_parsed value will not be present in the results dictionary. The original date string will still be available in the appropriate element in the results dictionary.

filename=”html-sanitization.html”

HTML Sanitization¶

The following HTML elements are allowed by default (all others are stripped):a, abbr, acronym, address, area, article, aside, audio, b, big, blockquote, br, button, canvas, caption, center, cite, code, col, colgroup, command, datagrid, datalist, dd, del, details, dfn, dialog, dir, div, dl, dt, em, event-source, fieldset, figure, footer, font, form, header, h1, h2, h3, h4, h5, h6, hr, i, img, input, ins, keygen, kbd, label, legend, li, m, map, menu, meter, multicol, nav, nextid, noscript, ol, output, optgroup, option, p, pre, progress, q, s, samp, section, select, small, sound, source, spacer, span, strike, strong, sub, sup, table, tbody, td, textarea, time, tfoot, th, thead, tr, tt, u, ul, var, video

The following HTML attributes are allowed by default (all others are stripped):abbr, accept, accept-charset, accesskey, action, align, alt, autoplay, autocomplete, autofocus, axis, background, balance, bgcolor, bgproperties, border, bordercolor, bordercolordark, bordercolorlight, bottompadding, cellpadding, cellspacing, ch, challenge, char, charoff, choff, charset, checked, cite, class, clear, color, cols, colspan, compact, contenteditable, coords, data, datafld, datapagesize, datasrc, datetime, default, delay, dir, disabled, draggable, dynsrc, enctype, end, face, for, form, frame, galleryimg, gutter, headers, height, hidefocus, hidden, high, href, hreflang, hspace, icon, id, inputmode, ismap, keytype, label, leftspacing, lang, list, longdesc, loop, loopcount, loopend, loopstart, low, lowsrc, max, maxlength, media, method, min, multiple, name, nohref, noshade, nowrap, open, optimum, pattern, ping, point-size, prompt, pqg, radiogroup, readonly, rel, repeat-max, repeat-min, replace, required, rev, rightspacing, rows, rowspan, rules, scope, selected, shape, size, span, src, start, step, summary, suppress, tabindex, target, template, title, toppadding, type, unselectable, usemap, urn, valign, value, variable, volume, vspace, vrml, width, wrap, xml:lang

SVG Sanitization¶

The following SVG elements are allowed by default (all others are stripped):a, animate, animateColor, animateMotion, animateTransform, circle, defs, desc, ellipse, foreignObject, font-face, font-face-name, font-face-src, g, glyph, hkern, linearGradient, line, marker, metadata, missing-glyph, mpath, path, polygon, polyline, radialGradient, rect, set, stop, svg, switch, text, title, tspan, use

The following SVG attributes are allowed by default (all others are stripped):accent-height, accumulate, additive, alphabetic, arabic-form, ascent, attributeName, attributeType, baseProfile, bbox, begin, by, calcMode, cap-height, class, color, color-rendering, content, cx, cy, d, dx, dy, descent, display, dur, end, fill, fill-opacity, fill-rule, font-family, font-size, font-stretch, font-style, font-variant, font-weight, from, fx, fy, g1, g2, glyph-name, gradientUnits, hanging, height, horiz-adv-x, horiz-origin-x, id, ideographic, k, keyPoints, keySplines, keyTimes, lang, mathematical, marker-end, marker-mid, marker-start, markerHeight, markerUnits, markerWidth, max, min, name, offset, opacity, orient, origin, overline-position, overline-thickness, panose-1, path, pathLength, points, preserveAspectRatio, r, refX, refY, repeatCount, repeatDur, requiredExtensions, requiredFeatures, restart, rotate, rx, ry, slope, stemh, stemv, stop-color, stop-opacity, strikethrough-position, strikethrough-thickness, stroke, stroke-dasharray, stroke-dashoffset, stroke-linecap, stroke-linejoin, stroke-miterlimit, stroke-opacity, stroke-width, systemLanguage, target, text-anchor, to, transform, type, u1, u2, underline-position, underline-thickness, unicode, unicode-range, units-per-em, values, version, viewBox, visibility, width, widths, x, x-height, x1, x2, xlink:actuate, xlink:arcrole, xlink:href, xlink:role, xlink:show, xlink:title, xlink:type, xml:base, xml:lang, xml:space, xmlns, xmlns:xlink, y, y1, y2, zoomAndPan

MathML Sanitization¶

The following MathML elements are allowed by default (all others are stripped):annotation, annotation-xml, maction, math, merror, mfenced, mfrac, mi, mmultiscripts, mn, mo, mover, mpadded, mphantom, mprescripts, mroot, mrow, mspace, msqrt, mstyle, msub, msubsup, msup, mtable, mtd, mtext, mtr, munder, munderover, none, semantics

The following MathML attributes are allowed by default (all others are stripped):actiontype, align, columnalign, columnalign, columnalign, close, columnlines, columnspacing, columnspan, depth, display, displaystyle, encoding, equalcolumns, equalrows, fence, fontstyle, fontweight, frame, height, linethickness, lspace, mathbackground, mathcolor, mathvariant, mathvariant, maxsize, minsize, open, other, rowalign, rowalign, rowalign, rowlines, rowspacing, rowspan, rspace, scriptlevel, selection, separator, separators, stretchy, width, width, xlink:href, xlink:show, xlink:type, xmlns, xmlns:xlink

CSS Sanitization¶

The following CSS properties are allowed by default in style attributes (all others are stripped):azimuth, background-color, border-bottom-color, border-collapse, border-color, border-left-color, border-right-color, border-top-color, clear, color, cursor, direction, display, elevation, float, font, font-family, font-size, font-style, font-variant, font-weight, height, letter-spacing, line-height, overflow, pause, pause-after, pause-before, pitch, pitch-range, richness, speak, speak-header, speak-numeral, speak-punctuation, speech-rate, stress, text-align, text-decoration, text-indent, unicode-bidi, vertical-align, voice-family, volume, white-space, width

Whitelist, Don’t Blacklist¶

I am often asked why Universal Feed Parser is so hard-assed about HTML and CSS sanitizing. To illustrate the problem, here is an incomplete list of potentially dangerous HTML tags and attributes:

• script, which can contain malicious script
• applet, embed, and object, which can automatically download and execute malicious code
• meta, which can contain malicious redirects
• onload, onunload, and all other on* attributes, which can contain malicious script
• style, link, and the style attribute, which can contain malicious script

style? Yes, style. CSS definitions can contain executable code.

This sample is taken from ` <http://feedparser.org/docs/examples/rss20.xml>`_: <description>Watch out for &lt;span style=”background: url(javascript:window.location=’http://example.org/‘)”&gt; nasty tricks&lt;/span&gt;</description>

This sample is more advanced, and does not contain the keyword javascript: that many naive HTML sanitizers scan for:<description>Watch out for &lt;span style=”any: expression(window.location=’http://example.org/‘)”&gt; nasty tricks&lt;/span&gt;</description>

Internet Explorer for Windows will execute the Javascript in both of these examples.

Now consider that in HTML, attribute values may be entity-encoded in several different ways.

To a browser, this:<span style=”any: expression(window.location=’http://example.org/‘)”>

is the same as this (without the line breaks):<span style=”&#97;&#110;&#121;&#58;&#32;&#101;&#120;&#112;&#114;&#101; &#115;&#115;&#105;&#111;&#110;&#40;&#119;&#105;&#110;&#100;&#111;&#119; &#46;&#108;&#111;&#99;&#97;&#116;&#105;&#111;&#110;&#61;&#39;&#104; &#116;&#116;&#112;&#58;&#47;&#47;&#101;&#120;&#97;&#109;&#112;&#108; &#101;&#46;&#111;&#114;&#103;&#47;&#39;&#41;”>

which is the same as this (without the line breaks):<span style=”&#x61;&#x6e;&#x79;&#x3a;&#x20;&#x65;&#x78;&#x70;&#x72; &#x65;&#x73;&#x73;&#x69;&#x6f;&#x6e;&#x28;&#x77;&#x69;&#x6e; &#x64;&#x6f;&#x77;&#x2e;&#x6c;&#x6f;&#x63;&#x61;&#x74;&#x69; &#x6f;&#x6e;&#x3d;&#x27;&#x68;&#x74;&#x74;&#x70;&#x3a;&#x2f; &#x2f;&#x65;&#x78;&#x61;&#x6d;&#x70;&#x6c;&#x65;&#x2e;&#x6f; &#x72;&#x67;&#x2f;&#x27;&#x29;”>

And so on, plus several other variations, plus every combination of every variation.

The more I investigate, the more cases I find where Internet Explorer for Windows will treat seemingly innocuous markup as code and blithely execute it. This is why Universal Feed Parser uses a whitelist and not a blacklist. I am reasonably confident that none of the elements or attributes on the whitelist are security risks. I am not at all confident about elements or attributes that I have not explicitly investigated. And I have no confidence at all in my ability to detect strings within attribute values that Internet Explorer for Windows will treat as executable code.

• Elsewhere
• How to consume RSS safely explains the platypus attack.

filename=”content-normalization.html”

Universal Feed Parser can parse many different types of feeds: Atom, CDF, and nine different versions of RSS. You should not be forced to learn the differences between these formats. Universal Feed Parser does its best to ensure that you can treat all feeds the same way, regardless of format or version.

Accessing an Atom feed as an RSS feed¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/atom10.xml>`_')
>>> d['channel']['title']
``u'Sample Feed'``
>>> d['channel']['link']
``u'http://example.org/'``
>>> d['channel']['description']
``u'For documentation <em>only</em>``
>>> len(d['items'])
``1``
>>> e = d['items'][0]
>>> e['title']
``u'First entry title'``
>>> e['link']
``u'http://example.org/entry/3'``
>>> e['description']
``u'Watch out for nasty tricks'``
>>> e['author']
``u'Mark Pilgrim (mark@example.org)'

The same thing works in reverse: you can access RSS feeds as if they were Atom feeds.

Accessing an RSS feed as an Atom feed¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/rss20.xml>`_')
>>> d.feed.subtitle_detail
``{'type': 'text/html',
'base': 'http://feedparser.org/docs/examples/rss20.xml',
'language': None,
'value': u'For documentation <em>only</em>'}``
>>> len(d.entries)
``1``
>>> e = d.entries[0]
>>> e.links
``[{'rel': 'alternate',
'type': 'text/html',
'href': u'http://example.org/item/1'}]``
>>> e.summary_detail
``{'type': 'text/html',
'base': 'http://feedparser.org/docs/examples/rss20.xml',
'language': u'en',
'value': u'Watch out for <span>nasty tricks</span>'}``
>>> e.updated_parsed
``(2002, 9, 5, 0, 0, 1, 3, 248, 0)

filename=”microformats.html”

An emerging trend in feed syndication is the inclusion of microformats. Besides the semantics defined by individual feed formats, publishers can add additional semantics using rel and class attributes in embedded HTML content.

rel=enclosure¶

The rel=enclosure microformat provides a way for embedded HTML content to specify that a certain link should be treated as an enclosure. Universal Feed Parser looks for links within embedded markup that meet any of the following conditions:

• rel attribute contains enclosure (note: rel attributes can contain a list of space-separated values)
• type attribute starts with audio/
• type attribute starts with video/
• type attribute starts with application/ but does not end with xml
• href attribute ends with one of the following file extensions: .zip, .rar, .exe, .gz, .tar, .tgz, .tbz2, .bz2, .z, .7z, .dmg, .img, .sit, .sitx, .hqx, .deb, .rpm, .bz2, .jar, .rar, .iso, .bin, .msi, .mp2, .mp3, .ogg, .ogm, .mp4, .m4v, .m4a, .avi, .wma, .wmv

When Universal Feed Parser finds a link that satisfies any of these conditions, it adds it to reference.entry.enclosures.

>>> import feedparser
>>> d = feedparser.parse('http://feedparser.org/docs/examples/rel-enclosure.xml')
>>> d.entries[0].enclosures``[{u'href': u'http://example.com/movie.mp4', 'title': u'awesome movie'}]

rel=tag¶

The rel=tag microformat allows you to define tags within embedded HTML content. Universal Feed Parser looks for these attribute values in embedded markup and maps them to reference.entry.tags.

>>> import feedparser
>>> d = feedparser.parse('http://feedparser.org/docs/examples/rel-tag.xml')
>>> d.entries[0].tags``[{'term': u'tech', 'scheme': u'http://del.icio.us/tag/', 'label': u'Technology'}]

XFN¶

The XFN microformat allows you to define human relationships between URIs. For example, you could link from your weblog to your spouse’s weblog with the rel="spouse" relation. It is intended primarily for “blogrolls” or other static lists of links, but the relations can occur anywhere in HTML content. If found, Universal Feed Parser will return the XFN information in reference.entry.xfn.

Universal Feed Parser supports all of the relationships listed in the XFN 1.1 profile, as well as the following variations:

• coworker in addition to co-worker
• coresident in addition to co-resident
• relative in addition to kin
• brother and sister in addition to sibling
• husband and wife in addition to spouse

>>> import feedparser
>>> d = feedparser.parse('http://feedparser.org/docs/examples/xfn.xml')
>>> person = d.entries[0].xfn[0]
>>> person.name``u'John Doe'``
>>> person.href``u'http://example.com/johndoe'``
>>> person.relationships``[u'coworker', u'friend']

hCard¶

The hCard microformat allows you to embed address book information within HTML content. If Universal Feed Parser finds an hCard within supported elements, it converts it into an RFC 2426-compliant vCard and returns it in reference.entry.vcard.

>>> import feedparser
>>> d = feedparser.parse('http://feedparser.org/docs/examples/hcard.xml')
>>> print d.entries[0].vCard
``BEGIN:vCard
VERSION:3.0
FN:Frank Dawson
N:Dawson;Frank
ADR;TYPE=work,postal,parcel:;;6544 Battleford Drive;Raleigh;NC;27613-3502;U
.S.A.
TEL;TYPE=WORK,VOICE,MSG:+1-919-676-9515
TEL;TYPE=WORK,FAX:+1-919-676-9564
EMAIL;TYPE=internet,pref:Frank_Dawson at Lotus.com
EMAIL;TYPE=internet:fdawson at earthlink.net
ORG:Lotus Development Corporation
URL:http://home.earthlink.net/~fdawson
END:vCard
BEGIN:vCard
VERSION:3.0
FN:Tim Howes
N:Howes;Tim
ADR;TYPE=work:;;501 E. Middlefield Rd.;Mountain View;CA;94043;U.S.A.
TEL;TYPE=WORK,VOICE,MSG:+1-415-937-3419
TEL;TYPE=WORK,FAX:+1-415-528-4164
EMAIL;TYPE=internet:howes at netscape.com
ORG:Netscape Communications Corp.
END:vCard

• Elsewhere
• Microformats.org
• rel=enclosure specification
• rel=tag specification
• XFN specification
• hCard specification

filename=”namespace-handling.html”

Universal Feed Parser attempts to expose all possible data in feeds, including elements in extension namespaces.

Accessing namespaced elements¶

>>> import feedparser
>>> d = feedparser.parse('http://feedparser.org/docs/examples/prism.rdf')
>>> d.feed.prism_issn
``u'0028-0836'``
>>> d.namespaces
``{'': u'http://purl.org/rss/1.0/',
'prism': u'http://prismstandard.org/namespaces/1.2/basic/',
'rdf': u'http://www.w3.org/1999/02/22-rdf-syntax-ns#'}

The prefix used to construct the variable name is not guaranteed to be the same as the prefix of the namespaced element in the original feed. If Universal Feed Parser recognizes the namespace, it will use the namespace’s preferred prefix to construct the variable name. It will also list the namespace in the namespaces dictionary using the namespace’s preferred prefix.

In the previous example, the namespace (http://prismstandard.org/namespaces/1.2/basic/) was defined with the namespace’s preferred prefix (prism), so the prism:issn element was accessible as the variable d.feed.prism_issn. However, if the namespace is defined with a non-standard prefix, Universal Feed Parser will still construct the variable name using the preferred prefix, not the actual prefix that is used in the feed.

This will become clear with an example.

Accessing namespaced elements with non-standard prefixes¶

>>> import feedparser
>>> d = feedparser.parse('http://feedparser.org/docs/examples/nonstandard_prefix.rdf')
>>> d.feed.prism_issn
``u'0028-0836'``
>>> d.feed.foo_issn
``Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "feedparser.py", line 158, in __getattr__
raise AttributeError, "object has no attribute '%s'" % key
AttributeError: object has no attribute 'foo_issn'``
>>> d.namespaces
``{'': u'http://purl.org/rss/1.0/',
'prism': u'http://prismstandard.org/namespaces/1.2/basic/',
'rdf': u'http://www.w3.org/1999/02/22-rdf-syntax-ns#'}

This is the complete list of namespaces that Universal Feed Parser recognizes and uses to construct the variable names for data in these namespaces:

Recognized Namespaces ———————PrefixNamespaceadminhttp://webns.net/mvcb/aghttp://purl.org/rss/1.0/modules/aggregation/annotatehttp://purl.org/rss/1.0/modules/annotate/audiohttp://media.tangent.org/rss/1.0/blogChannelhttp://backend.userland.com/blogChannelModulecchttp://web.resource.org/cc/creativeCommonshttp://backend.userland.com/creativeCommonsRssModulecohttp://purl.org/rss/1.0/modules/companycontenthttp://purl.org/rss/1.0/modules/content/cphttp://my.theinfo.org/changed/1.0/rss/dchttp://purl.org/dc/elements/1.1/dctermshttp://purl.org/dc/terms/emailhttp://purl.org/rss/1.0/modules/email/evhttp://purl.org/rss/1.0/modules/event/feedburnerhttp://rssnamespace.org/feedburner/ext/1.0fmhttp://freshmeat.net/rss/fm/foafhttp://xmlns.com/foaf/0.1/geohttp://www.w3.org/2003/01/geo/wgs84_pos#icbmhttp://postneo.com/icbm/imagehttp://purl.org/rss/1.0/modules/image/ituneshttp://www.itunes.com/DTDs/PodCast-1.0.dtdituneshttp://example.com/DTDs/PodCast-1.0.dtdlhttp://purl.org/rss/1.0/modules/link/mediahttp://search.yahoo.com/mrsspingbackhttp://madskills.com/public/xml/rss/module/pingback/prismhttp://prismstandard.org/namespaces/1.2/basic/rdfhttp://www.w3.org/1999/02/22-rdf-syntax-ns#rdfshttp://www.w3.org/2000/01/rdf-schema#refhttp://purl.org/rss/1.0/modules/reference/reqvhttp://purl.org/rss/1.0/modules/richequiv/searchhttp://purl.org/rss/1.0/modules/search/slashhttp://purl.org/rss/1.0/modules/slash/soaphttp://schemas.xmlsoap.org/soap/envelope/sshttp://purl.org/rss/1.0/modules/servicestatus/strhttp://hacks.benhammersley.com/rss/streaming/subhttp://purl.org/rss/1.0/modules/subscription/syhttp://purl.org/rss/1.0/modules/syndication/szfhttp://schemas.pocketsoap.com/rss/myDescModule/taxohttp://purl.org/rss/1.0/modules/taxonomy/thrhttp://purl.org/rss/1.0/modules/threading/tihttp://purl.org/rss/1.0/modules/textinput/trackbackhttp://madskills.com/public/xml/rss/module/trackback/wfwhttp://wellformedweb.org/CommentAPI/wikihttp://purl.org/rss/1.0/modules/wiki/xhtmlhttp://www.w3.org/1999/xhtmlxlinkhttp://www.w3.org/1999/xlinkxmlhttp://www.w3.org/XML/1998/namespace

filename=”resolving-relative-links.html”

Many feed elements and attributes are URIs. Universal Feed Parser resolves relative URIs according to the XML:Base specification. We’ll see how that works in a minute, but first let’s talk about which values are treated as URIs.

Which Values Are URIs¶

These feed elements are treated as URIs, and resolved if they are relative:

• reference.feed.link
• feed.links[i].href
• feed.generator_detail.href
• reference.feed.id
• feed.image.href
• feed.image.link
• feed.textinput.link
• feed.author_detail.href
• feed.publisher_detail.href
• feed.contributors[i].href
• reference.feed.docs
• reference.feed.license
• reference.entry.link
• entries[i].links[j].href
• reference.entry.id
• entries[i].author_detail.href
• entries[i].publisher_detail.href
• entries[i].contributors[j].href
• entries[i].enclosures[j].href
• reference.entry.source.author_detail.href
• reference.entry.source.contributors.href
• reference.entry.source.links.href
• reference.entry.comments
• reference.entry.license

In addition, several feed elements may contain HTML or XHTML markup. Certain elements and attributes in HTML can be relative URIs, and Universal Feed Parser will resolve these URIs according to the same rules as the feed elements listed above.

These feed elements may contain HTML or XHTML markup. In Atom feeds, whether these elements are treated as HTML depends on the value of the type attribute. In RSS feeds, these values are always treated as HTML.

• reference.feed.title (feed.title_detail.value)
• reference.feed.subtitle (feed.subtitle_detail.value))
• reference.feed.info (feed.info_detail.value)
• reference.feed.rights (feed.rights_detail.value)
• reference.entry.title (entries[i].title_detail.value)
• reference.entry.summary (entries[i].summary_detail.value)
• entries[i].content[j].value

When any of these feed elements contains HTML or XHTML markup, the following HTML elements are treated as URIs and are resolved if they are relative:

• <a href=”...”>
• <applet codebase=”...”>
• <area href=”...”>
• <blockquote cite=”...”>
• <body background=”...”>
• <del cite=”...”>
• <form action=”...”>
• <frame longdesc=”...”>
• <frame src=”...”>
• <iframe longdesc=”...”>
• <iframe src=”...”>
• <head profile=”...”>
• <img longdesc=”...”>
• <img src=”...”>
• <img usemap=”...”>
• <input src=”...”>
• <input usemap=”...”>
• <ins cite=”...”>
• <link href=”...”>
• <object classid=”...”>
• <object codebase=”...”>
• <object data=”...”>
• <object usemap=”...”>
• <q cite=”...”>
• <script src=”...”>

How Relative URIs Are Resolved¶

Universal Feed Parser resolves relative URIs according to the XML:Base specification. This defines a hierarchical inheritance system, where one element can define the base URI for itself and all of its child elements, using an xml:base attribute. A child element can then override its parent’s base URI by redeclaring xml:base to a different value.

If no xml:base is specified, the feed has a default base URI defined in the Content-Location HTTP header.

If no Content-Location HTTP header is present, the URL used to retrieve the feed itself is the default base URI for all relative links within the feed. If the feed was retrieved via an HTTP redirect (any HTTP 3xx status code), then the final URL of the feed is the default base URI.

For example, an xml:base on the root-level element sets the base URI for all URIs in the feed.

>>> import feedparser
>>> d = feedparser.parse("http://feedparser.org/docs/examples/base.xml")
>>> d.feed.link``u'http://example.org/index.html'``
>>> d.feed.generator_detail.href``u'http://example.org/generator/'

An xml:base attribute on an <entry> overrides the xml:base on the parent <feed>.

>>> import feedparser
>>> d = feedparser.parse("http://feedparser.org/docs/examples/base.xml")
>>> d.entries[0].link``u'http://example.org/archives/000001.html'``
>>> d.entries[0].author_detail.href``u'http://example.org/about/'

An xml:base on <content> overrides the xml:base on the parent <entry>. In addition, whatever the base URI is for the <content> element (whether defined directly on the <content> element, or inherited from the parent element) is used as the base URI for the embedded HTML or XHTML markup within the content.

>>> import feedparser
>>> d = feedparser.parse("http://feedparser.org/docs/examples/base.xml")
>>> d.entries[0].content[0].value``u'<p id="anchor1"><a href="http://example.org/archives/000001.html#anchor2">skip to anchor 2</a></p>
<p>Some content</p>
<p id="anchor2">This is anchor 2</p>'

The xml:base affects other attributes in the element in which it is declared.

>>> import feedparser
>>> d = feedparser.parse("http://feedparser.org/docs/examples/base.xml")
>>> d.entries[0].links[1].rel``u'service.edit'``
>>> d.entries[0].links[1].href``u'http://example.com/api/client/37'

If no xml:base is specified on the root-level element, the default base URI is given in the Content-Location HTTP header. This can still be overridden by any child element that declares an xml:base attribute.

>>> import feedparser
>>> d = feedparser.parse("http://feedparser.org/docs/examples/http_base.xml")
>>> d.feed.link``u'http://example.org/index.html'``
>>> d.entries[0].link``u'http://example.org/archives/000001.html'

Finally, if no root-level xml:base is declared, and no Content-Location HTTP header is present, the URL of the feed itself is the default base URI. Again, this can still be overridden by any element that declares an xml:base attribute.

>>> import feedparser
>>> d = feedparser.parse("http://feedparser.org/docs/examples/no_base.xml")
>>> d.feed.link``u'http://feedparser.org/docs/examples/index.html``
>>> d.entries[0].link``u'http://example.org/archives/000001.html'

Disabling Relative URIs Resolution¶

Though not recommended, it is possible to disable Universal Feed Parser‘s relative URI resolution by setting feedparser.RESOLVE_RELATIVE_URIS to 0.

>>> import feedparser
>>> d = feedparser.parse('http://feedparser.org/docs/examples/base.xml')
>>> d.entries[0].content[0].base``u'http://example.org/archives/000001.html'``
>>> print d.entries[0].content[0].value``<p id="anchor1"><a href="http://example.org/archives/000001.html#anchor2">skip to anchor 2</a></p>
<p>Some content</p>
<p id="anchor2">This is anchor 2</p>``
>>> feedparser.RESOLVE_RELATIVE_URIS = 0
>>> d2 = feedparser.parse('http://feedparser.org/docs/examples/base.xml')
>>> d2.entries[0].content[0].base``u'http://example.org/archives/000001.html'``
>>> print d2.entries[0].content[0].value``<p id="anchor1"><a href="#anchor2">skip to anchor 2</a></p>
<p>Some content</p>
<p id="anchor2">This is anchor 2</p>

filename=”version-detection.html”

Universal Feed Parser attempts to autodetect the type and version of the feeds it parses. There are many subtle and not-so-subtle differences between the different versions of RSS, and applications may choose to handle different feed types in different ways.

Accessing feed version¶

>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/atom10.xml>`_')
>>> d.version``'atom10'``
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/atom03.xml>`_')
>>> d.version``'atom03'``
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/rss20.xml>`_')
>>> d.version``'rss20'``
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/rss20dc.xml>`_')
>>> d.version``'rss20'``
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/rss10.rdf>`_')
>>> d.version``'rss10'

Here is the complete list of known feed types and versions that may be returned in version:

rss090

RSS 0.90

rss091n

Netscape RSS 0.91

rss091u

Userland RSS 0.91 (differences from Netscape RSS 0.91)

rss10

RSS 1.0

rss092

RSS 0.92

rss093

RSS 0.93

rss094

RSS 0.94 (no accurate specification is known to exist)

rss20

RSS 2.0

rss

RSS (unknown or unrecognized version)

atom01

Atom 0.1

atom02

Atom 0.2

atom03

Atom 0.3

atom10

Atom 1.0

atom

Atom (unknown or unrecognized version)

cdf

CDF

hotrss

Hot RSS

If the feed type is completely unknown, version will be an empty string.

filename=”character-encoding.html”

Introduction to Character Encoding¶

In XML, the character encoding is optional and may be given in the XML declaration in the first line of the document, like this: <xml version=”1.0” encoding=”utf-8”?>

If no encoding is given, XML supports the use of a Byte Order Mark to identify the document as some flavor of UTF-32, UTF-16, or UTF-8. Section F of the XML specification outlines the process for determining the character encoding based on unique properties of the Byte Order Mark in the first two to four bytes of the document.

If no encoding is specified and no Byte Order Mark is present, XML defaults to UTF-8.

HTTP uses MIME to define a method of specifying the character encoding, as part of the Content-Type HTTP header, which looks like this: Content-Type: text/html; charset=”utf-8”

If no charset is specified, HTTP defaults to iso-8859-1, but only for text/* media types. For other media types, the default encoding is undefined, which is where RFC 3023 comes in.

According to RFC 3023, if the media type given in the Content-Type HTTP header is application/xml, application/xml-dtd, application/xml-external-parsed-entity, or any one of the subtypes of application/xml such as application/atom+xml or application/rss+xml or even application/rdf+xml, then the encoding is

# the encoding given in the charset parameter of the Content-Type HTTP header, or

# the encoding given in the encoding attribute of the XML declaration within the document, or

# utf-8.

On the other hand, if the media type given in the Content-Type HTTP header is text/xml, text/xml-external-parsed-entity, or a subtype like text/AnythingAtAll+xml, then the encoding attribute of the XML declaration within the document is ignored completely, and the encoding is

# the encoding given in the charset parameter of the Content-Type HTTP header, or

# us-ascii.

Handling Incorrectly-Declared Encodings¶

Universal Feed Parser initially uses the rules specified in RFC 3023 to determine the character encoding of the feed. If parsing succeeds, then that’s that. If parsing fails, Universal Feed Parser sets the bozo bit to 1 and sets bozo_exception to feedparser.CharacterEncodingOverride. Then it tries to reparse the feed with the following character encodings:

# the encoding specified in the XML declaration

# the encoding sniffed from the first four bytes of the document (as per Section F)

# the encoding auto-detected by the :program:`Universal Encoding Detector <http://chardet.feedparser.org/>`_, if installed

# utf-8

# windows-1252

If the character encoding can not be determined, Universal Feed Parser sets the bozo bit to 1 and sets bozo_exception to feedparser.CharacterEncodingUnknown. In this case, parsed values will be strings, not Unicode strings.

Handling Incorrectly-Declared Media Types¶

RFC 3023 only applies when the feed is served over HTTP with a Content-Type that declares the feed to be some kind of XML. However, some web servers are severely misconfigured and serve feeds with a Content-Type of text/plain, application/octet-stream, or some completely bogus media type.

Universal Feed Parser will attempt to parse such feeds, but it will set the bozo bit to 1 and set bozo_exception to feedparser.NonXMLContentType.

• Elsewhere
• RFC 3023
• Section F of the XML specification
• On the well-formedness of XML documents served as text/plain
• CJKCodecs and iconv_codec

filename=”bozo.html”

Universal Feed Parser can parse feeds whether they are well-formed XML or not. However, since some applications may wish to reject or warn users about non-well-formed feeds, Universal Feed Parser sets the bozo bit when it detects that a feed is not well-formed. Thanks to Tim Bray for suggesting this terminology.

Detecting a non-well-formed feed¶

>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/atom10.xml>`_')
>>> d.bozo``0``
>>> d = feedparser.parse('` <http://feedparser.org/tests/illformed/rss/aaa_illformed.xml>`_')
>>> d.bozo``1``
>>> d.bozo_exception``<xml.sax._exceptions.SAXParseException instance at 0x00BAAA08>``
>>> exc = d.bozo_exception
>>> exc.getMessage()``"expected '>'\\n"``
>>> exc.getLineNumber()``6

There are many reasons an XML document could be non-well-formed besides this example (incomplete end tags) See advanced.encoding for some other ways to trip the bozo bit.

filename=”http.html”

Using ETags to reduce bandwidth¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/atom10.xml>`_')
>>> d.etag``'"6c132-941-ad7e3080"'``
>>> d2 = feedparser.parse('` <http://feedparser.org/docs/examples/atom10.xml>`_', etag=d.etag)
>>> d2.status``304``
>>> d2.feed``{}``
>>> d2.entries``[]``
>>> d2.debug_message``'The feed has not changed since you last checked, so
the server sent no data.  This is a feature, not a bug!'

There is a related concept which accomplishes the same thing, but slightly differently. In this case, the server publishes the last-modified date of the feed in the HTTP header. You can send this back to the server on subsequent requests, and if the feed has not changed, the server will return HTTP status code 304 and no feed data.

Using Last-Modified headers to reduce bandwidth¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/atom10.xml>`_')
>>> d.modified``(2004, 6, 11, 23, 0, 34, 4, 163, 0)``
>>> d2 = feedparser.parse('` <http://feedparser.org/docs/examples/atom10.xml>`_', modified=d.modified)
>>> d2.status``304``
>>> d2.feed``{}``
>>> d2.entries``[]``
>>> d2.debug_message``'The feed has not changed since you last checked, so
the server sent no data.  This is a feature, not a bug!'

Clients should support both ETag and Last-Modified headers, as some servers support one but not the other.

• Elsewhere
• HTTP Conditional Get For RSS Hackers
• HTTP Web Services

filename=”http-useragent.html”

Universal Feed Parser sends a default User-Agent string when it requests a feed from a web server.

Customizing the User-Agent¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/atom10.xml>`_', \\

...`` agent='MyApp/1.0 +http://example.com/')

You can also set the User-Agent once, globally, and then call the parse function normally.

Customizing the User-Agent permanently¶

>>> import feedparser
>>> feedparser.USER_AGENT = "MyApp/1.0 +http://example.com/"
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/atom10.xml>`_')``

Universal Feed Parser also lets you set the referrer when you download a feed from a web server. This is discouraged, because it is a violation of RFC 2616. The default behavior is to send a blank referrer, and you should never need to override this.

Customizing the referrer¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/atom10.xml>`_', \\

...`` referrer='http://example.com/')

filename=”http-redirect.html”

When you download a feed from a remote web server, Universal Feed Parser exposes the HTTP status code. You need to understand the different codes, including permanent and temporary redirects, and feeds that have been marked gone.

Noticing temporary redirects¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/temporary.xml>`_')
>>> d.status``302``
>>> d.href``'http://feedparser.org/docs/examples/atom10.xml'``
>>> d.feed.title``u'Sample Feed'

When a feed has permanently moved to a new location, the web server will return a 301 status code. Again, Universal Feed Parser makes this available in d.status.

If you are polling a feed on a regular basis, it is very important to check the status code (d.status) every time you download. If the feed has been permanently redirected, you should update your database or configuration file with the new address (d.href). Repeatedly requesting the original address of a feed that has been permanently redirected is very rude, and may get you banned from the server.

Noticing permanent redirects¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/permanent.xml>`_')
>>> d.status``301``
>>> d.href``'http://feedparser.org/docs/examples/atom10.xml'``
>>> d.feed.title``u'Sample Feed'

When a feed has been permanently deleted, the web server will return a 410 status code. If you ever receive a 410, you should stop polling the feed and inform the end user that the feed is gone for good.

Repeatedly requesting a feed that has been marked as gone is very rude, and may get you banned from the server.

Noticing feeds marked gone¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/gone.xml>`_')
>>> d.status``410

filename=”http-authentication.html”

Universal Feed Parser supports downloading and parsing password-protected feeds that are protected by HTTP authentication. Both basic and digest authentication are supported.

Downloading a feed protected by basic authentication (the easy way)¶

In this example, the username is test and the password is basic.

>>> import feedparser
>>> d = feedparser.parse('`http://test:basic@feedparser.org/docs/examples/basic_auth.xml <http://feedparser.org/docs/examples/basic_auth.xml>`_')
>>> d.feed.title``u'Sample Feed'

The same technique works for digest authentication. (Technically, Universal Feed Parser will attempt basic authentication first, but if that fails and the server indicates that it requires digest authentication, Universal Feed Parser will automatically re-request the feed with the appropriate digest authentication headers. This means that this technique will send your password to the server in an easily decryptable form.)

Downloading a feed protected by digest authentication (the easy but horribly insecure way)¶

In this example, the username is test and the password is digest.

>>> import feedparser
>>> d = feedparser.parse('`http://test:digest@feedparser.org/docs/examples/digest_auth.xml <http://feedparser.org/docs/examples/digest_auth.xml>`_')
>>> d.feed.title``u'Sample Feed'

You can also construct a HTTPBasicAuthHandler that contains the password information, then pass that as a handler to the parse function. HTTPBasicAuthHandler is part of the standard :file:`urllib2 <http://docs.python.org/lib/module-urllib2.html>`_ module.

Downloading a feed protected by HTTP basic authentication (the hard way) ————————————————————————import urllib2, feedparser

# Construct the authentication handler auth = urllib2.HTTPBasicAuthHandler()

# Add password information: realm, host, user, password. # A single handler can contain passwords for multiple sites; # urllib2 will sort out which passwords get sent to which sites # based on the realm and host of the URL you’re retrieving auth.add_password(‘BasicTest’, ‘feedparser.org’, ‘test’, ‘basic’)

# Pass the authentication handler to the feed parser. # handlers is a list because there might be more than one # type of handler (urllib2 defines lots of different ones, # and you can build your own) d = feedparser.parse(‘http://feedparser.org/docs/examples/basic_auth.xml‘, \ handlers=[auth])

Digest authentication is handled in much the same way, by constructing an HTTPDigestAuthHandler and populating it with the necessary realm, host, user, and password information. This is more secure than stuffing the username and password in the URL, since the password will be encrypted before being sent to the server.

Downloading a feed protected by HTTP digest authentication (the secure way) —————————————————————————import urllib2, feedparser

auth = urllib2.HTTPDigestAuthHandler() auth.add_password(‘DigestTest’, ‘feedparser.org’, ‘test’, ‘digest’) d = feedparser.parse(‘http://feedparser.org/docs/examples/digest_auth.xml‘, \ handlers=[auth])

The examples so far have assumed that you know in advance that the feed is password-protected. But what if you don’t know?

If you try to download a password-protected feed without sending all the proper password information, the server will return an HTTP status code 401. Universal Feed Parser makes this status code available in d.status.

Details on the authentication scheme are in d.headers['www-authenticate']. Universal Feed Parser does not do any further parsing on this field; you will need to parse it yourself. Everything before the first space is the type of authentication (probably Basic or Digest), which controls which type of handler you’ll need to construct. The realm name is given as realm=”foo” – so foo would be your first argument to auth.add_password. Other information in the www-authenticate header is probably safe to ignore; the urllib2 module will handle it for you.

Determining that a feed is password-protected¶

>>> import feedparser
>>> d = feedparser.parse('http://feedparser.org/docs/examples/basic_auth.xml')
>>> d.status``401``
>>> d.headers['www-authenticate']``'Basic realm="Use test/basic"'``
>>> d = feedparser.parse('http://feedparser.org/docs/examples/digest_auth.xml')
>>> d.status``401``
>>> d.headers['www-authenticate']``'Digest realm="DigestTest",
nonce="+LV/uLLdAwA=5d77397291261b9ef256b034e19bcb94f5b7992a",
algorithm=MD5,
qop="auth"'

filename=”http-other.html”

You can specify extra HTTP request headers as a dictionary. When you download a feed from a remote web server, Universal Feed Parser exposes the complete set of HTTP response headers as a dictionary.

Sending custom HTTP request headers¶

>>> import feedparser
>>> d = feedparser.parse('http://feedparser.org/docs/examples/atom03.xml',
extra_headers={'Cache-control': 'max-age=0'})``

Accessing other HTTP response headers¶

>>> import feedparser
>>> d = feedparser.parse('` <http://feedparser.org/docs/examples/atom03.xml>`_')
>>> d.headers``{'date': 'Fri, 11 Jun 2004 23:57:50 GMT',
'server': 'Apache/2.0.49 (Debian GNU/Linux)',
'last-modified': 'Fri, 11 Jun 2004 23:00:34 GMT',
'etag': '"6c132-941-ad7e3080"',
'accept-ranges': 'bytes',
'vary': 'Accept-Encoding,User-Agent',
'content-encoding': 'gzip',
'content-length': '883',
'connection': 'close',
'content-type': 'application/xml'}

filename=”command-line.html”

Available options¶

``mark@atlantis$

./feedparser.py –help``usage: feedparser.py [options] url_or_filename_or_-

options: –version show program’s version number and exit -h, –help show this help message and exit -A AGENT, –user-agent=AGENT User-Agent for HTTP URLs -e URL, –referer=URL, –referrer=URL Referrer for HTTP URLs -t TAG, –etag=TAG ETag/If-None-Match for HTTP URLs -m DATE, –last-modified=DATE Last-modified/If-Modified-Since for HTTP URLs (any supported date format) -f FORMAT, –format=FORMAT output results in FORMAT (text, pprint) -v, –verbose write debugging information to stderr

filename=”command-line-output-formats.html”

TODO

Annotated Atom 1.0 feed¶

<?xml version=”1.0” encoding=”utf-8”?> <feed xmlns=”http://www.w3.org/2005/Atom” xml:base=”http://example.org/” xml:lang=”en“> <title type=”text“> Sample Feed </title> <subtitle type=”html“> For documentation &lt;em&gt;only&lt;/em&gt; </subtitle> <link rel=”alternate” type=”html” href=”/“/> <link rel=”self” type=”application/atom+xml” href=”http://www.example.org/atom10.xml“/> <rights type=”html“> &lt;p>Copyright 2005, Mark Pilgrim&lt;/p> </rights> <generator uri=”http://example.org/generator/” version=”4.0“> Sample Toolkit </generator> <id>:ref:tag:feedparser.org,2005-11-09:/docs/examples/atom10.xml <reference.feed.id>`</id> <updated>:ref:`2005-11-09T11:56:34Z <reference.feed.updated>`</updated> <entry> <title>:ref:`First entry title <reference.entry.title>`</title> <link rel=”:ref:`alternate <reference.entry.links.type>” href=”/entry/3“/> <link rel=”related” type=”text/html” href=”http://search.example.com/“/> <link rel=”via” type=”text/html” href=”http://toby.example.com/examples/atom10“/> <link rel=”enclosure” type=”video/mpeg4” href=”http://www.example.com/movie.mp4” length=”42301“/> <id>:ref:tag:feedparser.org,2005-11-09:/docs/examples/atom10.xml:3 <reference.entry.id>`</id> <published>:ref:`2005-11-09T00:23:47Z <reference.entry.published>`</published> <updated>:ref:`2005-11-09T11:56:34Z <reference.entry.updated>`</updated> <author> <name>:ref:`Mark Pilgrim <reference.entry.author_detail.name>`</name> <uri>:ref:`http://diveintomark.org/ <reference.entry.author_detail.href>`</uri> <email>:ref:`mark@example.org <reference.entry.author_detail.email>`</email> </author> <contributor> <name>:ref:`Joe <reference.entry.contributors.name>`</name> <url>:ref:`http://example.org/joe/ <reference.entry.contributors.href>`</url> <email>:ref:`joe@example.org <reference.entry.contributors.email>`</email> </contributor> <contributor> <name>:ref:`Sam <reference.entry.contributors.name>`</name> <url>:ref:`http://example.org/sam/ <reference.entry.contributors.href>`</url> <email>:ref:`sam@example.org <reference.entry.contributors.email>`</email> </contributor> <summary type=”:ref:`text <reference.entry.summary_detail.type>“> Watch out for nasty tricks </summary> <content type=”xhtml” xml:base=”http://example.org/entry/3” xml:lang=”en-US“>:ref:<div xmlns=”http://www.w3.org/1999/xhtml”>Watch out for <span style=”background-image: url(javascript:window.location=’http://example.org/’)”> nasty tricks</span></div> <reference.entry.content.value> </content> </entry> </feed>

filename=”annotated-atom03.html”

This is a sample Atom 0.3 feed, annotated with links that show how each value can be accessed once the feed is parsed.

feed.title_detail.value¶

Same as reference.feed.title.

If this contains HTML or XHTML, it is sanitized by default.

If this contains HTML or XHTML, certain (X)HTML elements within this value may contain relative URIs. If so, they are resolved according to a set of rules.

feed.title_detail.type¶

The content type of the feed title.

Most likely values for type:

• text/plain
• text/html
• application/xhtml+xml

For Atom feeds, the content type is taken from the type attribute, which defaults to text/plain if not specified. For RSS feeds, the content type is auto-determined by inspecting the content, and defaults to text/html. Note that this may cause silent data loss if the value contains plain text with angle brackets. There is nothing I can do about this problem; it is a limitation of RSS.

Future enhancement: some versions of RSS clearly specify that certain values default to text/plain, and Universal Feed Parser should respect this, but it doesn’t yet.

feed.title_detail.language¶

The language of the feed title.

language is supposed to be a language code, as specified by RFC 3066, but publishers have been known to publish random values like English or German. Universal Feed Parser does not do any parsing or normalization of language codes.

language may come from the element’s xml:lang attribute, or it may inherit from a parent element’s xml:lang, or the Content-Language HTTP header. If the feed does not specify a language, language will be None, the Python null value.

feed.title_detail.base¶

The original base URI for links within the feed title.

base is only useful in rare situations and can usually be ignored. It is the original base URI for this value, as specified by the element’s xml:base attribute, or a parent element’s xml:base, or the appropriate HTTP header, or the URI of the feed. (See advanced.base for more details.) By the time you see it, Universal Feed Parser has already resolved relative links in all values where it makes sense to do so. Clients should never need to manually resolve relative links.

filename=”reference-feed-link.html”

feed.links[i].rel¶

The relationship of this feed link.

Atom 1.0 defines five standard link relationships and describes the process for registering others. Here are the five standard rel values:

• alternate
• self
• related
• via
• enclosure

feed.links[i].type¶

The content type of the page that this feed link points to.

feed.links[i].href¶

The URL of the page that this feed link points to.

If this is a relative URI, it is resolved according to a set of rules.

feed.links[i].title¶

The title of this feed link.

filename=”reference-feed-subtitle.html”

feed.subtitle_detail.value¶

Same as reference.feed.subtitle.

If this contains HTML or XHTML, it is sanitized by default.

If this contains HTML or XHTML, certain (X)HTML elements within this value may contain relative URIs. If so, they are resolved according to a set of rules.

feed.subtitle_detail.type¶

The content type of the feed subtitle.

Most likely values for type:

• text/plain
• text/html
• application/xhtml+xml

For Atom feeds, the content type is taken from the type attribute, which defaults to text/plain if not specified. For RSS feeds, the content type is auto-determined by inspecting the content, and defaults to text/html. Note that this may cause silent data loss if the value contains plain text with angle brackets. There is nothing I can do about this problem; it is a limitation of RSS.

Future enhancement: some versions of RSS clearly specify that certain values default to text/plain, and Universal Feed Parser should respect this, but it doesn’t yet.

feed.subtitle_detail.language¶

The language of the feed subtitle.

language is supposed to be a language code, as specified by RFC 3066, but publishers have been known to publish random values like English or German. Universal Feed Parser does not do any parsing or normalization of language codes.

language may come from the element’s xml:lang attribute, or it may inherit from a parent element’s xml:lang, or the Content-Language HTTP header. If the feed does not specify a language, language will be None, the Python null value.

feed.subtitle_detail.base¶

The original base URI for links within the feed subtitle.

base is only useful in rare situations and can usually be ignored. It is the original base URI for this value, as specified by the element’s xml:base attribute, or a parent element’s xml:base, or the appropriate HTTP header, or the URI of the feed. (See advanced.base for more details.) By the time you see it, Universal Feed Parser has already resolved relative links in all values where it makes sense to do so. Clients should never need to manually resolve relative links.

filename=”reference-feed-rights.html”

feed.rights_detail.value¶

Same as reference.feed.rights.

If this contains HTML or XHTML, it is sanitized by default.

If this contains HTML or XHTML, certain (X)HTML elements within this value may contain relative URIs. If so, they are resolved according to a set of rules.

feed.rights_detail.type¶

The content type of the feed copyright.

Most likely values for type:

• text/plain
• text/html
• application/xhtml+xml

For Atom feeds, the content type is taken from the type attribute, which defaults to text/plain if not specified. For RSS feeds, the content type is auto-determined by inspecting the content, and defaults to text/html. Note that this may cause silent data loss if the value contains plain text with angle brackets. There is nothing I can do about this problem; it is a limitation of RSS.

Future enhancement: some versions of RSS clearly specify that certain values default to text/plain, and Universal Feed Parser should respect this, but it doesn’t yet.

feed.rights_detail.language¶

The language of the feed copyright.

language is supposed to be a language code, as specified by RFC 3066, but publishers have been known to publish random values like English or German. Universal Feed Parser does not do any parsing or normalization of language codes.

language may come from the element’s xml:lang attribute, or it may inherit from a parent element’s xml:lang, or the Content-Language HTTP header. If the feed does not specify a language, language will be None, the Python null value.

feed.rights_detail.base¶

The original base URI for links within the feed copyright.

base is only useful in rare situations and can usually be ignored. It is the original base URI for this value, as specified by the element’s xml:base attribute, or a parent element’s xml:base, or the appropriate HTTP header, or the URI of the feed. (See advanced.base for more details.) By the time you see it, Universal Feed Parser has already resolved relative links in all values where it makes sense to do so. Clients should never need to manually resolve relative links.

filename=”reference-feed-generator.html”

feed.generator_detail.name¶

Same as reference.feed.generator.

feed.generator_detail.href¶

The URL of the application used to generate the feed.

If this is a relative URI, it is resolved according to a set of rules.

feed.generator_detail.version¶

The version number of the application used to generate the feed. There is no required format for this, but most applications use a MAJOR.MINOR version number.

filename=”reference-feed-info.html”

feed.info_detail.value¶

Same as reference.feed.info.

If this contains HTML or XHTML, it is sanitized by default.

If this contains HTML or XHTML, certain (X)HTML elements within this value may contain relative URIs. If so, they are resolved according to a set of rules.

feed.info_detail.type¶

The content type of the feed info.

Most likely values for type:

• text/plain
• text/html
• application/xhtml+xml

For Atom feeds, the content type is taken from the type attribute, which defaults to text/plain if not specified. For RSS feeds, the content type is auto-determined by inspecting the content, and defaults to text/html. Note that this may cause silent data loss if the value contains plain text with angle brackets. There is nothing I can do about this problem; it is a limitation of RSS.

Future enhancement: some versions of RSS clearly specify that certain values default to text/plain, and Universal Feed Parser should respect this, but it doesn’t yet.

feed.info_detail.language¶

The language of the feed info.

language is supposed to be a language code, as specified by RFC 3066, but publishers have been known to publish random values like English or German. Universal Feed Parser does not do any parsing or normalization of language codes.

language may come from the element’s xml:lang attribute, or it may inherit from a parent element’s xml:lang, or the Content-Language HTTP header. If the feed does not specify a language, language will be None, the Python null value.

feed.info_detail.base¶

The original base URI for links within the feed copyright.

base is only useful in rare situations and can usually be ignored. It is the original base URI for this value, as specified by the element’s xml:base attribute, or a parent element’s xml:base, or the appropriate HTTP header, or the URI of the feed. (See advanced.base for more details.) By the time you see it, Universal Feed Parser has already resolved relative links in all values where it makes sense to do so. Clients should never need to manually resolve relative links.

filename=”reference-feed-updated.html”

feed.author_detail.name¶

The name of the feed author.

feed.author_detail.href¶

The URL of the feed author. This can be the author’s home page, or a contact page with a webmail form.

If this is a relative URI, it is resolved according to a set of rules.

feed.author_detail.email¶

The email address of the feed author.

filename=”reference-feed-contributors.html”

feed.contributors[i].name¶

The name of this contributor.

feed.contributors[i].href¶

The URL of this contributor. This can be the contributor’s home page, or a contact page with a webmail form.

If this is a relative URI, it is resolved according to a set of rules.

feed.contributors[i].email¶

The email address of this contributor.

filename=”reference-feed-image.html”

feed.image.title¶

The alternate text of the feed image, which would go in the alt attribute if you rendered the feed image as an HTML img element.

feed.image.href¶

The URL of the feed image itself, which would go in the src attribute if you rendered the feed image as an HTML img element.

If this is a relative URI, it is resolved according to a set of rules.

feed.image.link¶

The URL which the feed image would point to. If you rendered the feed image as an HTML img element, you would wrap it in an a element and put this in the href attribute.

If this is a relative URI, it is resolved according to a set of rules.

feed.image.width¶

The width of the feed image, which would go in the width attribute if you rendered the feed image as an HTML img element.

feed.image.height¶

The height of the feed image, which would go in the height attribute if you rendered the feed image as an HTML img element.

feed.image.description¶

A short description of the feed image, which would go in the title attribute if you rendered the feed image as an HTML img element. This element is rare; it was available in Netscape RSS 0.91 but was dropped from Userland RSS 0.91.

Annotated example¶

This is a feed image:<image> <title>Feed logo</title> <url>http://example.org/logo.png</url> <link>http://example.org/</link> <width>80</width> <height>15</height> <description>Visit my home page</description> </image>

This feed image could be rendered in HTML as this:<a href=”http://example.org/“> <img src=”http://example.org/logo.png” width=”80” height=”15” alt=”Feed logo” title=”Visit my home page”> </a>

filename=”reference-feed-icon.html”

feed.textinput.title¶

The title of the text input form, which would go in the value attribute of the form’s submit button.

feed.textinput.link¶

The link of the script which processes the text input form, which would go in the action attirbute of the form.

If this is a relative URI, it is resolved according to a set of rules.

feed.textinput.name¶

The name of the text input box in the form, which would go in the name attribute of the form’s input box.

feed.textinput.description¶

A short description of the text input form, which would go in the label element of the form.

Annotated example¶

This is a text input in a feed:<textInput> <title>Go!</title> <link>http://example.org/search</link> <name>keyword</name> <description>Search this site:</description> </textInput>

This is how it could be rendered in HTML:<form method=”get” action=”http://example.org/search“> <label for=”keyword”>Search this site:</label> <input type=”text” id=”keyword” name=”keyword” value=”“> <input type=”submit” value=”Go!”> </form>

filename=”reference-feed-cloud.html”

feed.cloud.domain¶

The domain of the cloud. Should be just the domain name, not including the http:// protocol. All clouds are presumed to operate over HTTP. The cloud specification does not support secure clouds over HTTPS, nor can clouds operate over other protocols.

feed.cloud.port¶

The port of the cloud. Should be an integer, but Universal Feed Parser currently returns it as a string.

feed.cloud.path¶

The URL path of the cloud.

feed.cloud.registerProcedure¶

The name of the procedure to call on the cloud.

feed.cloud.protocol¶

The protocol of the cloud. Documentation differs on what the acceptable values are. Acceptable values definitely include xml-rpc and soap, although only in lowercase, despite both being acronyms.

There is no way for a publisher to specify the version number of the protocol to use. soap refers to SOAP 1.1; the cloud interface does not support SOAP 1.0 or 1.2.

post or http-post might also be acceptable values; nobody really knows for sure.

filename=”reference-feed-publisher.html”

feed.publisher_detail.name¶

The name of this feed’s publisher.

feed.publisher_detail.href¶

The URL of this feed’s publisher. This can be the publisher’s home page, or a contact page with a webmail form.

If this is a relative URI, it is resolved according to a set of rules.

feed.publisher_detail.email¶

The email address of this feed’s publisher.

filename=”reference-feed-tags.html”

feed.tags[i].term¶

The category term (keyword).

feed.tags[i].scheme¶

The category scheme (domain).

feed.tags[i].label¶

A human-readable label for the category.

filename=”reference-feed-docs.html”

entries[i].title_detail.value¶

Same as reference.entry.title.

If this contains HTML or XHTML, it is sanitized by default.

If this contains HTML or XHTML, certain (X)HTML elements within this value may contain relative URIs. If so, they are resolved according to a set of rules.

entries[i].title_detail.type¶

The content type of the entry title.

Most likely values for type:

• text/plain
• text/html
• application/xhtml+xml

For Atom feeds, the content type is taken from the type attribute, which defaults to text/plain if not specified. For RSS feeds, the content type is auto-determined by inspecting the content, and defaults to text/html. Note that this may cause silent data loss if the value contains plain text with angle brackets. There is nothing I can do about this problem; it is a limitation of RSS.

Future enhancement: some versions of RSS clearly specify that certain values default to text/plain, and Universal Feed Parser should respect this, but it doesn’t yet.

entries[i].title_detail.language¶

The language of the entry title.

language is supposed to be a language code, as specified by RFC 3066, but publishers have been known to publish random values like English or German. Universal Feed Parser does not do any parsing or normalization of language codes.

language may come from the element’s xml:lang attribute, or it may inherit from a parent element’s xml:lang, or the Content-Language HTTP header. If the feed does not specify a language, language will be None, the Python null value.

entries[i].title_detail.base¶

The original base URI for links within the entry title.

base is only useful in rare situations and can usually be ignored. It is the original base URI for this value, as specified by the element’s xml:base attribute, or a parent element’s xml:base, or the appropriate HTTP header, or the URI of the feed. (See advanced.base for more details.) By the time you see it, Universal Feed Parser has already resolved relative links in all values where it makes sense to do so. Clients should never need to manually resolve relative links.

filename=”reference-entry-link.html”

entries[i].links[j].rel¶

The relationship of this entry link.

Atom 1.0 defines five standard link relationships and describes the process for registering others. Here are the five standard rel values:

• alternate
• self
• related
• via
• enclosure

entries[i].links[j].type¶

The content type of the page that this entry link points to.

entries[i].links[j].href¶

The URL of the page that this entry link points to.

If this is a relative URI, it is resolved according to a set of rules.

entries[i].links[j].title¶

The title of this entry link.

filename=”reference-entry-summary.html”

entries[i].summary_detail.value¶

Same as reference.entry.summary.

If this contains HTML or XHTML, it is sanitized by default.

If this contains HTML or XHTML, certain (X)HTML elements within this value may contain relative URIs. If so, they are resolved according to a set of rules.

If this contains HTML or XHTML, it will be parsed for microformats.

entries[i].summary_detail.type¶

The content type of the entry summary.

Most likely values for type:

• text/plain
• text/html
• application/xhtml+xml

For Atom feeds, the content type is taken from the type attribute, which defaults to text/plain if not specified. For RSS feeds, the content type is auto-determined by inspecting the content, and defaults to text/html. Note that this may cause silent data loss if the value contains plain text with angle brackets. There is nothing I can do about this problem; it is a limitation of RSS.

Future enhancement: some versions of RSS clearly specify that certain values default to text/plain, and Universal Feed Parser should respect this, but it doesn’t yet.

entries[i].summary_detail.language¶

The language of the entry summary.

language is supposed to be a language code, as specified by RFC 3066, but publishers have been known to publish random values like English or German. Universal Feed Parser does not do any parsing or normalization of language codes.

language may come from the element’s xml:lang attribute, or it may inherit from a parent element’s xml:lang, or the Content-Language HTTP header. If the feed does not specify a language, language will be None, the Python null value.

entries[i].summary_detail.base¶

The original base URI for links within the entry summary.

base is only useful in rare situations and can usually be ignored. It is the original base URI for this value, as specified by the element’s xml:base attribute, or a parent element’s xml:base, or the appropriate HTTP header, or the URI of the feed. (See advanced.base for more details.) By the time you see it, Universal Feed Parser has already resolved relative links in all values where it makes sense to do so. Clients should never need to manually resolve relative links.

filename=”reference-entry-content.html”

entries[i].content[j].value¶

The value of this piece of content.

If this contains HTML or XHTML, it is sanitized by default.

If this contains HTML or XHTML, certain (X)HTML elements within this value may contain relative URIs. If so, they are resolved according to a set of rules.

If this contains HTML or XHTML, it will be parsed for microformats.

entries[i].content[j].type¶

The content type of this piece of content.

Most likely values for type:

• text/plain
• text/html
• application/xhtml+xml

For Atom feeds, the content type is taken from the type attribute, which defaults to text/plain if not specified. For RSS feeds, the content type is auto-determined by inspecting the content, and defaults to text/html. Note that this may cause silent data loss if the value contains plain text with angle brackets. There is nothing I can do about this problem; it is a limitation of RSS.

Future enhancement: some versions of RSS clearly specify that certain values default to text/plain, and Universal Feed Parser should respect this, but it doesn’t yet.

entries[i].content[j].language¶

The language of this piece of content.

language is supposed to be a language code, as specified by RFC 3066, but publishers have been known to publish random values like English or German. Universal Feed Parser does not do any parsing or normalization of language codes.

language may come from the element’s xml:lang attribute, or it may inherit from a parent element’s xml:lang, or the Content-Language HTTP header. If the feed does not specify a language, language will be None, the Python null value.

entries[i].content[j].base¶

The original base URI for links within this piece of content.

base is only useful in rare situations and can usually be ignored. It is the original base URI for this value, as specified by the element’s xml:base attribute, or a parent element’s xml:base, or the appropriate HTTP header, or the URI of the feed. (See advanced.base for more details.) By the time you see it, Universal Feed Parser has already resolved relative links in all values where it makes sense to do so. Clients should never need to manually resolve relative links.

filename=”reference-entry-published.html”

entries[i].author_detail.name¶

The name of this entry’s author.

entries[i].author_detail.href¶

The URL of this entry’s author. This can be the author’s home page, or a contact page with a webmail form.

If this is a relative URI, it is resolved according to a set of rules.

entries[i].author_detail.email¶

The email address of this entry’s author.

filename=”reference-entry-contributors.html”

entries[i].contributors[j].name¶

The name of this contributor.

entries[i].contributors[j].href¶

The URL of this contributor. This can be the contributor’s home page, or a contact page with a webmail form.

If this is a relative URI, it is resolved according to a set of rules.

entries[i].contributors[j].email¶

The email address of this contributor.

filename=”reference-entry-enclosures.html”

entries[i].enclosures[j].href¶

The URL of the linked file.

If this is a relative URI, it is resolved according to a set of rules.

entries[i].enclosures[j].length¶

The length of the linked file.

entries[i].enclosures[j].type¶

The content type of the linked file.

filename=”reference-entry-publisher.html”

entries[i].publisher_detail.name¶

The name of this entry’s publisher.

entries[i].publisher_detail.href¶

The URL of this entry’s publisher. This can be the publisher’s home page, or a contact page with a webmail form.

If this is a relative URI, it is resolved according to a set of rules.

entries[i].publisher_detail.email¶

The email address of this entry’s publisher.

filename=”reference-entry-tags.html”

entries[i].tags[j].term¶

The category term (keyword).

entries[i].tags[j].scheme¶

The category scheme (domain).

entries[i].tags[j].label¶

A human-readable label for the category.

filename=”reference-entry-source.html”

entries[i].source.author¶

The author of the source of this entry.

entries[i].source.author_detail¶

A dictionary containing details about the author of the source of this entry.

The name of the author of the source of this entry.

The URL of the author of the source of this entry. This can be the author’s home page, or a contact page with a webmail form.

If this is a relative URI, it is resolved according to a set of rules.

The email address of the author of the source of this entry.

entries[i].source.contributors¶

A list of contributors to the source of this entry.

The name of a contributor to the source of this entry.

The URL of a contributor to the source of this entry. This can be the contributor’s home page, or a contact page with a webmail form.

If this is a relative URI, it is resolved according to a set of rules.

The email address of a contributor to the source of this entry.

entries[i].source.icon¶

The URL of an icon representing the source of this entry.

If this is a relative URI, it is resolved according to a set of rules.

entries[i].source.id¶

A globally unique identifier for the source of this entry.

entries[i].source.link¶

The primary permanent link of the source of this entry

entries[i].source.links¶

A list of all links defined by the source of this entry.

The relationship of a link defined by the source of this entry.

Atom 1.0 defines five standard link relationships and describes the process for registering others. Here are the five standard rel values:

• alternate
• self
• related
• via
• enclosure

The content type of the page pointed to by a link defined by the source of this entry.

The URL of the page pointed to by a link defined by the source of this entry.

If this is a relative URI, it is resolved according to a set of rules.

The title of a link defined by the source of this entry.

entries[i].source.logo¶

The URL of a logo representing the source of this entry.

If this is a relative URI, it is resolved according to a set of rules.

entries[i].source.rights¶

A human-readable copyright statement for the source of this entry.

entries[i].source.rights_detail¶

A dictionary containing details about the copyright statement for the source of this entry.

Same as entries[i].source.rights.

If this contains HTML or XHTML, it is sanitized by default.

If this contains HTML or XHTML, certain (X)HTML elements within this value may contain relative URIs. If so, they are resolved according to a set of rules.

The content type of the copyright statement for the source of this entry.

Most likely values for type:

• text/plain
• text/html
• application/xhtml+xml

For Atom feeds, the content type is taken from the type attribute, which defaults to text/plain if not specified. For RSS feeds, the content type is auto-determined by inspecting the content, and defaults to text/html. Note that this may cause silent data loss if the value contains plain text with angle brackets. There is nothing I can do about this problem; it is a limitation of RSS.

Future enhancement: some versions of RSS clearly specify that certain values default to text/plain, and Universal Feed Parser should respect this, but it doesn’t yet.

The language of the copyright statement for the source of this entry.

language is supposed to be a language code, as specified by RFC 3066, but publishers have been known to publish random values like English or German. Universal Feed Parser does not do any parsing or normalization of language codes.

language may come from the element’s xml:lang attribute, or it may inherit from a parent element’s xml:lang, or the Content-Language HTTP header. If the feed does not specify a language, language will be None, the Python null value.

The original base URI for links within the copyright statement for the source of this entry.

base is only useful in rare situations and can usually be ignored. It is the original base URI for this value, as specified by the element’s xml:base attribute, or a parent element’s xml:base, or the appropriate HTTP header, or the URI of the feed. (See advanced.base for more details.) By the time you see it, Universal Feed Parser has already resolved relative links in all values where it makes sense to do so. Clients should never need to manually resolve relative links.

entries[i].source.subtitle¶

A subtitle, tagline, slogan, or other short description of the source of this entry.

If this contains HTML or XHTML, it is sanitized by default.

If this contains HTML or XHTML, certain (X)HTML elements within this value may contain relative URIs. If so, they are resolved according to a set of rules.

entries[i].source.subtitle_detail¶

A dictionary containing details about the subtitle for the source of this entry.

Same as entries[i].source.subtitle.

If this contains HTML or XHTML, it is sanitized by default.

If this contains HTML or XHTML, certain (X)HTML elements within this value may contain relative URIs. If so, they are resolved according to a set of rules.

The content type of the subtitle of the source of this entry.

Most likely values for type:

• text/plain
• text/html
• application/xhtml+xml

For Atom feeds, the content type is taken from the type attribute, which defaults to text/plain if not specified. For RSS feeds, the content type is auto-determined by inspecting the content, and defaults to text/html. Note that this may cause silent data loss if the value contains plain text with angle brackets. There is nothing I can do about this problem; it is a limitation of RSS.

Future enhancement: some versions of RSS clearly specify that certain values default to text/plain, and Universal Feed Parser should respect this, but it doesn’t yet.

The language of the subtitle of the source of this entry.

language is supposed to be a language code, as specified by RFC 3066, but publishers have been known to publish random values like English or German. Universal Feed Parser does not do any parsing or normalization of language codes.

language may come from the element’s xml:lang attribute, or it may inherit from a parent element’s xml:lang, or the Content-Language HTTP header. If the feed does not specify a language, language will be None, the Python null value.

The original base URI for links within the subtitle of the source of this entry.

base is only useful in rare situations and can usually be ignored. It is the original base URI for this value, as specified by the element’s xml:base attribute, or a parent element’s xml:base, or the appropriate HTTP header, or the URI of the feed. (See advanced.base for more details.) By the time you see it, Universal Feed Parser has already resolved relative links in all values where it makes sense to do so. Clients should never need to manually resolve relative links.

entries[i].source.title¶

The title of the source of this entry.

If this contains HTML or XHTML, it is sanitized by default.

If this contains HTML or XHTML, certain (X)HTML elements within this value may contain relative URIs. If so, they are resolved according to a set of rules.

entries[i].source.title_detail¶

A dictionary containing details about the title for the source of this entry.

Same as entries[i].source.title.

If this contains HTML or XHTML, it is sanitized by default.

If this contains HTML or XHTML, certain (X)HTML elements within this value may contain relative URIs. If so, they are resolved according to a set of rules.

The content type of the title of the source of this entry.

Most likely values for type:

• text/plain
• text/html
• application/xhtml+xml

For Atom feeds, the content type is taken from the type attribute, which defaults to text/plain if not specified. For RSS feeds, the content type is auto-determined by inspecting the content, and defaults to text/html. Note that this may cause silent data loss if the value contains plain text with angle brackets. There is nothing I can do about this problem; it is a limitation of RSS.

Future enhancement: some versions of RSS clearly specify that certain values default to text/plain, and Universal Feed Parser should respect this, but it doesn’t yet.

The language of the title of the source of this entry.

language is supposed to be a language code, as specified by RFC 3066, but publishers have been known to publish random values like English or German. Universal Feed Parser does not do any parsing or normalization of language codes.

language may come from the element’s xml:lang attribute, or it may inherit from a parent element’s xml:lang, or the Content-Language HTTP header. If the feed does not specify a language, language will be None, the Python null value.

The original base URI for links within the title of the source of this entry.

base is only useful in rare situations and can usually be ignored. It is the original base URI for this value, as specified by the element’s xml:base attribute, or a parent element’s xml:base, or the appropriate HTTP header, or the URI of the feed. (See advanced.base for more details.) By the time you see it, Universal Feed Parser has already resolved relative links in all values where it makes sense to do so. Clients should never need to manually resolve relative links.

entries[i].source.updated¶

The date the source of this entry was last updated, as a string in the same format as it was published in the original feed.

This element is parsed as a date and stored in entries[i].source.updated_parsed.

entries[i].source.updated_parsed¶

The date this entry was last updated, as a standard Python 9-tuple.

filename=”reference-entry-comments.html”

entries[i].xfn[j].relationships¶

A list of relationships for this person. Each list item is a string, either one of the constants defined in the XFN 1.1 profile or one of these variations.

entries[i].xfn[j].href¶

The URI for this person.

If this is a relative URI, it is resolved according to a set of rules.

entries[i].xfn[j].name¶

The name of this person, a string.

filename=”reference-entry-vcard.html”