bufrpy¶

Bufrpy is a pure-Python BUFR message decoder. BUFR messages are typically used to transmit meteorological observations. Bufrpy was developed to work with NWCSAF RDT and HRW data, but there should not be any reason why it could not be used with any BUFR messages, with the following limitations:

Messages with multiple data subsets are not supported
Compressed messages are not supported
Operator descriptors are not supported
Sequence descriptors are not supported

Adding support for any of the above should be feasible, but we have not encountered such data in our use of the library.

Examples¶

Usage examples

Examples¶

RDT Contour Extraction¶

import bufrpy
from bufrpy.template import safnwc
import re
import itertools
import sys

def to_rings(bufrmsg):
    """ Return a list of cloud contours as closed rings, in lat/lon ordinate order"""
    out = []
    subsets = bufrmsg.section4.subsets
    for subset in subsets:
        for cloudsystem in subset.values[19]: # cloud systems are at index 19
            contour = cloudsystem[0] # contour is at index 0 of cloud system
            point_vals = list(itertools.chain(contour, itertools.islice(contour,1))) # close contour
            points = []
            for (lat, lon) in point_vals:
                points.append((lat.value, lon.value)) # extract values
            out.append(points)
    return out

def to_wkt(rings):
    out = []
    for ring in rings:
        # Change ordinate order, convert to strings, join and stick in a POLYGON
        points = [" ".join(map(str, reversed(latlon))) for latlon in ring]
        edge = ",".join(points)
        out.append("POLYGON((" + edge + "))")
    return "\n".join(out)

if __name__ == '__main__':

    template = safnwc.read_template(open(sys.argv[1], 'rb'))
    msg = bufrpy.decode_file(open(sys.argv[2], 'rb'), template)

    rings = to_rings(msg)
    wkt = to_wkt(rings)

    print wkt

Data Model¶

Data model

Data Model¶

Bufrpy data model matches that of the BUFR messages fairly closely, though not as closely as libbufr or the Python libbufr wrappers. Typically BUFR messages are stored in a file so that the file contains a single BUFR message. Some applications (e.g. SAFNWC HWR) store multiple messages in a single file, requiring some tricks to read them.

The top-level data structure in bufrpy is Message that corresponds to one BUFR message. A BUFR message contains 6 sections, some of which contain metadata and some the actual data. The most important sections are numbers 3 and 4. They contain data descriptors and the data itself respectively. Logically BUFR files contain at least one subset of data. Each subset follows the same structure, defined by the data descriptors in section 3. In bufrpy, sections are attributes of Message. Section 4 contains the data subsets as a list of BufrSubset objects.

Logically each subset is a linear sequence of data values. All metadata about the data is stored in the descriptors. However, there are certain looping constructs (replication descriptors) that allow repetition of a sequence of data items. This leads to hierarchical logical structure. Physically the data is still stored in a linear sequence (or in interleaved fashion if a message has multiple data subsets). In bufrpy the data of each dataset is represented by a list of BufrValue objects. A replicated sequence in the data is represented by a list of lists of BufrValue objects so that the outer list contains one list for each repetition, and the inner lists are equivalent to values of a BufrSubset. Hierarchical data is thus naturally represented as nested lists.

API Documentation¶

API documentation covers the public API

API¶

Reading BUFR messages¶

bufrpy.decode_file(f, b_table)¶

Decode BUFR message from a file into a Message object.

Parameters:	f (file) – File that contains the bufr message b_table (Mapping\|Template) – Either a mapping from BUFR descriptor codes to descriptors or a Template describing the message

bufrpy.decode(stream, b_table, skip_data=False)¶

Decode BUFR message from stream into a Message object.

See WMO306_vl2_BUFR3_Spec_en.pdf for BUFR format specification.

Parameters:	stream (ByteStream) – Stream that contains the bufr message b_table (Mapping\|Template) – Either a mapping from BUFR descriptor codes to descriptors or a Template describing the message skip_data (bool) – Skip decoding data? Can be used to get only extract metadata of a file to e.g. analysis of decoding errors.

bufrpy.decode_all(stream, b_table)¶

Decode all BUFR messages from stream into a list of Message objects and a list of decoding errors.

Reads through the stream, decoding well-formed BUFR messages. BUFR messages must start with BUFR and end with 7777. Data between messages is skipped.

Parameters:	stream (ByteStream) – Stream that contains the bufr message b_table (Mapping\|Template) – Either a mapping from BUFR descriptor codes to descriptors or a Template describing the message

BUFR message representation¶

BUFR Message¶

BUFR Message is represented as a Message.

class bufrpy.Message¶

Represents a complete BUFR message, of either edition 3 or edition 4.

Variables:	section0 (Section0) – Section 0, start token and length section1 (Section1v3\|Section1v4) – Section 1, time and source metadata section2 (Section2) – Section 2, optional metadata, not processed section3 (Section3) – Section 3, message structure section4 (Section4) – Section 4, message contents section5 (Section5) – Section 5, end token

The Message contains several sections. Of these, sections 1, 3 and 4 are the most interesting. Section 1 contains message-level metadata, section 3 describes message structure and section 4 contains the actual data. Note that section 1 comes in multiple variants. The variant used depends on which edition of the BUFR specification was used to code the message.

class bufrpy.Section1v3¶

Section 1 of a BUFR edition 3 message.

Variables:

length (int) – Length of Section 1
master_table_id (int) – Master table identifier
originating_centre (int) – Originating/generating centre
originating_subcentre (int) – Originating/generating subcentre
update_sequence_number (int) – Update sequence number
optional_section (int) – 0 if optional section not present, 1 if present
data_category (int) – Data category (identifies Table A to be used)
data_subcategory (int) – Data subcategory
master_table_version (int) – Master table version
local_table_version (int) – Local table version
year (int) – Year (originally of century, converted to AD by adding 1900)
month (int) – Month
day (int) – Day
hour (int) – Hour
minute (int) – Minute

class bufrpy.Section1v4¶

Section 1 of a BUFR edition 4 message.

Variables:

length (int) – Length of Section 1
master_table_id (int) – Master table identifier
originating_centre (int) – Originating/generating centre
originating_subcentre (int) – Originating/generating subcentre
update_sequence_number (int) – Update sequence number
optional_section (int) – 0 if optional section not present, 1 if present
data_category (int) – Data category (identifies Table A to be used)
data_subcategory (int) – Data subcategory
local_subcategory (int) – Local subcategory
master_table_version (int) – Master table version
local_table_version (int) – Local table version
year (int) – Year (four digits)
month (int) – Month
day (int) – Day
hour (int) – Hour
minute (int) – Minute
second (int) – Second

class bufrpy.Section3¶

Section 3 of a BUFR message.

Section 3 contains descriptors that describe the actual data format. Descriptors are instances of one of the descriptor classes: descriptors.ElementDescriptor, descriptors.ReplicationDescriptor, descriptors.OperatorDescriptor, descriptors.SequenceDescriptor.

Variables:	length (int) – Length of Section 3 n_subsets (int) – Number of data subsets in the data section flags (int) – Flags describing the data set. See BUFR specification for details. descriptors – List of descriptors that describe the contents of each data subset.

class bufrpy.Section4¶

Section 4 of a BUFR message.

Section 4 contains the actual message data.

Variables:	length (int) – Length of Section 4 subsets – Message data as a list of BufrSubsets.

BUFR Template¶

BUFR Template is represented as a Template.

class bufrpy.template.Template[source]¶

BUFR message template

Describes a BUFR message by listing the descriptors that are present in the message. The same descriptors must be present in the same order in Section 3 of the actual message. The template includes descriptions, length, units and other information required to interpret the descriptor references in the message.

Possible descriptor types are ElementDescriptor, ReplicationDescriptor, SequenceDescriptor and OperatorDescriptor.

Variables:	name – Name of the template descriptors – List of descriptors

BUFR Descriptors¶

BUFR descriptors describe the data of the message. Atomic data elements are described by ElementDescriptor, multiple repetitions of a set of descriptors by ReplicationDescriptor, fixed sequences of elements by SequenceDescriptor and more complex structures by OperatorDescriptor.

class bufrpy.descriptors.ElementDescriptor[source]¶

Describes single value

Data element described with an ElementDescriptor is decoded into a BufrValue, with either textual or numeric value.

Variables:

code (int) – Descriptor code
length (int) – Length of data, in bits
scale (int) – Scaling factor applied to value to get scaled value for encoding
ref (float) – Reference value, subtracted from scaled value to get encoded value
significance (str) – Semantics of the element
unit (str) – Unit of the element, affects interpretation of the encoded data in case of e.g. textual vs. numeric values

class bufrpy.descriptors.ReplicationDescriptor[source]¶

Describes a repeating collection of values

Data described with a ReplicationDescriptor is decoded into a list of lists of values. The outer list has one element per replication and the inner lists one element per replicated field.

Variables:	code (int) – Descriptor code length (int) – Length of data, always 0 fields (int) – Number of following descriptors to replicate count (int) – Number of replications significance (str) – Meaning of the replicated list

class bufrpy.descriptors.OperatorDescriptor[source]¶

Operators 201-208 are supported.

Variables:	code (int) – Descriptor code, the whole FXY as int length (int) – Length of data. Always 0. operation (int) – Opcode, X of FXY operand (int) – Operand, Y of FXY. operator (Operator) – Operator object for checking conflicts and parsing operand. significance (str) – Operator definition.

class bufrpy.descriptors.SequenceDescriptor[source]¶

Describes a fixed sequence of elements in compact form

Similar to a replication with count 1, but the encoded form is more compact, since the sequence of fields is implicit. Except that at least in NWCSAF Templates the constituent elements of the sequence are also present in the template.

Variables:	code (int) – Descriptor code length (int) – Length of data, sum of lengths of constituent descriptors descriptor_codes (int) – Sequence containing constituent descriptor codes significance (str) – Meaning of the sequence, always empty string descriptors – Sequence containing constituent descriptors.

BUFR Values¶

BUFR values are represented a instances of BufrValue. The values of one BUFR message subset are contained in a BufrSubset.

class bufrpy.value.BufrValue[source]¶

Contains single value

Contains single value, both in raw and decoded form, plus a link to its descriptor.

Variables:	raw_value (str\|int) – Raw value. Either as hex-encoded string for textual values or an unsigned integer for numeric values value (str\|int\|float\|None) – Decoded value. Value decoded according to its descriptor. Textual values are strings and numeric values floats or ints. Missing value is indicated by `None`. descriptor (ElementDescriptor) – The descriptor of this value

class bufrpy.value.BufrSubset[source]¶

Single BUFR message data subset

Variables:	values – Subset data as a list of BufrValues.

Reading BUFR tables¶

bufrpy.table.libbufr.read_tables(b_line_stream, d_line_stream=None)[source]¶

Read BUFR table(s) in from libbufr text file(s).

The return value is a dict that combines the tables read.

Parameters:	b_line_stream – Iterable of lines, contents of the B-table file d_line_stream – Iterable of lines, contents of the D-table file
Returns:	Mapping from FXY integers to descriptors
Return type:	dict
Raises:	NotImplementedError – if the table contains sequence descriptors ValueError – if the table contains descriptors with illegal class (outside range [0,3])

Reading BUFR templates¶

bufrpy.template.safnwc.read_template(line_stream)[source]¶

Read SAFNWC message template into a Template

SAFNWC template lines look as follows:

1 001033 0 0 8 Code table Identification of originating/generating centre

Raises ValueError:
Parameters:	line_stream – Lines of SAFNWC template file
Returns:	the template as Template
Return type:	Template
	if the template contains a desccriptor outside range [0,3]

JSON encoding/decoding¶

bufrpy.to_json(msg)¶

Convert a BUFR message into JSON-encodable form.

The conversion preserves descriptors in Sections 3 and data in Section 4 but not the other metadata.

The resulting JSON is considerably larger than BUFR messages themselves, but compresses to smaller size with e.g. gzip. The JSON is also faster to read and self-descriptive, i.e. a separate descriptor table is no longer necessary.

Parameters:	msg (Message) – Message to convert
Returns:	Dict that can be converted to JSON using the standard json module
Return type:	dict

bufrpy.from_json(json_obj)¶

Convert an object decoded from JSON into a BUFR message

The conversion reads partial contents of BUFR sections 3 and 4 from a dict generated by to_json().

The resulting bufrpy.Message contains only section 3 and 4, and the sections only contain descriptors and data, respectively.

Parameters:	json_obj (dict) – JSON object to decode
Returns:	BUFR message with original descriptors and data
Return type:	Message

bufrpy¶

Examples¶

Examples¶

RDT Contour Extraction¶

Data Model¶

Data Model¶

API Documentation¶

API¶

Reading BUFR messages¶

BUFR message representation¶

BUFR Message¶

BUFR Template¶

BUFR Descriptors¶

BUFR Values¶

Reading BUFR tables¶

Reading BUFR templates¶

JSON encoding/decoding¶

Indices and tables¶