nmeta module¶

This is the main module of the nmeta suite running on top of Ryu SDN controller to provide network identity and flow (traffic classification) metadata . Do not use this code for production deployments - it is proof of concept code and carries no warrantee whatsoever. You have been warned.

class nmeta.NMeta(*args, **kwargs)¶

Bases: ryu.base.app_manager.RyuApp, baseclass.BaseClass

This is the main class used to run nmeta

OFP_VERSIONS = [1, 4]¶

_CONTEXTS = {'wsgi': <class 'ryu.app.wsgi.WSGIApplication'>}¶

_add_flow(ev, in_port, out_port, out_queue)¶

Add a flow entry to a switch Prefer to do fine-grained match where possible

_port_status_handler(ev)¶

Switch Port Status event

desc_stats_reply_handler(ev)¶

Receive a reply from a switch to a description statistics request

error_msg_handler(ev)¶

A switch has sent us an error event

flow_removed_handler(ev)¶

A switch has sent an event to us because it has removed a flow from a flow table

packet_in(ev)¶

This method is called for every Packet-In event from a Switch. We receive a copy of the Packet-In event, pass it to the traffic classification area for analysis, work out the forwarding, update flow metadata, then add a flow entry to the switch (when appropriate) to suppress receiving further packets on this flow. Finally, we send the packet out the switch port(s) via a Packet-Out message, with appropriate QoS queue set.

switch_connection_handler(ev)¶

A switch has connected to the SDN controller. We need to do some tasks to set the switch up properly such as setting it’s config for fragment handling and table miss packet length and requesting the switch description

nmeta.ipv4_text_to_int(ip_text)¶

Takes an IP address string and translates it to an unsigned integer

tc_policy module¶

This module is part of the nmeta suite running on top of Ryu SDN controller to provide network identity and flow (Traffic Classification - TC) metadata. It expects a file called “main_policy.yaml” to be in the config subdirectory containing properly formed YAML that conforms the the particular specifications that this program expects. See constant tuples at start of program for valid attributes to use.

class tc_policy.TrafficClassificationPolicy(config, pol_dir='config', pol_file='main_policy.yaml')¶

Bases: baseclass.BaseClass

This class is instantiated by nmeta.py and provides methods to ingest the policy file main_policy.yaml and check flows against policy to see if actions exist

class Condition¶

Bases: object

An object that represents a traffic classification condition, including any decision collateral on match test

to_dict()¶

Return a dictionary object of the condition object

class TrafficClassificationPolicy.Conditions¶

Bases: object

An object that represents traffic classification conditions, including any decision collateral on matches and actions

to_dict()¶

Return a dictionary object of the condition object

class TrafficClassificationPolicy.Rule¶

Bases: object

An object that represents a traffic classification rule (a set of conditions), including any decision collateral on matches and actions

to_dict()¶

Return a dictionary object of the condition object

TrafficClassificationPolicy.check_policy(flow, ident)¶

Passed a flows object, set in context of current packet-in event, and an identities object. Check if packet matches against any policy rules and if it does, update the classifications portion of the flows object to reflect details of the classification.

TrafficClassificationPolicy.qos(qos_treatment)¶

Passed a QoS treatment string and return the relevant QoS queue number to use, otherwise 0. Works by lookup on qos_treatment section of main_policy

TrafficClassificationPolicy.validate_policy()¶

Check main policy to ensure that it is in correct format so that it won’t cause unexpected errors during packet checks.

tc_static module¶

This module is part of the nmeta suite running on top of Ryu SDN controller to provide network identity and flow (traffic classification) metadata

class tc_static.StaticInspect(config)¶

Bases: baseclass.BaseClass

This class is instantiated by tc_policy.py (class: TrafficClassificationPolicy) and provides methods to query static traffic classification matches

check_static(condition, pkt)¶

Passed condition and flows packet objects Update the condition match with boolean of result of match checks

is_match_ethertype(value_to_check1, value_to_check2)¶

Passed a two prospective EtherTypes and check to see if they are the same. Return 1 for both the same EtherType and 0 for different Values can be hex or decimal and are 2 bytes in length

is_match_ip_space(ip_addr, ip_space)¶

Passed an IP address and an IP address space and check if the IP address belongs to the IP address space. If it does return 1 otherwise return 0

is_match_macaddress(value_to_check1, value_to_check2)¶

Passed a two prospective MAC addresses and check to see if they are the same address. Return 1 for both the same MAC address and 0 for different

is_valid_ethertype(value_to_check)¶

Passed a prospective EtherType and check that it is valid. Can be hex (0x*) or decimal Return 1 for is valid IP address and 0 for not valid

is_valid_ip_space(value_to_check)¶

Passed a prospective IP address and check that it is valid. Can be IPv4 or IPv6 and can be range or have CIDR mask Return 1 for is valid IP address and 0 for not valid

is_valid_macaddress(value_to_check)¶

Passed a prospective MAC address and check that it is valid. Return 1 for is valid IP address and 0 for not valid

is_valid_transport_port(value_to_check)¶

Passed a prospective TCP or UDP port number and check that it is an integer in the correct range. Return 1 for is valid port number and 0 for not valid port number

tc_identity module¶

This module is part of the nmeta suite running on top of Ryu SDN controller to provide network identity and flow (traffic classification) metadata

class tc_identity.IdentityInspect(config)¶

Bases: baseclass.BaseClass

This class is instantiated by tc_policy.py (class: TrafficClassificationPolicy) and provides methods to ingest identity updates and query identities

check_dns(dns_name, pkt, ident, is_regex=False)¶

Passed a DNS name, flows packet object, an instance of the identities class and a regex boolean (if true, DNS name is treated as regex). Return True or False based on whether or not the packet has a source or destination IP address that has been resolved from the DNS name. Uses methods of the Identities class to work this out. Returns boolean

check_identity(condition, pkt, ident)¶

Checks if a given packet matches a given identity match rule. Passed condition, flows packet and identities objects and update the condition match based on whether or not either of the packet IP addresses matches the identity attribute/value. Uses methods of the Identities class to work this out

check_lldp(host_name, pkt, ident, is_regex=False)¶

Passed a hostname, flows packet object, an instance of the identities class and a regex boolean (if true, hostname is treated as regex). Return True or False based on whether or not the packet has a source or destination IP address that matches the IP address registered to the given hostname (if one even exists). Uses methods of the Identities class to work this out. Returns boolean

tc_custom module¶

This module is part of the nmeta suite running on top of Ryu SDN controller to provide network identity and flow (traffic classification) metadata

class tc_custom.CustomInspect(config)¶

Bases: baseclass.BaseClass

This class is instantiated by tc_policy.py (class: TrafficClassificationPolicy) and provides methods to run custom traffic classification modules

check_custom(condition, flow, ident)¶

Passed condition, flows and identities objects. Call the named custom classifier with these values so that it can update the condition match as appropriate.

instantiate_classifiers(custom_list)¶

Dynamically import and instantiate classes for any custom classifiers specified in the controller nmeta2 main_policy.yaml

Passed a deduplicated list of custom classifier names (without .py) to load.

Classifier modules live in the ‘classifiers’ subdirectory

api module¶

This module is part of the nmeta suite running on top of Ryu SDN controller to provide network identity and flow metadata. It provides methods for RESTful API connectivity.

class api.Api(_nmeta, _config, _wsgi)¶

Bases: object

This class is instantiated by nmeta.py and provides methods for RESTful API connectivity.

IP_PATTERN = '\\b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\\.|$){4}\\b'¶

url_data_size_rows = '/nmeta/measurement/tablesize/rows/'¶

url_flowtable = '/nmeta/flowtable/'¶

url_flowtable_augmented = '/nmeta/flowtable/augmented/'¶

url_flowtable_by_ip = '/nmeta/flowtable/{ip}'¶

url_identity_ip = '/nmeta/identity/ip/'¶

url_identity_mac = '/nmeta/identity/mac/'¶

url_identity_nic_table = '/nmeta/identity/nictable/'¶

url_identity_service = '/nmeta/identity/service/'¶

url_identity_system_table = '/nmeta/identity/systemtable/'¶

url_measure_event_rates = '/nmeta/measurement/eventrates/'¶

url_measure_pkt_time = '/nmeta/measurement/metrics/packet_time/'¶

exception api.NotFoundError(msg=None, **kwargs)¶

Bases: ryu.exception.RyuException

message = 'Error occurred talking to function <TBD>'¶

class api.RESTAPIController(req, link, data, **config)¶

Bases: ryu.app.wsgi.ControllerBase

This class is used to control REST API access to the nmeta data and control functions

get_data_structure_size_rows(*args, **kwargs)¶

Run a REST command and return appropriate response

get_event_rates(*args, **kwargs)¶

Run a REST command and return appropriate response

get_id_ip(*args, **kwargs)¶

Run a REST command and return appropriate response

get_id_mac(*args, **kwargs)¶

Run a REST command and return appropriate response

get_id_service(*args, **kwargs)¶

Run a REST command and return appropriate response

get_packet_time(*args, **kwargs)¶

Run a REST command and return appropriate response

list_flow_table(*args, **kwargs)¶

Run a REST command and return appropriate response

list_flow_table_augmented(*args, **kwargs)¶

Run a REST command and return appropriate response

list_flow_table_by_IP(*args, **kwargs)¶

Run a REST command and return appropriate response

list_identity_nic_table(*args, **kwargs)¶

Run a REST command and return appropriate response

list_identity_system_table(*args, **kwargs)¶

Run a REST command and return appropriate response

api.rest_command(func)¶

REST API command template

api_external module¶

The api_external module is part of the nmeta suite, but is run separately

This module runs a class and methods for an API that exposes an interface into nmeta MongoDB collections.

It leverages the Eve Python REST API Framework

class api_external.ExternalAPI(config)¶

Bases: baseclass.BaseClass

This class provides methods for the External API

ingest_dictionary(filename)¶

Read text file that is in dictionary format into a Python dictionary object. Uses ast module.

m_pi_rate_response(items)¶

Update the response with the packet_in rate. Hooked from on_fetched_resource_<name>

on_inserted_callback(resource_name, items)¶

Runs on Decision API database inserts, after database insertion completed. It places a message onto the multi-process queue that contains link to resource in database

post_get_callback(resource, request, payload)¶

TBD

pre_get_callback(resource, request, lookup)¶

Runs on GET request pre database lookup

run()¶

Run the External API instance

config module¶

This module is part of the nmeta suite running on top of the Ryu SDN controller to provide network identity and flow (traffic classification) metadata. It expects a file called “config.yaml” to be in the same directory containing properly formed YAML

class config.Config(config_dir='config', config_filename='config.yaml')¶

Bases: object

This class is instantiated by nmeta.py and provides methods to ingest the configuration file and provides access to the keys/values that it contains. Config file is in YAML in config subdirectory and is called ‘config.yaml’

get_value(config_key)¶

Passed a key and see if it exists in the config YAML. If it does then return the value, if not return 0

flows module¶

The flows module is part of the nmeta suite

It provides an abstraction for conversations (flows), using a MongoDB database for storage and data retention maintenance.

Flows are identified via an indexed bi-directionally-unique hash value, derived from IP-value-ordered 5-tuple (source and destination IP addresses, IP protocol and transport source and destination port numbers).

Ingesting a packet puts the flows object into the context of the packet that flow belongs to, and updates the database object for that flow with information from the current packet.

There are various methods (see class docstring) that provide views into the state of the flow.

class flows.Flow(config)¶

Bases: baseclass.BaseClass

An object that represents a flow that we are classifying

Intended to provide an abstraction of a flow that classifiers can use to make determinations without having to understand implementations such as database lookups etc.

Be aware that this module is not very mature yet. It does not cover some basic corner cases such as packet retransmissions and out of order or missing packets.

Read a packet_in event into flows (assumes class instantiated as an object called ‘flow’):

flow.ingest_packet(dpid, in_port, pkt, timestamp)

Variables available for Classifiers (assumes class instantiated as an object called ‘flow’):

Variables for the current packet:

flow.packet.flow_hash

The hash of the 5-tuple of the current packet

flow.packet.packet_hash

The hash of the current packet used for deduplication. It is an indexed uni-directionally packet identifier, derived from ip_src, ip_dst, proto, tp_src, tp_dst, tp_seq_src, tp_seq_dst

flow.packet.dpid

The DPID that the current packet was received from via a Packet-In message

flow.packet.in_port

The switch port that the current packet was received on before being sent to the controller

flow.packet.timestamp

The time in datetime format that the current packet was received at the controller

flow.packet.length

Length in bytes of the current packet on wire

flow.packet.eth_src

Ethernet source MAC address of current packet

flow.packet.eth_dst

Ethernet destination MAC address of current packet

flow.packet.eth_type

Ethertype of current packet in decimal

flow.packet.ip_src

IP source address of current packet

flow.packet.ip_dst

IP destination address of current packet

flow.packet.proto

IP protocol number of current packet

flow.packet.tp_src

Source transport-layer port number of current packet

flow.packet.tp_dst

Destination transport-layer port number of current packet

flow.packet.tp_flags

Transport-layer flags of the current packet

flow.packet.tp_seq_src

Source transport-layer sequence number (where existing) of current packet

flow.packet.tp_seq_dst

Destination transport-layer sequence number (where existing) of current packet

flow.packet.payload

Payload data of current packet

flow.packet.tcp_fin()

True if TCP FIN flag is set in the current packet

flow.packet.tcp_syn()

True if TCP SYN flag is set in the current packet

flow.packet.tcp_rst()

True if TCP RST flag is set in the current packet

flow.packet.tcp_psh()

True if TCP PSH flag is set in the current packet

flow.packet.tcp_ack()

True if TCP ACK flag is set in the current packet

flow.packet.tcp_urg()

True if TCP URG flag is set in the current packet

flow.packet.tcp_ece()

True if TCP ECE flag is set in the current packet

flow.packet.tcp_cwr()

True if TCP CWR flag is set in the current packet

Variables for the whole flow:

flow.packet_count()

Unique packets registered for the flow

flow.client()

The IP that is the originator of the flow (if known, otherwise 0)

flow.server()

The IP that is the destination of the flow (if known, otherwise 0)

flow.packet_direction()

c2s (client to server) or s2c directionality based on first observed packet direction in the flow. Source of first packet in flow is assumed to be the client

flow.max_packet_size()

Size of largest packet in the flow

flow.max_interpacket_interval()

TBD

flow.min_interpacket_interval()

TBD

Variables for the whole flow relating to classification:

classification.TBD

Challenges (not handled - yet):

• duplicate packets due to retransmissions or multiple switches in path
• IP fragments
• Flow reuse - TCP source port reused

class Classification(flow_hash, clsfn, time_limit)¶

Bases: object

An object that represents an individual traffic classification

commit()¶

Record current state of flow classification into MongoDB classifications collection.

dbdict()¶

Return a dictionary object of traffic classification parameters for storing in the database

class Flow.Packet¶

Bases: object

An object that represents the current packet

dbdict()¶

Return a dictionary object of metadata parameters of current packet (excludes payload), for storing in database

tcp_ack()¶

Does the current packet have the TCP ACK flag set?

tcp_cwr()¶

Does the current packet have the TCP CWR flag set?

tcp_ece()¶

Does the current packet have the TCP ECE flag set?

tcp_fin()¶

Does the current packet have the TCP FIN flag set?

tcp_psh()¶

Does the current packet have the TCP PSH flag set?

tcp_rst()¶

Does the current packet have the TCP RST flag set?

tcp_syn()¶

Does the current packet have the TCP SYN flag set?

tcp_urg()¶

Does the current packet have the TCP URG flag set?

Flow.client()¶

The IP that is the originator of the flow (if known, otherwise 0)

Finds first packet seen for the flow_hash within the time limit and returns the source IP

Flow.ingest_packet(dpid, in_port, pkt, timestamp)¶

Ingest a packet into the packet_ins collection and put the flow object into the context of the packet. Note that timestamp MUST be in datetime format

Flow.max_interpacket_interval()¶

Return the size of the largest inter-packet time interval in the flow (assessed per direction in flow) as seconds (type float)

Note: c2s = client to server direction s2c = server to client direction

Note: results are slightly inaccurate due to floating point rounding.

Flow.max_packet_size()¶

Return the size of the largest packet in the flow (in either direction)

Flow.min_interpacket_interval()¶

Return the size of the smallest inter-packet time interval in the flow (assessed per direction in flow) as seconds (type float)

Note: c2s = client to server direction s2c = server to client direction

Note: results are slightly inaccurate due to floating point rounding.

Flow.packet_count()¶

Return the number of packets in the flow (counting packets in both directions). This method should deduplicate for where the same packet is received from multiple switches, but is TBD...

Works by retrieving packets from packet_ins database with current packet flow_hash and within flow reuse time limit.

Flow.packet_direction()¶

Return the direction of the current packet in the flow where c2s is client to server and s2c is server to client.

Flow.server()¶

The IP that is the destination of the flow (if known, otherwise 0)

Finds first packet seen for the hash within the time limit and returns the destination IP

Flow.suppress_flow()¶

Set the suppressed attribute in the flow database object to the current packet count so that future suppressions of the same flow can be backed off to prevent overwhelming the controller

identities module¶

The identities module is part of the nmeta suite

It provides an abstraction for participants (identities), using a MongoDB database for storage and data retention maintenance.

Identities are identified via TBD....

There are methods (see class docstring) that provide harvesting of identity metadata and various retrieval searches

class identities.Identities(config)¶

Bases: baseclass.BaseClass

An object that represents identity metadata

Variables available for Classifiers (assumes class instantiated as an object called ‘ident’):

ident.TBD

TBD

ident.harvest(pkt, flow.packet)

TBD

ident.findbymac(mac_address)

Challenges (not handled - yet):

• TBD

class Identity¶

Bases: object

An object that represents an individual Identity Indicator

dbdict()¶

Return a dictionary object of identity metadata parameters for storing in the database

Identities.findbymac(mac_addr)¶

TEST FIND BY MAC ADDR DOC TBD

Identities.findbynode(host_name, harvest_type='any', regex=False)¶

Find by node name Pass it the name of the node to search for. Additionally, can set:

regex=True Treat service_name as a regular expression harvest_type= Specify what type of harvest (i.e. DHCP)

Returns boolean

Identities.findbyservice(service_name, harvest_type='any', regex=False, ip_address='any')¶

Find by service name Pass it the name of the service to search for. Additionally, can set:

regex=True Treat service_name as a regular expression harvest_type= Specify what type of harvest (i.e. DNS_A) ip_address= Look for specific IP address

Returns boolean

Identities.harvest(pkt, flow_pkt)¶

Passed a raw packet and packet metadata from flow object. Check a packet_in event and harvest any relevant identity indicators to metadata

Identities.harvest_arp(pkt, flow_pkt)¶

Harvest ARP identity metadata into database. Passed packet-in metadata from flow object. Check ARP reply and harvest identity indicators to metadata

Identities.harvest_dhcp(flow_pkt)¶

Harvest DHCP identity metadata into database. Passed packet-in metadata from flow object. Check LLDP TLV fields and harvest any relevant identity indicators to metadata

Identities.harvest_dns(flow_pkt)¶

Harvest DNS identity metadata into database. Passed packet-in metadata from flow object. Check DNS answer(s) and harvest any relevant identity indicators to metadata

Identities.harvest_lldp(flow_pkt)¶

Harvest LLDP identity metadata into database. Passed packet-in metadata from flow object. Check LLDP TLV fields and harvest any relevant identity indicators to metadata

identities.mac_addr(address)¶

Convert a MAC address to a readable/printable string

forwarding module¶

This module is part of the nmeta suite running on top of Ryu SDN controller to provide network identity and flow metadata. It provides methods for forwarding functions.

class forwarding.Forwarding(_config)¶

Bases: object

This class is instantiated by nmeta.py and provides methods for making forwarding decisions and transformations to packets.

basic_switch(event, in_port)¶

Passed a packet in event and return an output port

switch_abstraction module¶

This module is part of the nmeta suite running on top of Ryu SDN controller. It provides functions that abstract the details of OpenFlow calls, including differences between OpenFlow versions where practical

class switch_abstraction.SwitchAbstract(config)¶

Bases: baseclass.BaseClass

This class is instantiated by various other modules and provides methods for interacting with switches that are safe to use without need to for the calling program to know calls specific to the version of OpenFlow that the switch runs (where practical...)

add_flow(datapath, match, actions, **kwargs)¶

Add a flow table entry to a switch. Returns 1 for success or 0 for any type of error

Required kwargs are:

priority (0) buffer_id (None) idle_timeout (5) hard_timeout (0)

add_flow_eth(datapath, msg, **kwargs)¶

Add an ethernet (non-IP) flow table entry to a switch. Returns 1 for success or 0 for any type of error Uses Ethertype in match to prevent matching against IPv4 or IPv6 flows

add_flow_ip(datapath, msg, **kwargs)¶

Add an IP (v4 or v6) flow table entry to a switch. Returns 1 for success or 0 for any type of error Uses IP protocol number to prevent matching on TCP flows

add_flow_tcp(datapath, msg, **kwargs)¶

Add a TCP flow table entry to a switch. Returns 1 for success or 0 for any type of error

get_actions(datapath, ofv, out_port, out_queue)¶

Passed a datapath, an OpenFlow version an out port, an out queue and flood port # and build and return an appropriate set of actions for this

get_flow_match(datapath, ofproto, **kwargs)¶

Passed a OF protocol version and a Flow Match keyword arguments dict and return an OF match tailored for the OF version otherwise 0 (false) if compatibility not possible. TBD: validating values...

get_friendly_of_version(ofproto)¶

Passed an OF Protocol object and return a human-friendly version of the protocol revision number

get_in_port(msg, datapath, ofproto)¶

Passed a msg, datapath and OF protocol version and return the port that the packet came in on (version specific)

packet_out(datapath, msg, in_port, out_port, out_queue, nq=0)¶

Sends a supplied packet out switch port(s) in specific queue. Set nq=1 if want no queueing specified (i.e. for a flooded packet)

request_switch_desc(datapath)¶

Request that a switch send us it’s description data

set_switch_config(datapath, config_flags, miss_send_len)¶

Set config on a switch including config flags that instruct fragment handling behaviour and miss_send_len which controls the number of bytes sent to the controller when the output port is specified as the controller

set_switch_table_miss(datapath, miss_send_len, hw_desc, sw_desc)¶

Set a table miss rule on table 0 to send packets to the controller. This is required for OF versions higher than v1.0. Do not set on older OpenvSwitch as it causes packets to be sent to controller with no buffer and OpenvSwitch doesn’t need this rule as it punts to the controller regardless (contrary to specification?) Note: OVS 2.5.0 doesn’t do this, but not sure what point fix was applied to OVS, somewhere between 2.0.2 and 2.5.0