ipfix
module¶IP Flow Information Export (IPFIX) (see RFC7011) is the IETF standard for export of network traffic flow data, and is comprised of:
The Python IPFIX module (pip install ipfix
) provides access to the data format and information model for bridging between IPFIX Messages and Python objects, and is useful for building implementations of the protocol, as well as for manipulating data stored in IPFIX files (see RFC5655). Documentation for the module is available at github. The package also contains an undocumented module for visualizing IPFIX Messages as SVG graphics representing bitfields. We'll use this functionality in this notebook to explore the structure of IPFIX Messages.
First, execute the following code to import and define a few functions we'll need:
import ipfix
import ipfix.vis
from ipaddress import ip_address
from datetime import datetime
from datetime import timezone
from IPython.display import SVG
def iso8601(x):
return datetime.strptime(x, "%Y-%m-%d %H:%M:%S.%f")
def draw_message(msg, length=256):
return SVG(ipfix.vis.MessageBufferRenderer(msg, raster=4).render(length=length))
def draw_template(tmpl):
ofd = ipfix.vis.OctetFieldDrawing(raster=4)
ipfix.vis.draw_template(ofd, tmpl)
return SVG(ofd.render((90,30)))
IPFIX identifies data record structures using Information Elements (IEs). The basic IEs are defined in the IANA IPFIX Information Element Registry, and a mechanism for applying these IEs to export bidirectional flow data is defined in RFC 5103. In order to work with these IEs, we'll need to tell the ipfix
module to use them:
ipfix.ie.use_iana_default()
ipfix.ie.use_5103_default()
Okay, now we're ready to begin.
A Template is an ordered sequence of IEs that describes the structure of a type of Data Record. If you're familiar with relational databases, you can think of a Template as defining a table, and an IE as a defined column name with a standard meaning, such that records containing the same IEs describe the same type of data.
Let's consider a template defining a simple IPv4 flow record, with start and end timestamps, source and destination addresses and ports, protocol identifier, octet and packet delta counts:
tmpl = ipfix.template.for_specs(261, "flowStartMilliseconds",
"flowEndMilliseconds",
"sourceIPv4Address",
"destinationIPv4Address",
"sourceTransportPort",
"destinationTransportPort",
"protocolIdentifier",
"octetDeltaCount",
"packetDeltaCount")
draw_template(tmpl)
Here we see that the use of an IE registry, which maps these numbers to names, allows IPFIX to represent this type information very efficiently: the template consists of a 16-bit template ID, a 16-bit field count, and 16 bits for the ID and 16 bits for the length for each IE.
To illustrate how this template is used to encode a record, let's add this template to a message, along with a record encoded using the template:
msg = ipfix.message.MessageBuffer()
msg.begin_export(8304) # Observation Domain ID
msg.add_template(tmpl)
msg.export_new_set(261) # The set ID here refers to the Template ID of the template
msg.export_namedict({ 'flowStartMilliseconds': iso8601('2012-10-22 09:29:07.170000'),
'flowEndMilliseconds': iso8601('2012-10-22 09:29:33.916000'),
'sourceIPv4Address': ip_address('192.0.2.11'),
'destinationIPv4Address': ip_address('192.0.2.212'),
'sourceTransportPort': 32798,
'destinationTransportPort': 80,
'protocolIdentifier': 6,
'packetDeltaCount': 17,
'octetDeltaCount': 3329})
draw_message(msg)
First we see a message header, containing the version (10) and length of the message in bytes, followed by a sequence number (which here advertises that 0 records have been sent for this observation domain ID in this file or session), an export time, and an observation domain ID. Observation domains separate parts of a metering infrastructure into logical domains within which a given packet is reported as observed at most once, and map to sets of components coordinated in measurement of passing flows (e.g., a line card).
Notice that the set ID matches the template ID, and the Information Elements in the Template appear in the same order as the fields in the Data Records in the Message; this is how IPFIX handles self-description.
Templates are persistent within a session; subsequent messages can refer to templates sent in previous messages:
msg.begin_export(8304)
msg.export_new_set(261)
msg.export_namedict({ 'flowStartMilliseconds': iso8601('2012-10-22 09:30:01.912000'),
'flowEndMilliseconds': iso8601('2012-10-22 09:31:15.009000'),
'sourceIPv4Address': ip_address('192.0.2.212'),
'destinationIPv4Address': ip_address('192.0.2.11'),
'sourceTransportPort': 80,
'destinationTransportPort': 32801,
'protocolIdentifier': 6,
'packetDeltaCount': 83,
'octetDeltaCount': 97501})
msg.export_namedict({ 'flowStartMilliseconds': iso8601('2012-10-22 09:30:08.182000'),
'flowEndMilliseconds': iso8601('2012-10-22 09:31:16.012000'),
'sourceIPv4Address': ip_address('192.0.2.212'),
'destinationIPv4Address': ip_address('192.0.2.11'),
'sourceTransportPort': 80,
'destinationTransportPort': 32802,
'protocolIdentifier': 6,
'packetDeltaCount': 99,
'octetDeltaCount': 136172})
draw_message(msg)
To see how IPFIX encodes reduced-length and variable-length IEs, let's define a new template including the 802.11 SSID, encoded as a UTF-8 string. Here, note that the octetDeltaCount and packetDeltaCount IEs are exported using 4 bytes instead of the native 8, as well, to illustrate reduced-length encoding.
vtmpl = ipfix.template.for_specs(262, "flowStartMilliseconds",
"flowEndMilliseconds",
"sourceIPv6Address",
"destinationIPv6Address",
"octetDeltaCount[4]",
"packetDeltaCount[4]",
"wlanSSID")
msg.begin_export(8304)
msg.add_template(vtmpl)
msg.export_new_set(262)
msg.export_namedict({'flowStartMilliseconds': iso8601('2012-10-22 09:31:54.903000'),
'flowEndMilliseconds': iso8601('2012-10-22 09:41:52.627000'),
'sourceIPv6Address': ip_address('2001:db8:c0:ffee::2'),
'destinationIPv6Address': ip_address('2001:bd8:b:ea75::3'),
'packetDeltaCount': 212,
'octetDeltaCount': 553290,
'wlanSSID': 'ietf-a-v6only'})
draw_message(msg)
Here we see the octetDeltaCount and PacketDeltaCount fields taking up only 4 as opposed to 8 bytes. In addition, note that the wlanSSID field is prefixed with a varlen
byte, counting the number of subsequent bytes containing the value. IPFIX uses this length-prefixing since no delimiter can be guaranteed never to appear in a variable-length binary value.
So far, we've explored the use of IPFIX for flow export; however, the protocol is useful for any application to which the following applies:
To take an example, let's use IPFIX to solve a common problem during meetings, conferences, and classes: meeting room temperature. Let's say we have a table of observation point identifiers to meeting room names, and use IPFIX to periodically export ambient temperature information for a given machine. In an ideal world, we'd write a proper description for the Information Element and send it to the Internet Assigned Numbers Authority (IANA), the body that maintains the Information Element Registry, as follows:
Name: ambientTemperatureCelsius
Description: The ambient temperature measured in the environment.
May be associated with an Observation Point or a Metering Process;
otherwise, taken to be the ambient temperature in the environment
of the Exporting Process.
Abstract Data Type: float32
Units: degrees Celsius
Range: -273.15 - +infinity
But this is just a two hour tutorial, so we'll define a new enterprise-specific IE instead:
ipfix.ie.for_spec("ambientTemperatureCelsius(35566/2)<float32>[4]")
InformationElement('ambientTemperatureCelsius', 35566, 2, ipfix.types.for_name('float32'), 4)
And now, as above, a template, a record, and a message:
ttmpl = ipfix.template.for_specs(263, "observationTimeMilliseconds",
"observationPointId[4]",
"ambientTemperatureCelsius")
msg = ipfix.message.MessageBuffer()
msg.begin_export(8304)
msg.add_template(ttmpl)
msg.export_new_set(263)
msg.export_namedict({'observationTimeMilliseconds': datetime.utcnow(),
'observationPointId': 1,
'ambientTemperatureCelsius': 22.3})
draw_message(msg)
Of course, physical measurements are more interesting if they're real. We've attached a cheap temperature-and-humidity sensor to a Raspberry Pi connected to the display laptop to demonstrate two things:
Here, we took an off-the-shelf user-space driver for the sensor attached to the GPIO pins, and added some C code to encode the result as IPFIX, attaching a static template and message header. This program doesn't even handle network communication; for that, we pipe its binary IPFIX output to nc, which acts as a TCP exporting process given an IPFIX message stream.
This sensor handles relative humidity as well as temperature, so we'll need an IE for that, too:
ipfix.ie.for_spec("relativeHumidityPercent(35566/3)<float32>[4]")
And now we'll need a way to collect from an external exporting process. Recall that the ipfix
module only handles the data format and the information model, not the full protocol stack. Fortunately, the protocol's dynamics are relatively simple: IPFIX over TCP, for instance, contains the a stream of data more or less identical to what would be found in an IPFIX file. Therefore, we can use Python's socketserver
module to build a quick and dirty IPFIX Collecting Process that draws SVG representations of the received messages and stores them for later display.
import socketserver
import ipfix.reader
import threading
msg_length = 512 # Maximum number of bytes per message to render
svgbuf = []
svgbuf_mtx = threading.Lock()
class StreamRendererHandler(socketserver.StreamRequestHandler):
def handle(self):
global svgbuf
print("connection from "+str(self.client_address)+".")
msr = ipfix.vis.MessageStreamRenderer(self.rfile, scale=(90,30), raster=4)
while True:
try:
svgbuf_mtx.acquire()
svgbuf.append(msr.render_next_message(msg_length))
svgbuf_mtx.release()
except:
break
print("connection from "+str(self.client_address)+" terminated.")
class ThreadingTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
pass
srv = None # shut down old server, if any, through loss of reference
srv = ThreadingTCPServer(("", 4739), StreamRendererHandler)
srvt = threading.Thread(target=srv.serve_forever)
srvt.daemon = True
srvt.start()
If you're not attending the course, and don't have a Raspberry Pi with the appropriate sensor attached, run the following code to simulate the connected device:
%pushd ../raspi
!./run_sim.sh localhost 4739
%popd
Once we're done receiving messages, we can shut the server down:
srv.shutdown()
And now we can draw the messages buffered by the server:
SVG(svgbuf[0])
SVG(svgbuf[1])
SVG(svgbuf[2])
This notebook is © 2013-2014 Brian Trammell, and is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.