As you know if you've been following this blog, at Igalia we build network functions using the Snabb toolkit. When we're not directly working on customer projects, we often invest time into building new features into Snabb.
One of these recent investments has been building a basic IP flow export (IPFIX) app for Snabb. The app is now available in the v2017.08 Snabb release. The app's documentation is up on the web here.
As you can see from the commit log, Andy Wingo helped out a great deal in building the app (and was responsible for much of the performance engineering).
What is IP flow export?
IPFIX refers to a widely used set of tools that let you monitor IP flows in your network. An IP flow is a set of IP packets that share some common characteristics. Often these are described by a flow key composed of a standard 5-tuple of source & destination address, protocol, and TCP/UDP/etc ports.
Monitoring flows can be useful for traffic measurement and management, usage-based billing, and other use cases (many of which are spelled out in RFC 3917).
You may have also heard of "Netflow", which is the trade name used by Cisco for these tools. IPFIX is the standardized version (see RFC 5470 and related RFCs) that is backwards compatible with Netflow.
(A great long-form overview of the topic is "Flow Monitoring Explained" by Hofstede et al, which was very helpful in designing the app)
The basic architecture of IPFIX consists of a flow metering process which monitors the traffic, an exporter that communicates the data collected by the meter, and finally a collector that aggregates the data. In practice, the metering process and exporter are often combined into one program (so I'll just refer to it as an exporter or as a probe).
Here's some ASCII art from the RFC showing the structure:
+----------------+ +----------------+
|[*Application 1]| ... |[*Application n]|
+--------+-------+ +-------+--------+
^ ^
| |
+ = = = = -+- = = = = +
^
|
+------------------------+ +-------+------------------+
|IPFIX Exporter | | Collector(1) |
|[Exporting Process(es)] |<---------->| [Collecting Process(es)] |
+------------------------+ +--------------------------+
.... ....
+------------------------+ +---------------------------+
|IPFIX Device(i) | | Collector(j) |
|[Observation Point(s)] |<--------->| [Collecting Process(es)] |
|[Metering Process(es)] | +---->| [*Application(s)] |
|[Exporting Process(es)] | | +---------------------------+
+------------------------+ .
.... . ....
+------------------------+ | +--------------------------+
|IPFIX Device(m) | | | Collector(n) |
|[Observation Point(s)] |<----+---->| [Collecting Process(es)] |
|[Metering Process(es)] | | [*Application(s)] |
|[Exporting Process(es)] | +--------------------------+
+------------------------+
(what the RFC calls a "Device" contains some number of "metering processes")
The exporter might track information such as the total number of packets, payload sizes, and start/end times for a unique flow (these are called information elements and are standardized by the IANA).
The exporter periodically sends its summarized data to a flow collector, which uses some kind of database to keep and track all of the flow information (unlike the exporter which may evict inactive flows). The collector can present this information to users in a variety of ways (e.g., a web UI).
An exporter communicates with a collector using the IPFIX protocol, so that you can use an exporter with any off-the-shelf collector.
Snabb app
For Snabb, we implemented an IPFIX exporter as an app that can be integrated
with other apps. For convenience, we provide a snabb ipfix probe commandline
program that runs an exporter with a simple configuration.
The flow keys and record fields to be collected are described using Lua tables, like the following:
v4 = make_template_info {
id = 256,
filter = "ip",
keys = { "sourceIPv4Address",
"destinationIPv4Address",
"protocolIdentifier",
"sourceTransportPort",
"destinationTransportPort" },
values = { "flowStartMilliseconds",
"flowEndMilliseconds",
"packetDeltaCount",
"octetDeltaCount"}
}
This table describes a flow record that's then used to dynamically generate the appropriate FFI data structures and configure the control flow of the app. The fields are described using the names from the IANA information element table.
Since the exporter collects some information from all of the IP packets going through it, it has to be performant and use minimal resources.
Making IPFIX perform well
Performance was the most difficult part of implementing the IPFIX app. The core of the app is fairly simple. For each packet, it looks at the fields that define a flow (currently we just support the usual 5-tuple) and uses this as a flow key to index a ctable.
Since ctables are implemented as a hashtable, this means that the hashing can be a key bottleneck for the app. This isn't really surprising since hashing is a key operation that is often optimized and also delegated to hardware in networking.
It turns out that the hashing algorithm used by ctable performed well with keys of certain sizes (4 or 8 bytes), but not with larger keys such as flow keys. We tried out several hash algorithms (e.g., FNV hashing) and got some incremental improvements. In the end, Andy was able to get better performance out of implementing SipHash using DynASM.
Andy also implemented a bunch of other performance improvements, such as separating out the IPv4 and IPv6 data paths and re-organizing the multiple original apps into a single app that contains several mini-queues and work loops. In the end, these improvements got the app to line-rate performance in our synthetic benchmark for most packet sizes (performance on 64-byte packet sizes needs more work).
In terms of real-world performance, we still need to do more work but are making progress thanks to our early adopters. Alexander Gall from SWITCH has been providing us with valuable feedback from running and hacking on the app with production data.
Future improvements
At this point, we have an IPFIX app that can be integrated into other Snabb solutions or used standalone on commodity Intel hardware. There's still a lot of work that can be done on it though. One idea for improvement on the performance side is to parallelize it using the RSS feature on 10G network cards. RSS (receive-side scaling) lets you use multiple processes running in parallel on several receive queues on a single NIC.
Conveniently, it turns out that at Igalia we've also been working on improving Snabb's network drivers to make it easier to use RSS. There's a good chance that we can parallelize IPFIX very easily, since there's minimal coordination that's needed between IPFIX instances.
There are a number of other improvements we could make too. Probably the most useful is to add support for more information elements, and to make the observed IEs more configurable.
Another area for improvement is the app configuration. Snabb has support for app configuration with YANG schemas and the IETF has a schema for IPFIX that we could use.
Another limitation is that the IPFIX app currently just exports over UDP to a collector. The RFCs technically requires support for SCTP as well (adding that could be a lot of work, maybe we would offload the work to an existing userspace library).
Final thoughts
Working on IPFIX was pretty fun overall. One of the rewarding aspects of working on the IPFIX app is that it's relatively easy to test it and see that it's doing something. For example, you can plug it up to Wireshark and manually check the IPFIX packets to see what's going on.
You can also plug it into an off-the-shelf flow collector like
nfdump and see some useful output. In fact,
we have a test written using nix-shell
that will spawn a shell in a new Nix enviroment with nfdump installed and test
the app with it. Nix can be very nice for this kind of test environment setup.
In any case, please feel free to try out the app. We would appreciate any feedback or bug reports!