There are certain scenarios where the High Performance Filtering feature added in Netmon 3.4 will provide the best performance for capturing with a filter. The idea is to filter frames before they hit the disk which can improve your performance by reducing the impact on the capturing machine. High Performance filtering can be performed with both the UI and NMCap. However, if performance is a concern, NMCap provides the best capturing performance overall. The following provides some guidance on when and how to use this feature. You should also consider reading our Avoiding Dropped Frames blog as background.
This article is split into 3 general areas. The first few sections describe what High Performance mode is. The next sections following that describe when to use the feature. And finally the last few sections describe how to use the feature in more detail.
First let’s review the various capture filtering methods:
Normal Buffered Capturing (Default Mode)
Frames are buffered to disk first and then evaluated afterwards. If your machine is unable to keep up with incoming traffic, your pending frame count will continue to grow. But as we buffer frames first, it’s much better at handling bursts of network traffic.
Any capture filter with at least one NON fully qualified data field or property.
High Performance Profile
This is an optimized parser set which provides better performance, but with less depth in terms of fields you can parse on. This profile is automatically used with High Performance Capture filters, but can also be used with normal buffered capturing.
You can enable this parser set using the /UseProfile with NMCap or selecting the profile in the UI. You should also turn off conversations for best performance. With NMCap this means adding the /DisableConversation switch.*
High Performance Capturing
Frames are evaluated before buffering as long as we detect we can keep up. This saves a write to the disk and uses a high performance filter to evaluate the frames.
Any capture filter where ALL fields are fully qualified and supported. See High Performance Capturing in the help file for a full list. Also reference the NM34 High Perf Filters in the standard filter list under the Load Filter drop down button.
*Note: That in general you can increase the performance of NMCap by disabling conversations and not enabling process tracking. Conversations are enabled by default when any filter is used, even in StopWhen/StartWhen filters. If you can verify conversations are not really needed then explicitly add the /DisableConversation switch. Conversations are not needed for TCP, IP or Ethernet data fields. Process tracking is disabled by default.
There are 3 external factors which contribute to the ability for your machine to keep up and filter incoming network traffic.
High Performance filtering is composed of two separate optimizations.
The range of system capturing performance, network throughput and network burst creates a huge matrix of possibilities. Rather than trying to provide guidance in terms of the types of machines and networks in which this feature will benefit, we instead provide a list of conditions to explain when and when not to use the various components of the feature.
For the most part, this feature is targeted at cases where you have a busy server with high traffic and you want to capture a narrow slice of the total network traffic. In cases where we can avoid buffering to disk, we lower the disk load. And in the case where we use the High Performance Profile we lower the load on the CPU.
If you are capturing a majority of the traffic you get the best performance using NMCap with no filter. That includes StopWhen or StartWhen filters as these will cause the parsing engine to engage. By writing directly to disk we don’t have to evaluate frames and therefore we don’t have to read and write the capture after a filter evaluation.
When using High Performance Filtering you should meet all of the following conditions. The main difference in this scenario when compared to the next is that we cannot handle network bursts as reliably. When filtering before we buffer to disk, all buffering is done in memory and this is a limited resource when compared to your disk space. For more information on configuring this buffer please see the Avoiding Dropped Frames blog.
You can also separately specify the High Performance Profile when running NMCap with the /UseProfile switch. Using this high performance filter set alone, you are able to get better capture performance than the default configuration.
When you are using the UI and want to view traffic live, you can use a fully qualified filter to restrict the amount of traffic you are looking at. But you might still want to use the full parsing to view the data. For this scenario here are the conditions which would warrant the High Performance feature.
As mentioned in the Avoiding Dropped Frames blog, the filter complexity affects the CPU load. And one possible solution is to use “weaker” filters. The type of filter you use affects your filtering speed. The more complex and deep a filter is, the longer it takes to parse a frame. For instance, parsing all the way to TCP.Port takes a lot longer parse than Ethernet.Address. But first let’s determine how to tell if you need to optimize your filter.
The first step is determining if your machine can keep up based on a specific filter. The only way to know for sure is to run NMCap using a high performance filter during the type of network traffic you expect to see. Then view the statistics that are printed out and view the Dropped and Pending frame counters.
For either of these cases, it might be possible to optimize the buffers and flow control to avoid dropping frames or make sure you stay in high performance mode rather than reverting back to buffering. See the “Modifying the High Performance Buffer Options” below for more details.
In some cases you can provide a simpler filter to achieve what you need. For instance, if you are filtering on an IPv4 or IPv6 address, it might be possible to use an Ethernet address instead. Ethernet is simpler to parse and therefore faster.
Also keep in mind that it does take longer to evaluate multiple fields than just one. While you might not have an option to simplify in this manner, this might be a place you can consider simplifying. For instance, rather than looking for both the source and destination addresses you might be able to get away with looking for only the destination. Since the source is often your machine, and traffic to the capturing machine assumes your address, it would be better to use Frame.Ethernet.DestinationAddress rather than Frame.Ethernet.Address.
If there is a filter that is not supported using High Performance mode, or you need even higher performance, you can use a pattern offset filter. The caveat here is that you have to validate if your pattern and offset is accurate for the scenario you are capturing. For instance, if you are filtering on a TCP port with the ESP protocol in the stack, there might be situations where IP options move the location and block traffic that you'd normally capture with a normal filter. You must evaluate the possibility that you might drop frames you'd normally capture and understand the impact of missing these frames. If ANY of these following conditions are true, then you might consider this option.
To create a filter using the Blob, you need to know the offset and length of the pattern you are matching. Often, the simplest way to do this is open a trace you’ve taken from the network you are interested in, and click on the field in question. Then look in the hex details for that location and offset.
For instance, if you wanted the IPv4 Destination Address you would see that on an Ethernet network that this is at offset 30 and the size is 4 octets. So the filter would be Blob(FrameData, 30, 4) == X. For X you would have to determine the 4 octet value that represents your IP address. Again if you already have a trace, looking at the hex details on an example filter is one easy way to find out what that value is. But if need be, you’ll have to translate the value to hex. If in this example the address was 220.127.116.11, the associated hex value is 0x01020304. So your blob filter would be:
Blob(FrameData, 30, 4) == 0x01020304
Blob(FrameData, 30, 4) == 0x01020304
With most high performance filters you must preface them with a fully qualified path. But Blob is the one exception. If it is used as a capture filter in the UI or NMCap, and the rest of the filter string qualifies, then the filter will be attempted using the high performance mode.
As networks continue to get faster, capturing that traffic becomes more and more challenging. With this new feature you have the ability to filter before you buffer to disk thus giving you the flexibility to control the load on your system. This can give you the ability to capture faster traffic with a lower tax on the capturing system.