I’ve recently played with a new tool here at MS, which analyzes HTTP traffic and provides performance information so that you can better tune your web servers and applications. I also have seen an internal SMB expert that summarizes SMB traffic, for instance open files and connects. These types of experts provide a protocol specific view of network data that is tailored to a specific protocol. The NMAPI provides a powerful way to access our parsing and capturing engine directly. This gives you a lot of flexibility to analyze network traffic in ways you can only imagine. I want to give a quick overview of how the API works so that you can harness the power of the NMAPI.
VRTA – Visual Round Trip Analyzer
Before I dig into the API, let me show you an example of what the API can accomplish for you. It is available here:
http://www.microsoft.com/downloads/details.aspx?familyid=119F3477-DCED-41E3-A0E7-D8B5CAE893A3&displaylang=en
This tool was recently released and allows you to visualize HTTP traffic so that you can diagnose performance of your HTTP server or Browser.
I won’t go into all the details, but the information displayed will let you understand performance based on the throughput as well as the number of parallel request made to understand how your browser or server is performing. It will also display the graphic images and traffic specifics as you hover over various sections of the graph. There are also overall statistics and a cool Analysis section which attempts to score your traffic based on some common rules of thumb.
What Does the API Allow?
I like to divide the API into 3 areas, though they tend to overlap some. They are Capturing, Parsing, and Capture File access. So let me start by explaining what each can do.
Capturing: You have most of the same capabilities as you do in the UI. You can Start/Stop/Pause the capture engine on any of the network interfaces. The API lets you enumerate the available adapters and setup a unique callback for each one, if you want. Each time a frame arrives, the callback is sent the raw frame which you can then evaluate or simply save to a capture file.
Parsing: With access to the parsing engine, you can inspect any data field that is parsed. You can reassemble data on the fly to quickly parse for a few fields or enumerate through them all. You also have the same UI filtering language available to you in the API. You can define a filter as part of your frame parser and then evaluate that filter, very quickly, on each frame.
Capture File Access: The API also allows you to read and write capture files using the Netmon 2.1 file format for NM3.2.
Where is the SDK?
Almost everything is included when you install NM3.2. This includes NetMonSDK.CHM which describes each of the API and contains examples as well. The only other requirement is that you need to install the WDK. This is because we reference the NDIS headers for each adapter, which is one of the functions the API provides.
For more information on setting up your environment, see the section called Network Monitor API Overview in the CHM file. This can be accessed from the help menu in NM3.2.
How Does Capturing Work with the API?
The API uses a callback to define where you want frames to go when they are received by the engine. So you simply have to enumerate the available interfaces, and define where you want that callback to go. You can use the same callback for all adapters, if you prefer.
So, in the example below, I iterate though all the adapters. In this case I use the same callback for each adapter. Once I’ve set these up, I can call StartCapture, and the callback will be called as each frame arrives.
NmOpenCaptureEngine(&myCaptureEngine);
ULONG AdpCount;
NmGetAdapterCount(myCaptureEngine, &AdpCount);
int NumAdpts = 0;
for(DWORD n=0; n< AdpCount;n ++){
ret = NmConfigAdapter(myCaptureEngine, n, myFrameIndication, NULL, NmReturnRemainFrames);
if(ret != ERROR_SUCCESS)
{
wprintf(L"Error configuration adapter %d, 0x%X\n", n, ret);
} else {
NumAdpts++;
}
}
In your callback, you can simply save all the frames by using NmAddFrame with a capture file handle you created using NmCreateCaptureFile.
void __stdcall
MyFrameIndication(HANDLE hCapEng, ULONG ulAdaptIdx, PVOID pContext, HANDLE hRawFrame)
{
HANDLE capFile = (HANDLE)pContext;
NmAddFrame(capFile, hRawFrame);
}
How Does Parsing Work with the API?
The main task at hand is getting a raw frame and breaking it apart so that you can reference the fields it contains. In the API this is the job of the FrameParser. This object, which is just a windows HANDLE, is applied to a raw frame. The result of this action, using NmParseFrame(…), is a ParsedFrame object. With this object you can access all the fields and apply filters.
So for instance, you would normally get a raw frame from your frame Callback or directly from a capture file. In this example, we’ll use a capture file.
NmGetFrame(myCaptureFile, 5, &myRawFrame);
NmParseFrame(myFrameParser, myRawFrame, 0, 0, &myParsedFrame, NULL);
The resulting myParsedFrame object can now be used in calls like NmGetFieldxxx(…) to retrieve the fields value, full name, offset, size, and other meta data. If you look in the documentation you can see all the NmGetField type API calls.
How to I Create a Frame Parser Object?
To create a FrameParser you start with a set of NPL files, which describes how frames are parsed. Using the NmLoadNplParser, you can load these NPLs to retrieve the NPLParser object. If there are errors while building the parser, you can get these reported to you via another callback.
With the NPLParser object, you can now create a FrameParserConfiguration object. This interim object is needed because this is how you configure the parser to add fields and filters. You can also configure how you want reassembly and conversations to work. When you add fields with NmAddField, you are limited to seeing only those fields when you parse a frame. This limited scope provides the best performance.
But if you instead want to enumerate all fields, simply don’t add any fields in this step. In this case you can get a count of fields for the given parsed frame, NmGetFieldCount. Then you just iterate through all the fields where the count is the field ID.
Once the Frame Parser Configuration is finished, you can then generate the Frame Parser object you use to parse a raw frame. There is also no reason you can’t generate multiple frame parsers as well as frame parser configurations, perhaps for different types of frames.
Managed Code
While we haven’t created an official Managed wrapper, we have included a simple NetmonAPI.CS file which uses PInvoke to call the DLL directly. We hope to release a more proper wrapper externally at some point. Perhaps in the shorter term we can also start an open project which wraps the API.
Gotcha’s using the API
In this section I wanted to explain some problems you might encounter when using the API. At one point or another I’ve run into these issues. I want to pass them on to you.
Parsing is State Full: When you are using a FrameParser object, the object keeps track of state information. This means that if you parse raw frames from more than one frame source, like a capture file and live adapter, the state information might get confused. You should create multiple Frame Parsing objects in that case, one for each source of raw frames.
Number of Object Handles Limited: While this may seem obvious, the number of engine handles is limited, though this can be configured. But if you don’t correctly clean up unused handles by calling NmCloseHanlde, you may get an error stating ERROR_PARAMETER_QUOTA_EXCEEDED. Also, if you are creating multiple parsers or parsing multiple frames simultaneously, you may have to increase the quota using NmApiInitialize.
RPC_E_CHANGED_MODE Returned when opening engine: When you call NmOpenCaptureEngine you may receive this or another COM related error. This is because our capture engine on down level systems like Windows 2003 and Windows XP are single threaded. So you have some options here, but my best advice is to read the section title “Working with Managed and Multi-Threaded Code” in the help.
Different Results than the GUI: The GUI performs one full pass through a capture file when it loads. The first pass parses the entire capture file which builds global and conversations properties. When we display a section of frames in the frame summary, those are parsed again to get all the columns properties. And then when you click on a frame, we parse again to get all the Frame Details. In some cases this can cause you to get different information the second time you parse a frame. This is because global and conversation properties retain the last value set. If you are concerned about this difference, parse through the entire trace once.
Let Your Imagination Run Wild
The API opens up a huge world for analyzing network traffic in a completely automated fashion. From simple experts that provide top level information to complex applications that diagnose difficult network protocols, the API gives you unprecedented access to network protocol data. And within the next few months, we hope to create an Open Source community on http://www.codeplex.com where we will start posting some experts we’ve created internally.
