Network Monitor : Experts

Reassembling Packets with the Network Monitor API

PaulELong — Mon, 12 Oct 2009 18:00:00 GMT

Network traffic by nature is fragmented. Limits of various network packet sizes force protocols to chop up data into multiple frames. When you capture data or read it from a trace with the API (NMAPI) you see only the fragments by default. But as the engine is collecting packets, it can be configured to pass up the reassembled payloads as well. For an intro to how assembly works in the UI, please see the video on reassembly. We also released a recent video on Channel 9 which has some information about the API and reassembly. I would also recommend reading the "Introduction to the Network Monitor API" in the help file for a general background.

Configuring the Parser

The first step is to configure your parser to reassemble. Your API tool for breaking apart a frame is called the Frame Parser object. But to create a frame parser, you start by creating a Frame Parser Configuration. This configuration allows you to add data fields and properties. But it also allows you configure your parser for Reassembly and Conversations. In this case Reassembly might depend on Conversations, so we will enable them both. Here's how I setup my Parser Configuration and Frame Parser.

// Returns a frame parser with a filter and one data field.
// INVALID_HANDLE_VALUE indicates failure.
HANDLE
MyLoadNPL(void)
{
    HANDLE myFrameParser = INVALID_HANDLE_VALUE;
    ULONG ret;

    // Use NULL to load default NPL set.
    ret = NmLoadNplParser(NULL, NmAppendRegisteredNplSets, MyParserBuild, 0, &g_NplParser);

    if(ret == ERROR_SUCCESS){
        ret = NmCreateFrameParserConfiguration(g_NplParser, MyParserBuild, 0, &g_FrameParserConfig);

        if(ret == ERROR_SUCCESS)
        {
            // Order is important here, must turn on Conversations before Reasembly.
            ret = NmConfigConversation(g_FrameParserConfig, NmConversationOptionNone , TRUE);
            if(ret != ERROR_SUCCESS)
            {
                wprintf(L"Failed to config reassembly, error 0x%X\n", ret);
            }

            ret = NmConfigReassembly(g_FrameParserConfig, NmReassemblyOptionNone , TRUE);
            if(ret != ERROR_SUCCESS)
            {
                wprintf(L"Failed to config reassembly, error 0x%X\n", ret);
            }

            // Property so we can show the highest protocol description.
            ret = NmAddProperty(g_FrameParserConfig, L"property.Description", &g_DescPropID);
            if(ret != ERROR_SUCCESS)
            {
                wprintf(L"Failed to add field, error 0x%X\n", ret);
            }

            ret = NmCreateFrameParser(g_FrameParserConfig, &myFrameParser, NmParserOptimizeNone);

            if(ret != ERROR_SUCCESS)
            {
                wprintf(L"Failed to create frame parser, error 0x%X\n", ret);
                NmCloseHandle(g_FrameParserConfig);
                NmCloseHandle(g_NplParser);
                return INVALID_HANDLE_VALUE;
            }
        }
        else
        {
            wprintf(L"Unable to load parser config, error 0x%X\n", ret);
            NmCloseHandle(g_NplParser);
            return INVALID_HANDLE_VALUE;
        }

    }
    else
    {
        wprintf(L"Unable to load NPL\n");
        return INVALID_HANDLE_VALUE;
    }

    return(myFrameParser);
}

After creating your Frame Parser Configuration Object, you'll want to set any options first. This will let the engine optimize properly when adding other things like properties and data fields. It's also important that you turn on conversations before reassembly. Placing them in the wrong order will turn off Reassembly due to a bug in our API.

Above we also added a property so that I can show the description of the current frame. This is not necessary for reassembly to work, but it helps us understand the example.

Parsing the Frames

It is up to the parsers (NPL) to mark each frames fragment type: First=1, Middle=2, Last=3 or None=0. The engine tracks these fragments and returns a new inserted raw frame once a Last fragment is detected for a specific protocol.

When you parse a raw frame using NmParseFrame, the last parameter passed is a pointer to a HANDLE that will contain an InsertedRawFrame if one is present. Otherwise this value will be set to INVALID_HANDLE_VALUE for any frame that doesn't return a reassembled payload. For frames that do have a reassembled payload, the handle returned will contain a raw frame. You can now use your frame parser to parse this raw frame.

The main part of my code simply retrieves frames from the capture file iteratively and calls ParseFrame, which does all the work. If an inserted frame is found, the function calls itself. The function is recursive because the handles for a RawFrame, ParsedFrame and InsertedRawFrame have to be closed in the order they were opened. There are other ways to do this, but for this example a recursive routine was the easiest. You will also want to insure the frames are in order. For instance you could use NmOpenCaptureFileInOrder to make sure the TCP frames are ordered correctly.

In my case I parse and display all the frames so that you can get a feel for the pattern that occurs as frames fragments are marked by the engine. It also helps to shows how fragmentation looks at different protocol layers. If you were interested in only the reassembled frames or frames that are not fragmented to begin with, you could identify those as having a fragment type of None and no InsertedRawFrame.

Here's the recursive frame parsing routine:

// Recursive Parsing routine.  If an inserted frame is found, the recusive routine is called again.  This
// allows us to close our handles in the order there were created.
void
MyParseFrame(HANDLE frameParser, HANDLE rawFrame, ULONG curFrame, PULONG reassembleFrames, int reassembleCount)
{
    ULONG ret;
    HANDLE ParsedFrame = INVALID_HANDLE_VALUE;
    HANDLE InsRawFrame = INVALID_HANDLE_VALUE;

    // NmUseFrameNumber and valid unique frame numbers are neccessary for Reassembly to work properly.
    ret = NmParseFrame(frameParser, rawFrame, curFrame + *reassembleFrames, NmFieldDisplayStringRequired | NmUseFrameNumberParameter, &ParsedFrame, &InsRawFrame); 
    if(ret == ERROR_SUCCESS)
    {
        // Returns the highest level protocol description just to show which
        // frame we are working on.
        PBYTE buf = GetDescription(frameParser);

        // Get the fragment information which helps understand what is happening,
        // but not needed for reassembly to work.
        NM_FRAGMENTATION_INFO FragInfo;
        GetFragType(ParsedFrame, &FragInfo);

        wprintf(L"%5d-%d: %5d      %-5.5s-%d    %-.45s\n", curFrame+1, reassembleCount, curFrame+(*reassembleFrames)+1, FragInfo.FragmentedProtocolName, FragInfo.FragmentType, buf);

        free(buf);

        if(InsRawFrame != INVALID_HANDLE_VALUE)
        {
            (*reassembleFrames)++;
            MyParseFrame(frameParser, InsRawFrame, curFrame, reassembleFrames, reassembleCount+1);

            NmCloseHandle(InsRawFrame);
        }
    }

    NmCloseHandle(ParsedFrame);
    NmCloseHandle(InsRawFrame);
}

When doing reassembly you must add the Frame Number parameter. It must also be unique, so you have to remember to increment when adding and parsing the reassembled frames. The GetFragType uses NmGetFrameFragmentInfo API call to determine the fragment type and protocol. You can look at the full example below to see how it works in details, but those ancillary pieces are pretty straight forward.

Looking at an Example

Below is the partial output for an example capture. In my notation, the Frame# contains a number after the dash that shows when multiple iterations occur on a frame. The Reassem# is the frame number that would appear in a reassembled trace in the UI and is what is used to seed each frame with a unique frame number.

Frame# Reassem# FragType Description

5-0: 5 TCP -1 HTTP:Response, HTTP/1.1, Status: Bad gateway,

6-0: 6 TCP -2 TCP:[Continuation to #5]Flags=...A...., SrcPo

7-0: 7 -0 TCP:Flags=...A...., SrcPort=49382, DstPort=HT

8-0: 8 TCP -3 TCP:[Continuation to #5]Flags=...AP..., SrcPo

8-1: 9 -0 HTTP:Response, HTTP/1.1, Status: Bad gateway,

...

In original frames 5-8, you can see a typical TCP fragmentation. Frame 5 is a TCP First fragment. Frame 6 is a middle fragment and frame 7 is traveling in the opposite direction so it's not part of this reassembly stream. Frame 8 is the last frame in the reassembled TCP payload which is marked as the Last fragment. This is where the Inserted Raw Frame is valid and the recursive call to parse the frame would occur. Frame 8-1, is the parsed inserted frame which you can see matches the description of frame #5, but if you looked at it, there would be two differences.

First, since it's an inserted frame it will have a PayloadHeader structure as its top protocol. This is a protocol we manufactured to take the place of the carrying protocol, in this case TCP. Having a duplicate TCP frame would confuse our parsers and perhaps the user as well. So this header takes it place and calls HTTP directly.

Second, this frame will have a larger payload. It will consist of all the payload data from frame 5, 6, and 8.

Two Level Reassembly

In this next example, both TCP and HTTP has fragmented data.

...

33-0: 36 TCP -1 HTTP:Response, HTTP/1.1, Status: Ok, URL: htt

34-0: 37 TCP -2 TCP:[Continuation to #36]Flags=...A...., SrcP

35-0: 38 -0 TCP:Flags=...A...., SrcPort=49384, DstPort=HT

36-0: 39 TCP -3 TCP:[Continuation to #36]Flags=...AP..., SrcP

36-1: 40 HTTP -1 HTTP:Response, HTTP/1.1, Status: Ok, URL: htt

37-0: 41 TCP -1 HTTP:HTTP Payload, URL: http://www.google.com

38-0: 42 -0 TCP:Flags=...A...., SrcPort=49384, DstPort=HT

39-0: 43 TCP -2 TCP:[Continuation to #41]Flags=...A...., SrcP

40-0: 44 TCP -2 TCP:[Continuation to #41]Flags=...A...., SrcP

41-0: 45 -0 TCP:Flags=...A...., SrcPort=49384, DstPort=HT

42-0: 46 TCP -3 TCP:[Continuation to #41]Flags=...AP..., SrcP

42-1: 47 HTTP -3 HTTP:HTTP Payload, URL: http://www.google.com

42-2: 48 -0 HTTP:Response, HTTP/1.1, Status: Ok, URL: htt

...

Frames 33-36 make up the first HTTP fragment. As you can see, the inserted frame at 36-1 is a First fragment, but the protocol is now HTTP. Frames 37-42 make up the next HTTP fragment which is inserted at frame 42-1. This inserted frame is the HTTP Last fragment so now there is yet another inserted raw frame that we must iterate through and parse. Frame 42-2 is the final reassembled frame and contains the original HTTP Response in its entirety. The description matches frame 33 because it the data starts with payload in that frame but it also includes the payloads from frames 34, 36, 37, 39, 40, and 42. However, from the engines point of view, it really collects the payloads from frame 36-1 and 42-1. But each of these is made up from the fragmented frames mentioned above.

The Whole Shebang

Below I've placed the entire source code for the example described in this blog. While it depends on which protocols you are interested in, having access to the reassembled data can provide you with the big picture especially when focusing on application layer traffic.

#include "stdafx.h"
#include "windows.h"
#include "stdio.h"
#include "stdlib.h"
#include "objbase.h"
#include "ntddndis.h"
#include "NMApi.h"

HANDLE g_NplParser = INVALID_HANDLE_VALUE;
HANDLE g_FrameParserConfig = INVALID_HANDLE_VALUE;

ULONG g_DescPropID = 0;    // Global Description Property ID.

// Callback for parser building messages
void __stdcall
MyParserBuild(PVOID Context, ULONG StatusCode, LPCWSTR lpDescription, ULONG ErrorType)
{
    wprintf(L"%s\n", lpDescription);
}

// Returns a frame parser with a filter and one data field.
// INVALID_HANDLE_VALUE indicates failure.
HANDLE
MyLoadNPL(void)
{
    HANDLE myFrameParser = INVALID_HANDLE_VALUE;
    ULONG ret;

    // Use NULL to load default NPL set.
    ret = NmLoadNplParser(NULL, NmAppendRegisteredNplSets, MyParserBuild, 0, &g_NplParser);

    if(ret == ERROR_SUCCESS){
        ret = NmCreateFrameParserConfiguration(g_NplParser, MyParserBuild, 0, &g_FrameParserConfig);

        if(ret == ERROR_SUCCESS)
        {
            // Order is important here, must turn on Conversations before Reasembly.
            ret = NmConfigConversation(g_FrameParserConfig, NmConversationOptionNone , TRUE);
            if(ret != ERROR_SUCCESS)
            {
                wprintf(L"Failed to config reassembly, error 0x%X\n", ret);
            }

            ret = NmConfigReassembly(g_FrameParserConfig, NmReassemblyOptionNone , TRUE);
            if(ret != ERROR_SUCCESS)
            {
                wprintf(L"Failed to config reassembly, error 0x%X\n", ret);
            }

            // Property so we can show the highest protocol description.
            ret = NmAddProperty(g_FrameParserConfig, L"property.Description", &g_DescPropID);
            if(ret != ERROR_SUCCESS)
            {
                wprintf(L"Failed to add field, error 0x%X\n", ret);
            }

            ret = NmCreateFrameParser(g_FrameParserConfig, &myFrameParser, NmParserOptimizeNone);

            if(ret != ERROR_SUCCESS)
            {
                wprintf(L"Failed to create frame parser, error 0x%X\n", ret);
                NmCloseHandle(g_FrameParserConfig);
                NmCloseHandle(g_NplParser);
                return INVALID_HANDLE_VALUE;
            }
        }
        else
        {
            wprintf(L"Unable to load parser config, error 0x%X\n", ret);
            NmCloseHandle(g_NplParser);
            return INVALID_HANDLE_VALUE;
        }

    }
    else
    {
        wprintf(L"Unable to load NPL\n");
        return INVALID_HANDLE_VALUE;
    }

    return(myFrameParser);
}

void
UnLoadNPL(void)
{
    NmCloseHandle(g_NplParser);
    NmCloseHandle(g_FrameParserConfig);
}

ULONG
GetFragType(HANDLE parsedFrame, NM_FRAGMENTATION_INFO *FragInfo)
{
    ULONG ret;

    FragInfo->Size = sizeof(FragInfo);
    ret = NmGetFrameFragmentInfo(parsedFrame, FragInfo);

    return ret;
}

PBYTE
GetDescription(HANDLE frameParser)
{
    ULONG ret;
    NM_PROPERTY_INFO PropInfo;

    // Find out the size of the description property so we can allocate a buffer.
    // MUST intialize the size and name pointer or NmGetPropertyInfo will fail.
    PropInfo.Size = sizeof(PropInfo);
    PropInfo.Name = NULL;
    ret = NmGetPropertyInfo(frameParser, g_DescPropID, &PropInfo);
    if(ret != ERROR_SUCCESS)
    {
        wprintf(L"Error calling NmGetPropertyInfo, %d\n", ret);
        return NULL;
    }

    ULONG retlen = 0;
    NmPropertyValueType propType;
    // Add size of WCHAR for null terminator
    PBYTE buf = (PBYTE)malloc(PropInfo.ValueSize + sizeof(WCHAR));
    ret = NmGetPropertyValueById(frameParser, g_DescPropID, PropInfo.ValueSize, buf, &retlen, &propType);
    if(ret != ERROR_SUCCESS)
    {
        wprintf(L"Error calling NmGetPropertyValueById, %d\n", ret);
        return NULL;
    }

    return buf;
}

// Recursive Parsing routine.  If an inserted frame is found, the recusive routine is called again.  This
// allows us to close our handles in the order there were created.
void
MyParseFrame(HANDLE frameParser, HANDLE rawFrame, ULONG curFrame, PULONG reassembleFrames, int reassembleCount)
{
    ULONG ret;
    HANDLE ParsedFrame = INVALID_HANDLE_VALUE;
    HANDLE InsRawFrame = INVALID_HANDLE_VALUE;

    // NmUseFrameNumber and valid unique frame numbers are neccessary for Reassembly to work properly.
    ret = NmParseFrame(frameParser, rawFrame, curFrame + *reassembleFrames, NmFieldDisplayStringRequired | NmUseFrameNumberParameter, &ParsedFrame, &InsRawFrame); 
    if(ret == ERROR_SUCCESS)
    {
        // Returns the highest level protocol description just to show which
        // frame we are working on.
        PBYTE buf = GetDescription(frameParser);

        // Get the fragment information which helps understand what is happening,
        // but not needed for reassembly to work.
        NM_FRAGMENTATION_INFO FragInfo;
        GetFragType(ParsedFrame, &FragInfo);

        wprintf(L"%5d-%d: %5d      %-5.5s-%d    %-.45s\n", curFrame+1, reassembleCount, curFrame+(*reassembleFrames)+1, FragInfo.FragmentedProtocolName, FragInfo.FragmentType, buf);

        free(buf);

        if(InsRawFrame != INVALID_HANDLE_VALUE)
        {
            (*reassembleFrames)++;
            MyParseFrame(frameParser, InsRawFrame, curFrame, reassembleFrames, reassembleCount+1);

            NmCloseHandle(InsRawFrame);
        }
    }

    NmCloseHandle(ParsedFrame);
    NmCloseHandle(InsRawFrame);
}

int __cdecl wmain(int argc, WCHAR* argv[])
{
    ULONG ret = ERROR_SUCCESS;
    // The first paramryrt should be a file.
    if(argc <= 1){
        wprintf(L"Expect a file name as the only command line parameter\n");
        return -1;
    }

    // Open the specified capture file.
    HANDLE myCaptureFile = INVALID_HANDLE_VALUE;
    if(ERROR_SUCCESS == NmOpenCaptureFile(argv[1], &myCaptureFile))
    {
        // Initialize the parser engine and return a frame parser.
        HANDLE myFrameParser = MyLoadNPL();
        if(myFrameParser != INVALID_HANDLE_VALUE)
        {
            ULONG myFrameCount = 0;
            ret = NmGetFrameCount(myCaptureFile, &myFrameCount); 
            if(ret == ERROR_SUCCESS)
            {
                ULONG totReassembledFrames = 0;
                HANDLE myRawFrame = INVALID_HANDLE_VALUE;

                wprintf(L"Frame#   Reassem#  FragType    Description\n");
                for(ULONG i = 0; i < myFrameCount; i++)
                {
                    HANDLE myParsedFrame = INVALID_HANDLE_VALUE;
                    ret = NmGetFrame(myCaptureFile, i, &myRawFrame); 
                    if(ret == ERROR_SUCCESS)
                    {
                        MyParseFrame(myFrameParser, myRawFrame, i, &totReassembledFrames, 0);

                        NmCloseHandle(myRawFrame);
                    }
                    else
                    {
                        // Print an error, but continue to loop.
                        wprintf(L"Errors getting raw frame %d\n", i+1);
                    }
                }
            }

            NmCloseHandle(myFrameParser);
        }
        else
        {
            wprintf(L"Errors creating frame parser\n");
        }

        NmCloseHandle(myCaptureFile);
    }
    else
    {
        wprintf(L"Errors openning capture file\n");
    }

    // Release global handles.
    UnLoadNPL();

    return 0;
}

Top Users Expert for Network Monitor 3.3

PaulELong — Fri, 01 May 2009 00:48:00 GMT

One of the major new features in Network Monitor 3.3 is the ability to run experts directly from the UI. And now NMTopUsers is available from our Experts Portal. Plus as it's a CodePlex project, we have opened the source code as well. It's a fairly simple C# project which uses the NMAPI to access data from a trace. You can view and download the source code if you are interested in more details.

What Does NMTopUsers Do?

The problem we are trying to solve is a way to quickly identify the heaviest users of network traffic. For instance you might want to understand if there's a computer on your network that is hogging all the bandwidth. You might also want to identify a chatty machine which could be an indication that it has been infected with a virus or adware.

There are actually two experts in one here. The Endpoint version shows traffic for each machine address, IPv4, or IPv6 address on your network. The Conversation version, on the other hand, shows traffic based on a pair of machines, IPv4, or IPv6 addresses so you can understand the traffic involved between machines.

How Does NMTopUsers Work?

Once you've installed the expert by running the MSI, "Top Users by Endpoint" and/or "Top Users by Conversation" will appear in the experts menu. Once you run the expert a new window will show up and display a data grid. The data will depend on any display filters applied or conversation tree nodes you might have selected. The data grid contains a list of nodes or conversations and then statics on the frames and bytes that have been sent and/or received.

By default the data is sorted by Total Bytes. But you can click on any column header to sort by that column.

The Address Type drop-down lets you select which types of addresses you want to see. By default we only show IPv4 addresses, but you can add Machine addresses and IPv6 addresses as well.

The Tree View allows you to see the IPv4 and IPv6 address as they relate to the Machine addresses that they belong too. But as you can imagine this option is only available if you've enabled the Machine address type. Also if you've re-sorted on a different column in the Tree View, you can reset to the original order by using the Reset Tree button.

Finally you can create a Pie or Bar chart of the information. While sometimes this can be cluttered due to the number of addresses, it can give you a high level overview of the usage. Keep in mind that the Bar chart can't display if you have Machine Addresses selected and multiple IPv4 or IPv6 addresses for a single machine address. This is because the bar chart attempts to line up each IPv4/IPv6 address with its matching Machine address and this doesn't make sense with multiple IP addresses.

Give it a Run

Please go to the Experts Portal and download both Top Users for Conversations and Endpoint. Try it out!

Network Monitor 3.3 Beta Available on Connect

PaulELong — Wed, 04 Mar 2009 23:25:00 GMT

It’s time again for a new Beta version of Network Monitor. Using your feedback we’ve added some compelling new features! One in particular is the ability to launch experts from the UI. And what use would this be without something to launch? We’ll also be releasing some cool experts, like Top Users and Simple Search. In fact Simple Search is already available, just use the Experts menu from a saved trace to connect to our Experts landing page. Return every so often to see what is new.

What’s an Expert? What is Simple Search?

An expert is a tool, run separately from Network Monitor that performs expert analysis on your trace data. For instance, Simple Search will allow you to search all trace data for strings and better yet, regular expressions! Need to find an email address or phone number, but don’t know exactly which one? A regular expression, using Simple Search, lets you find this type of string. Just launch the expert from the menu and search away.

But there’s so much more, so let’s list out what else is new.

What’s New in NM3.3 Beta

Ability to capture on WWAN and Tunnel interfaces on Win7
Critical fixes to NM3.3 to operate correctly with Hyper-V
Right-click add to alias. Right-click a frame in the Frame Summary window with an IPv4, IPv6, or MAC address to add that address as a new alias.

Right click go to definition: Right-click a field in the Frame Details windows and select Go To Data Field Definition or Go To Data Type Definition to see where the field is defined in the NPL parsers.
Auto-scroll. See the most recent traffic as it comes in. In a live capture, click the Autoscroll button on the main toolbar to have the Frame Summary window automatically scroll down to display the most recent frames as they come in. Click Autoscroll again to freeze the view in its present location.
Experts available online: Experts are stand-alone applications that analyze Network Monitor capture data. Various experts are available online at http://go.microsoft.com/fwlink/?LinkID=133950.
Frame Comments: Attach comments to frames in a saved capture file. Select the Frame Comments tab in the lower-right window to add, view, edit, or delete comments.

API Extensions: API methods have been added to enable access to conversation information, properties, field display strings, and comments.
Ability to open ETL files and correlate information by Network Tracing scenario.

See our Release Notes for a complete list of new features and known issues.

Gimme, Gimme, Gimme!

The Beta is available on our connect site at https://connect.microsoft.com/site/sitehome.aspx?SiteID=216. But it is only available as a download if you join Connect. By joining you’ll also get updated when we release and be able to file bugs with us if you find issues. So give NM3.3 Beta a test drive and please let us know what you think!

Provide feedback at http://go.microsoft.com/fwlink/?LinkID=142458 (you will need to login with your Connect Network Monitor credentials to see the Beta survey). Our voting website, input.microsoft.com has an expired certificate that we are working to renew. For now, you can tell your browser to ignore the expired certificate as we address this problem.

Using Columns and Properties

PaulELong — Thu, 01 Mar 2007 20:26:00 GMT

You might have had an occasion to add a new column in Network Monitor 3.0. But the list of available choices might be quite daunting. What you may not know, however, is that the list is derived from properties in the NPL script that makes up each parser. This means you can add any kind of information you want as a column.

This blog will attempt to show you some of the more useful columns you can add. But we will also explore the creation of properties in NPL for more advanced analysis.

The Skinny on Columns

It’s probably important to give some background about how columns work in NM3. A common misconception is that the column information is actually the data, but in reality it’s a string describing the data. That means that what’s in the Source Column is not the hex data for 192.168.0.1, but a string that describes it.

This is important because a property is just a variable in NPL that can change based on other factors. For instance the Source column can display the Hardware Address, IPv4 Address, or IPv4 Address based on the frame in question. But in each case, the property is still just a string representation.

Time Related Properties

For starters, let’s display what the default column layout is.

By default, NM3 show “Time Offset” as one of its columns. But it’s often useful to see either the Time of Day or the Time Delta (time from last packet). Well both of these items are available for addition. To add these in, right mouse click any of the column headers and select “Choose Columns…”.

Note that you can also save the current column layout as the default for new sessions, or restore the default column layout. Once you select the menu item, you will see a dialog like the one below.

On the left is a list of column headers you can add, and on the right, your currently enabled column headers. To add a column, select the column on the left and hit add. Or to remove, click on an item on the right and select remove. You can also order the columns using the Move Up/Move Down buttons. It is also possible to move the columns around from the Frame Summary by drag and dropping the columns around, so you don’t need to go to the Choose Frame Summary Columns dialog to do this.

Finding the Column to Add

The first problem you may encounter is that the list is huge, which is why I want to highlight some of the more useful headers you can add without NPL modification. But you still have to find the column, which you could do by just scrolling through the list. But there is a faster way! If you start typing a few of the letters, it should get you to that section more quickly. So, in this example I want to show you how to add the “Time of Day” and “Time Delta” columns. So first click in the Disabled Columns portion of the dialog, and start typing “T I m e” without the spaces, and you see that the items that start with Time appear. The first one you can see is “Time Delta”, so let’s add this to the list by clicking the “Add” button. Then also click on the “Time Of Day” and add it.

Once you click OK from the dialog, you should now see the new columns appear. The “Time Of Day” column will show you the time when the frame was captured. The “Time Delta” displays the time from the last frame. This is updated when a filter is applied, so if frames 1 and 2 are 1 second apart, and frames 2 and 2 are 1 second apart, a filter which only shows frames 1 and 3 will display a time delta of 2 seconds. For example given the following data:

Time Of Day	Time Delta	Frame	Time Offset
15:41:35.331	0.000000	1	0.000000
15:41:36.331	1.000000	2	1.000000
15:41:37.331	1.000000	3	2.000000

If filter that includes frames 2 and 3 are added, the resulting Rime Delta column would look as follows.

Time Of Day	Time Delta	Frame	Time Offset
15:41:35.331	0.000000	1	0.000000
15:41:37.331	2.000000	3	2.000000

Note: If you try to filter on the property for Time Delta in NM3.0, say to find all frames with a delta greater than 1 second, you won’t return any frames. We currently don’t have a way to properly apply this property in a filter. We hope to address this in future version of Network Monitor.

Other Useful Properties

So I wanted to provide a list of other useful properties and their descriptions. This should provide you a reference of some of the more popular column entries.

TCPDescription

It’s often useful to always see the TCP Description rather than the upper protocols. By default, NM3 shows the highest level protocol in the Frame Summary Description field. So by adding the TCPDescription property to the columns, you can always see this information. This is very useful for t-shooting strictly TCP problems.

Other Protocol Description Fields

As with TCP, you may want to see other protocol header rather than other upper level protocols. For instance SMBDescription or HTTPDescription, could be added as columns.

TCPAckNumber, TCPSequenceRange, and TCPFlags

These three properties are also great for t-shooting TCP issues. You may actually find this more useful than the TCP Description as the column information lines up the ACK/SEQ number for easy reference.

SourceNetworkAddress and DestinationNetworkAddress

The way the Source/Destination columns are populated today, you don’t always see the real IPv4 or IPv6 address. Instead we show this based on the following rules. If a higher numbered item exists, we show that item.

Alias for the IP address
Resolved Name of IP address
Real IP address
Hardware Address

So in cases where you’d like to see the real IP address and a resolved name exists, turning off the aliases doesn’t show you the real IP address. So you view the real IP address all the time by adding these columns.

SourceHardwareAddress and DestinationHardwareAddress

Like the IP address, the Source and Destination Hardware addresses don’t always show up because of the same rules above. So adding these columns will allow you to always see the Hardware Addresses.

Creating Your Own Properties

On occasion you might find it useful to create your own properties for viewing as a column in NM3. There may be a frame summary data field that you want to show up as a column. You might also want to combine a few data fields and create a column that combines that information. So let’s create a few example properties to show you how this is done.

IPv4 Identification: Adding just a simple property

In this case we’ll create a property for IPv4’s Identification field. This is the field that uniquely tags an IPv4 packet and can be used to associate the same frame from two different captures. We’ll start by opening ipv4.npl. In NM3, you can do this in the parsers tab by searching for the file in the Parser Files in the Object View. Once you find IPv4.npl, double click it to edit. Next, search for the field called Identification, it should look like this:

UINT16 TotalLength;
UINT16 Identification;

UINT16 FragmentFlags
{
…

Now we add in a property by placing it in square brackets above the data field we want to create a property for.

UINT16 TotalLength;
[IPv4Identification]
UINT16 Identification;

UINT16 FragmentFlags
{
…

Now save your changed NPL, and reload the parser. From the menu select Tools, Reload parsers. Once the build is complete you can load a trace with IPv4 traffic and add the column you just created.

TCP Port Pair: Adding a derived property with two data fields

In this example, we are going to get a bit fancier. We will create a property that shows both the source and destination port in Hex, along with the friendly port name. Remember that since a property is just a string, we can set it to anything we want. The property does not have to only represent one value.

So in this case we will edit TCP.NPL like we did IPv4.NPL before and look for this section of code.

UINT16 SrcPort = PortNameTable(this);
[Pair = Port, DestinationPort]
UINT16 DstPort = PortNameTable(this);

We’ll want to add in our property right before the last data field we will reference. In this case there is already a bracketed section above DstPort, but these directive sections can have multiple lines separated by a comma. So we will change this to

UINT16 SrcPort = PortNameTable(this);
[
    Pair = Port, DestinationPort,
    TcpPortDesc=FormatString(“%d (%s) -> %d (%s)”, 
    SrcPort, 
    PortNameTable(SrcPort),
    DestPort,
    PortNameTable(DstPort))
]
UINT16 DstPort = PortNameTable(this);

As you see we’ve called our new property TcpPortDesc and we are using FormatString, which is like printf in C, to format the way our property will look. We have 4 parameters for FormatString, the first two are the Source port and friendly name, and the second two are for the destination. We use PortNameTable to convert the number of the port to a descriptive string. PortNameTable is just a table that has already been defined for you in GlobalTables.NPL. Tables are simple structures that given one value, we convert and return a different value.

Once you add in this new column, as described before, you will see a column that displays as follows:

0x50 (HTTP(80)) -> 0xC0E6 (49382)

TCPPayload Info: Adding a property to display TCP payload as ASCII

Often you want to look at the TCP data as ASCII. For FTP conversations, the text reads like a story. But we must also work around a bug in NM3 which limits the text in a field to 0x103 bytes. So what we’ll do is limit the text when we collect it.

Here’s the original section of TCP.NPL we are going to modify.

//This maybe useful for TCP payload data filter for all TCP based protocols
//A possible filter maybe: Property.TCPPayload.Contains("*****")
//[TCPPayload = AsciiString(frameData, offset, TCPPayloadLength)]
[
DataFieldFrameLength = frameOffset + TCPPayloadLength,

In this case the statement for the entire payload is commented out. You can see that this property may be useful to enable if you want to search TCP Payloads using something like

Property.TCPPayload.Contains(“Anything”)

So we are going to enable this property and add another. Let’s change the code to this:

//This maybe useful for TCP payload data filter for all TCP based protocols
//A possible filter maybe: Property.TCPPayload.Contains("*****")
[
TCPPayload = AsciiString(FrameData, FrameOffset, TCPPayloadLength),
TCPPayloadDisp = AsciiString(FrameData, FrameOffset, 0x103),
DataFieldFrameLength = frameOffset + TCPPayloadLength,

This adds in two new properties. The first, TCPPayload, will allow us to do searching like described above. The first parameter is the data we are going to search in. When you use FrameData, it means the current frame information. The second parameter is the offset in the buffer, and another global property FrameOffset points to the current location in the frame. The final parameter is the length, and the property TCPPayloadLength holds that value for TCP.

But we can’t use TCPPayload for display because if the length is greater than 0x103 bytes, it won’t get displayed at all. For TCPPayloadDisp we fix this by explicitly setting the length.

It’s All About What You Want to See

Using columns for displaying useful information can save you time and help you t-shoot problems. Perhaps in future version of NM3 we’ll have a way to define standard column setups so you can switch from you TCP T-shooting setup to your SMB t-shooting column setup. Also we do plan to provide some organization to the properties to make it easier to find things in that long list.

Part 2: TCP Performance Expert and General Trouble Shooting

PaulELong — Fri, 26 Jan 2007 21:58:00 GMT

Performance issues are one of the more difficult problems to trouble shoot. Without a baseline, it's often hard to determine if something is really slower. But TCP does contain some built-in behavioral patterns that can be used as a signal to tell you something may be wrong with your network.

So the purpose of this article is to provide some indicators in TCP you can look at to investigate and a way to provide a simple graph of TCP traffic that can help you determine if there is a problem and if so what kind.

TCP Clues

TCP is the layer that is in charge of making sure your packet gets delivered. It tags each packet with a sequence number and when something is missing, the client informs the sender. Below I've listed some general things you can filter on in NM3 to give you clues to see if your network is working properly.

TCP Retransmits:

When dissecting a TCP trace, one of the more obvious problems you can spot are TCP retransmits. A retransmit occurs when a client detects a missing packet. From the sender's perspective, he now has sent the packet twice, so the second packet is called a retransmit. While a certain number of retransmits may occur without causing problem, excessive retransmits may be an indication that your network is sick. In NM3, you can search for retransmitted frames by using the following filter.

Property.TCPRetransmit

One you apply the filter, all frames that have been retransmitted will be displayed. Also each displayed frame will tell you the original frame that it is a retransmit of in the TCP Frame Summary. Some level of retransmits may be acceptable, but varies based on your network topology. Here's an example of how it displays in NM3:

Frame	Time Offs	Source	Destination	Description
457	16.375976	Sndr	Rcvr	TCP: [Continuation to #402]Flags=....A..., SrcPort=1236, DstPort=Microsoft-DS(445), Len=1460, Seq=658111387 - 658112847, Ack=2995420839, Win=65484 (scale factor 0) = 65484
464	16.577148	Sndr	Rcvr	TCP: [ReTransmit #457][Continuation to #402]Flags=....A..., SrcPort=1236, DstPort=Microsoft-DS(445), Len=1460, Seq=658111387 - 658112847, Ack=2995420839, Win=65484 (scale factor 0) = 65484

TCP Fast Retransmits

In some cases you may see multiple ACKs one after another in quick succession. The receiver can send these ACKs to the sender to indicate it is missing a TCP sequence range. Normally, a timeout would occur for the acknowledgement of a particular sequence before a retransmit occurs. However, if the TCP sender supports fast retransmit then it will occur after receiving these multiple ACKs. A retransmission generated by fast retransmit also changes the back-off algorithm used. If a retransmit occurs due to a timeout, then the sender reverts back to "slow start." However, if a retransmit occurs because of a fast retransmit then the sender goes into "congestion avoidance." [See RFC 2581 for more information on Congestion Avoidance and Slow Start.]

The response is called a "Fast Retransmit". This can be the behavior you see from one side when packets get lost in another segment of your network. In NM3 you can search for Fast Retransmits with this filter.

Property.TCPFastRetransmits

An example of a Fast retransmit looks like this:

TCP: [Request Fast-Retransmits #370]Flags=....A..., SrcPort=1268, DstPort=LDAP(389), Len=0, Seq=2021124596, Ack=1458852541, Win=64240 (scale factor 0) = 0

TCP SACK option

The SACK option (selective acknowledgments) is like an ACK, but the difference is that it can keep track of multiple sections of missing data. A normal ACK acknowledges the last consecutive sequence number that it received. In contrast a SACK can keep track of multiple missing segments. The SACK option contains multiple segments relating which pieces it has acknowledged and which are now lost. The number of segments is constrained to the amount of space available for TCP options. You can filter on these as well to see if your network is losing packets.

tcp.TCPOptions.Option.SACK

TCP Resets

Resets aren't always a sign that something is wrong. You should try to look at the traffic around a reset to determine if it looks normal or not. Resets can occur when an application shuts down, or if a router is configure to block a port. But they also occur when a problem occurs in a TCP session. So these can sometimes be an indication that something is wrong. The filter to find resets is:

tcp.Flags.Reset

Where's Waldo – Spotting a bad TCP Connection

Being able to take a trace and visualize it can be a useful way to look for performance issues. Most humans are better at looking at a picture and finding differences than analyzing data. Especially the large sequence numbers involved with TCP traffic.

Limitations with this Expert

Unfortunately, there are limitations to that amount of data this expert can deal with. The specific limitations will be due to the memory/CPU power of the machine you are work on. Excel, which this expert relies on, isn't build to plot 1000's of points. So you may want to limit that amount of data you try to analyze. For the following examples, a 500 MB file was transferred from the client to the server.

Using the TCP Expert

If you are familiar with the Part 1, (http://blogs.technet.com/netmon/archive/2006/11/30/part-1-poor-man-s-expert-using-excel-top-users.aspx), expert that locates the Top Users, then the procedure is much the same. Once you have the Excel Sheet Ready and the NM3 columns aligned, it's just a matter of copying the data in to the clipboard and hitting a button which launches a macro in Excel and creates the graphs we will examine later on.

Setting Up Your Excel Spreadsheet

Basically you just create a new spreadsheet, create a new macro (TCPPerf) and edit it, then paste the code at the bottom of this article. Once you complete this step, you can then take one of the sheets, (I delete all but one of them), and give it a default name like TCPPerf. And finally, create a button on this sheet and attach the TCPPerf macro to it.

Grabbing the data to Analyze from Network Monitor

This section describes how to prepare NM3 so we get the necessary columns for our calculations to appear. In NM3 we can add columns for any property that is exposed by the parsers. Actually you can also add any properties you want as well, so basically any piece of data is fair game. In the case of TCP, we want to add in the Seq/Ack Numbers as well as the Window Size and Payload length. So specifically add in columns for TCP Seq Number, TCP Ack Number, Windows Size, and TCPPayload Length in that order. Place them right after Time Offset. The resulting layout should look as follows.

After opening an existing trace or taking a new one, the next step is to filter down the specific data you want to analyze. In my case, I copied a file from my machine to a server. Since Window Size is negotiated in the TCP 3-way handshake, I made sure to disconnect to my client from the server so that may trace contains the entire conversation. I did this from the server in Computer Management under Shared Folders, Sessions. You can simply right click the session in question and select Close Session. Obviously you don't want to do this if you have something important occurring between this session and the server.

In NM3 it's easy to filter the conversations down, by using the Conversation Tree on the left. [Note: This requires that conversations are enabled when the trace is opened or a capture is started. You can set this option from the Start Page.]

Once you find the IP address pair, you can select each TCP conversation below, and look for the traffic relating to your copy. This traffic will appear as a bunch of SMB Reads (or SMB Writes if you copy to the server), intermixed with a bunch of TCP Continuation traffic.

SMB: R; Write Andx, FID = 0x4004 (\t1_up.tst@#24), 61440 bytes

SMB: C; Write Andx, FID = 0x4004 (\t1_up.tst@#24), 61440 bytes at Offset 184320

TCP: [Continuation to #256]Flags=....A..., SrcPort=1236, DstPort=Microsoft-DS(445), Len=1460, Seq=657940191 - 657941651, Ack=2995420737, Win=64118 (scale factor 0) = 64118

TCP: [Continuation to #256]Flags=....A..., SrcPort=1236, DstPort=Microsoft-DS(445), Len=1460, Seq=657941651 - 657943111, Ack=2995420737, Win=64118 (scale factor 0) = 64118

Once you have located and selected the appropriate TCP Conversation, highlight all the frames involved and copy them to the clipboard. I often use keystrokes, so I did this by selecting the first frame, hit Shift+Ctrl+End, and then right mouse click and copied them to the clipboard.

Now you simply open the Excel Spread sheet you created earlier and click the button. This will results in 4 sheets. The first sheet, PerfData, is just a copy of the data from NM3, plus some new columns that are based off this data. The second sheet, TCPPerf_PerfData, is data copied from the first sheet so that we can sort based off the data we calculated. The final two sheets contain the charts for each side of the transfer. The chart is named based on the machine adress that is sending the data.

Interpreting the Data

Since the resulting data we'll examine is a graph, it's useful to examine different graphs that are a result of different issues so we can refer to these examples as an indicator to the type of problem we may have. In my tests, I have duplicated the following situations: Increased Round Trip Time, Packet Loss, Small Window Size, and Bandwidth Reduction. All traces are taken from the sender (client) which is sending data to the server.

The data is graphed so that the left side axis contains the sequence numbers scaled for Length and UnACKed data. The right side axis is the scaled for the Window Size. Two different scales are used because often the Window size is much larger than the data being sent. UnACKed data is data which have been sent by the sender but has not been acknowledged by the receiver after the TCP timeout expires.

The Graphs

Each test was a copy of around a 500 Meg file from my machine to a server. I used the CMD prompt to do the copy to avoid unwanted Explorer traffic. So we'll start by looking at a base line capture. In this case the Round-Trip-Delay is less than a millisecond, and there are no packets lost. The bandwidth is around 100 MB.

Baseline - < 1 Millisecond RTT, 100 MB, No Packet Loss

In many of the examples I change the horizontal axis min/max so that the data isn't so compressed. You may find it useful to adjust both the max/min scales on the X-axis depending on how much extra trace traffic you took in your original trace.

The pink line represents the length of the data we are sending. For the most part it's a full packet size of 1480. The only reason it dips any lower than that is because SMB sends 61440 bytes of info (0xF000), so the dip at the end is the remainder.

Let's look at the UnACKed data (yellow line). The client continues to send data as fast as it can until it reaches the advertised window size (blue line). Note that since the scales for Window Size and Packet size can be vastly different, window size is put on the right vertical Axis. In this case, however, they happen to line up. This is helpful, because it shows that we keep sending data until the UnACKed data reaches the max advertised window size. Once we get to this point, we have to wait for an ACK from the server before we send more data. So we get in a state where we have to wait for an ACK before sending more data. Then once this particular SMB command completes, another delay occurs while waiting for the SMB response which for the next WRITE command.

Since our RTT is very low, we tend to see little effect of this delay. But note that if we had a larger window size we would have been able to even send more data before waiting for an ACK and thus improve the transfer time.

100 Millisecond RTT, 100 MB

In this sample, I purposely set the Round Trip Time to 100 Milliseconds. The basic effect is that anything that requires a response before it can continue, will incur a 100 millisecond delay. So this will certainly affect the time required to transfer the 500 MB file. As you can see here, a total of about 4 seconds is needed now. Another side affect is that since it's slower, the server is able to keep up with the request so the amount of UnACKed data is generally lower.

100 Millisecond RTT DownStream, 100 MB, 5376 Window Size

In this example we've cut the Windows size down to 5376 bytes. I've zoomed into a small segment of the entire transfer to show the details. The main thing to note is that we cannot fill the pipe up more than two segments. This means the number of segments we have to wait for an ACK for goes up, thus the total time to transfer also goes up. You can see here, for instance that there are many times we have to wait the return ACK before we can send a new packet (about 20 as compared to 5 in the baseline capture).

Down Stream 100 Millisecond RTT, %5 Packet Loss

Now we'll add in some packet loss. In this case the loss is on the Down Stream side, which means that responses from the server never make it to the client. Remember that the client is where the capture is being taken, so the resulting trace won't show any retransmits. Since the server's ACKs are being dropped, the client is affected because he cannot move his sliding window due to the fact that he thinks there is more unACKed data than there truly is. Thus the client has a difficult time sending data at a rate that fills the server's receive window. This is shown in the graph by the slow approach of the UnACKed data to the servers advertised window size.

Down Stream 100 Millisecond RTT, Up Stream %5 Packet Loss

The difference between this graph and the one above it is that in this case the packet loss is on the Up Stream side. So now the client does have to send retransmits, because now the server is complaining that data is missing to the client, instead of the other way around. So now the graph shows up inverted because data to the client can never fill up the window. Instead the client has to resend old data that has been lost. So this resets the UnACKed data to something lower than it was before.

56K Bandwidth Up and Down Stream

The final example is to show the difference when bandwidth is reduced. This is similar to the baseline test, but the main difference is the time line below is much longer. For each SMB Write it takes around 10 seconds. Another difference is that since there is a RTT delay inherent in this connection, you don't see much overlap once we hit the window size. Basically every request has enough of a delay so the we see a acknowledgement before sending the next TCP segment. So we tend to stay right at the Window edge rather than fluctuating, like we do in the baseline test.

TCPPerf Macro Code

You should be able to copy this into Excel's Macro Editor. The one problem I can foresee is that if formatting in your browser causes the text to wrap, Excel will complain. So keep this in mind.

A general overview of this code is that it copies the text from the clipboard and creates some columns to calculate UnACKed data and Advertised Window. We must also recalculate the Seq/Ack columns as there are not represented as numbers (they are numbers with hex in parens). Then we copy all the formulated data to another sheet so we can sort the resulting data. And finally we create a chart for the both sides of the conversation.

Sub TCPPerf()
' TCPPerf Macro

Application.ScreenUpdating = False
Application.Calculation = xlManual

' You can name your sheet different for multiple traces. The resulting
' sheet is created based on this name.
CurSheet = ActiveSheet.Name
' Populate the column headers
[A1].Value = "Frame"
[B1].Value = "Time"
[C1].Value = "TCPSeqData"
[D1].Value = "TCPAckData"
[E1].Value = "Window"
[F1].Value = "Len"
[G1].Value = "ConvID"
[H1].Value = "Source"
[I1].Value = "Dest"
[J1].Value = "Prot"
[K1].Value = "Desc"
[L1].Value = "Seq"
[M1].Value = "Ack"
[N1].Value = "Unack"
[O1].Value = "SrcData"
[P1].Value = "DstData"
  [Q1].Value = "SrcWindow"
[R1].Value = "DstWindow"
  [S1].Value = "AdvertisedWindow"

' Paste In Data from clipboard
Range("A2").Select
ActiveSheet.Paste

' Find Last Row in Data and save in LastRow
Range("A2").Select

Selection.End(xlDown).Select
Dim LastRow As Integer
LastRow = ActiveCell.Row

' Next take the text version of Seq/Ack from NM and convert to a number
Call SeqAckForm(LastRow)

' Create a column for UnAck'd data populate with calculation
Call UnAckData(LastRow)

  ' Create a column that gets the Src/Dest Seq and Advertised Window data
  ' depending on the sender.
Call SrcDestData(LastRow)

' Since we diabled calculations, cause one to occur now before we
' copy data around.
Calculate

' Define the name of the sheet we'll use to build our chart from
DataSheetName = "TCPPerf_" + CurSheet

' Transfer the data and calculated data to a new chart so we can sort.
Call TransferData(LastRow, CurSheet, DataSheetName)

' Build two charts, one for each client/server
Call BuildCharts(LastRow, DataSheetName)
Application.Calculation = xlAutomatic
Application.ScreenUpdating = True

End Sub

Sub SeqAckForm(LastRow)
'
' SeqAckForm Macro
'
' Just cut off the number at the first space and convert to a Number.
[L2].Value = "=VALUE(MID(RC[-9], 1, FIND("" "", RC[-9])))"
Range("L2").Select
Selection.AutoFill Destination:=Range("L2:L" & LastRow), _
Type:=xlFillDefault

[M2].Value = "=VALUE(MID(RC[-9], 1, FIND("" "", RC[-9])))"
Range("M2").Select
Selection.AutoFill Destination:=Range("M2:M" & LastRow), _
Type:=xlFillDefault
End Sub

Sub SrcDestData(LastRow)
' If the Source matches the first source, then take this seq, otherwise
' take the last value found.
[O2].Value = "=IF(H2=H$2, O1, M2)"
Range("O2").Select
Selection.AutoFill Destination:=Range("O2:O" & LastRow), _
Type:=xlFillDefault

' If the Source matches the first dest, then take this seq, otherwise
' take the last value we found.
[P2].Value = "=IF(H2=I$2, IF(P1<>0, P1, L2), M2)"
Range("P2").Select
Selection.AutoFill Destination:=Range("P2:P" & LastRow), _
    Type:=xlFillDefault

' If the Source matches the first source, then take window to be the
' src window, otherwise take the last value we found.
[Q2].Value = "=IF(H2=H$2, Q1, E2)"
Range("Q2").Select
Selection.AutoFill Destination:=Range("Q2:Q" & LastRow), _
    Type:=xlFillDefault

' If the Source matches the first dest, then take window to be the
' src window, otherwise take the last value we found.
[R2].Value = "=IF(H2=I$2, R1, E2)"
Range("R2").Select
Selection.AutoFill Destination:=Range("R2:R" & LastRow), _
Type:=xlFillDefault

' Now get the advertised window size from the other side based on which
' is the source address
[S2].Value = "=IF(H2=H$2, Q2, R2)"
Range("S2").Select
Selection.AutoFill Destination:=Range("S2:S" & LastRow), _
    Type:=xlFillDefault
End Sub

Sub UnAckData(LastRow)
' Calculate the Unack'd data by looking at the seq/ack columns we created
' before and the len field.
[N2].Value = _
    "=IF(IF(H2=H$2,L2+F2-O2,L2+F2-P2)<0,-1,IF(H2=H$2,L2+F2-O2,L2+F2-P2))"
Range("N2").Select
Selection.AutoFill Destination:=Range("N2:N" & LastRow), _
    Type:=xlFillDefault
End Sub

Sub TransferData(LastRow, OriginalSheetName, DataSheetName)
' TransferData Macro
'
' Add a new sheet for new data and charts.
Sheets.Add
ActiveSheet.Name = DataSheetName

' Copy all of the various columns we need from the original sheet.
' This is done so that when we resort the data, we reference the
' formula results.

Sheets(OriginalSheetName).Select
Range("B1:B" & LastRow).Copy
Sheets(DataSheetName).Select
Range("A1").Select
ActiveSheet.Paste

Sheets(OriginalSheetName).Select
Range("F1:F" & LastRow).Copy
Sheets(DataSheetName).Select
Range("C1").Select
ActiveSheet.Paste

Sheets(OriginalSheetName).Select
  Range("N1:N" & LastRow).Copy
Sheets(DataSheetName).Select
Range("D1").Select
Selection.PasteSpecial Paste:=xlPasteValues, Operation:=xlNone, _
    SkipBlanks:=False, Transpose:=False

Sheets(OriginalSheetName).Select
Range("H1:H" & LastRow).Copy
Sheets(DataSheetName).Select
Range("E1").Select
ActiveSheet.Paste

Sheets(OriginalSheetName).Select
Range("L1:M" & LastRow).Copy
Sheets(DataSheetName).Select
Range("F1").Select
Selection.PasteSpecial Paste:=xlPasteValues, Operation:=xlNone, _
    SkipBlanks:=False, Transpose:=False

Sheets(OriginalSheetName).Select
Range("A1:A" & LastRow).Copy
Sheets(DataSheetName).Select
Range("H1").Select
ActiveSheet.Paste

Sheets(OriginalSheetName).Select
Range("S1:S" & LastRow).Copy
Sheets(DataSheetName).Select
Range("B1").Select
Selection.PasteSpecial Paste:=xlPasteValues, Operation:=xlNone, _
    SkipBlanks:=False, Transpose:=False

' Now sort this data by Source so we can split and create 2 charts.
Range("A1:I" & LastRow).Select
Selection.Sort Key1:=Range("E2"), Order1:=xlAscending, Header:=xlGuess, _
    OrderCustom:=1, MatchCase:=False, Orientation:=xlTopToBottom, _
DataOption1:=xlSortNormal
End Sub

Sub BuildCharts(LastRow, MainSheetName)
' Find where the source changes to destination.
Range("A1:G1").Copy

For Each Cell In Range("E2:E" & LastRow)
    If Cell.Value <> [E2] Then
      Exit For
    End If

    Cell.Select
Next

' Paste in the original column headers
CurCellRow = Selection.Row + 1
Range(CurCellRow & ":" & CurCellRow).Insert Shift:=xlDown

' Create two names for each chart based on the the senders name.
ChartName1 = Range("E" & (CurCellRow - 1)).Value
ChartName2 = Range("E" & (CurCellRow + 2)).Value
Range("A1:D" & CurCellRow).Select

' Now create a chart for each address.
Call Chart(1, CurCellRow - 1, ChartName1, MainSheetName)
Call Chart(CurCellRow, LastRow + 1, ChartName2, MainSheetName)
End Sub

Sub Chart(FirstRow, LastRow, Name, MainSheetName)
' Set the range of the data we will use for our chart.
MinScale = Range(MainSheetName & "!A" & FirstRow + 1).Value
Sheets.Add

' Create a Name for this chart and set teh active sheet to that name.
TopUsersChart1Sheet = "Chart_" + Name + CurSheet
ActiveSheet.Name = TopUsersChart1Sheet

' Add in a new chart.
Charts.Add
ActiveChart.Location Where:=xlLocationAsObject, Name:=TopUsersChart1Sheet
ActiveChart.ChartType = xlXYScatterLines
ActiveChart.SetSourceData Source:= _
    Range(MainSheetName & "!$A" & FirstRow & ":$D" & LastRow)

' Create a secondary access
ActiveChart.SeriesCollection(1).AxisGroup = 2

' Set the min/max scale based on the time in the first/last row.
ActiveChart.Axes(xlCategory).MinimumScale = _
    Range(MainSheetName & "!A" & FirstRow + 1).Value
ActiveChart.Axes(xlCategory).MaximumScale = _
    Range(MainSheetName & "!A" & LastRow).Value

' Make the min Y scale -1
ActiveChart.Axes(xlValue).MinimumScale = -1

' Make the long time lables slant a bit.
ActiveChart.Axes(xlCategory).TickLabels.Orientation = -25
End Sub

And Much More…

The general idea here is that you can take the TCP data and look at it in a graphic form to help you to see at a high level if there's a problem. There are so many other things that one could add. Resets could be added in to visually indicate those in the graph. You could also create other graphs to represent the seq/ack responses that would give you another view of the data. But hopefully this will give you a simple tool and some specific filters to spot check performance issues on your network.

Part 1: Poor Man’s Expert Using Excel – Top Users

PaulELong — Fri, 01 Dec 2006 01:36:00 GMT

One item we are evaluating for future versions of Network Monitor are experts. These are the tools that among other things, allow you to get a high level view of a problem. But using a simple new feature in the released version of NM3 will allow us to do some simple expert-like analysis.

Cut and Paste of Multiple Frame Summaries

NM3 allows you to select multiple frames, then right-click those frames and store the resulting output in the clipboard buffer. The resulting data is tab delimited and based on the columns you have showing in your frame summary. This is the key to creating simple experts with Excel. This TAB delimited data can be imported into Excel and manipulated to pull the information we want to summarize.

The Top Users Expert

The idea of this simple expert is to display the chattiest machines on your network. So it basically sorts the Source/Destination/Protocol fields and provides a count of each. This expert provides the top users that send and receive data. It also shows a protocol usage, but only at the top most level. So while this may not be as useful as a count of all protocols used, this still may help show where most of your bandwidth is being used protocol wise.

This expert can be useful to find a machine that is infected with a virus and attempting to pass that virus on to another machine. It can also identify a machine that is overloaded with requests and thus the center of a bottle neck.

How it works

You start by selecting the frames you want in NM3. Then right click those frames and select Copy. You could also just type Ctrl+C to copy these frames into the clipboard. Next you run Excel and paste the data, starting at line number 2. This is so we can add column headers for the data. Then you create 3 pivot tables, one each for Source/Destination/Protocol. For each pivot table, you select the appropriate column for the row data and row count. Once you do this you can sort the data, and the count of the highest users appears at the top.

Devil's in the Details

So rather than explaining each step, I'll annotate a Macro I created to do this job. If you cut and paste this macro into Excel and run, it will copy the data in the clipboard, and create the column labels, and then create a new page with the 3 pivot tables. NOTE: the data is assumed to be in the default layout (Frame Number, Time Offset, ConvID, Source, Destination, Protocol Name, Description). If you have a different layout, you'll have to modify the columns section in this macro.

This macro was created for Excel 2003. It should be easy to modify for future versions as well. In Excel 2003, I go to Tools, Macros. Then I typed in a name, TopUsers, and hit create. This gives me a blank macro, which you can populate with the following information.

Sub TopUsers()
' This creates the text for each column heading
' Match these to your column layout in NM3.
[A1].Value = "Frame"
[B1].Value = "Time"
[C1].Value = "ConvID"
[D1].Value = "Source"
[E1].Value = "Dest"
[F1].Value = "Protocol Name"
[G1].Value = "Description"

' Now make Cell A2 the current one and Past the clipboard results into Excel.
Range("A2").Select
ActiveSheet.Paste

' Get the range of all the data you've pasted in.
Dim CapRange As Range
Set CapRange = ActiveSheet.UsedRange

' You can name your sheet different for multiple traces. The resulting
' sheet is created based on this name.
CurSheet = ActiveSheet.Name

' Add a new sheet and define a variable for its name. Set it to the active sheet.
Sheets.Add

TopUsersSheet = "TopUsers_" + CurSheet
ActiveSheet.Name = TopUsersSheet

' Create Source Pivot Table
ActiveWorkbook.PivotCaches.Add(SourceType:=xlDatabase, SourceData:= _
    CapRange).CreatePivotTable _
    TableDestination:=TopUsersSheet + "!R3C1", TableName:="Source"

' Add Count of Source as xlCount of this Pivot Table.
ActiveSheet.PivotTables("Source").AddDataField ActiveSheet.PivotTables( _
    "Source").PivotFields("Source"), "Count of Source", xlCount

' Add Source as xlRowField of this Pivot Table.
With ActiveSheet.PivotTables("Source").PivotFields("Source")
    .Orientation = xlRowField
    .Position = 1
End With

' Sort data based on the Count.
ActiveSheet.PivotTables("Source").PivotFields("Source").AutoSort _
    xlDescending, "Count of Source"

' Create Dest Pivot Table, same as Source
ActiveWorkbook.PivotCaches.Add(SourceType:=xlDatabase, SourceData:= _
    CapRange).CreatePivotTable _
    TableDestination:=TopUsersSheet + "!R3C4", TableName:="Dest"

ActiveSheet.PivotTables("Dest").AddDataField ActiveSheet.PivotTables( _
    "Dest").PivotFields("Dest"), "Count of Dest", xlCount

With ActiveSheet.PivotTables("Dest").PivotFields("Dest")
    .Orientation = xlRowField
    .Position = 1
End With

ActiveSheet.PivotTables("Dest").PivotFields("Dest").AutoSort _
    xlDescending, "Count of Dest"

' Create Prot Pivot Table Same as Source

ActiveWorkbook.PivotCaches.Add(SourceType:=xlDatabase, SourceData:= _
    CapRange).CreatePivotTable _
    TableDestination:=TopUsersSheet + "!R3C7", TableName:="Protocols"

ActiveSheet.PivotTables("Protocols").AddDataField ActiveSheet.PivotTables( _
    "Protocols").PivotFields("Protocol Name"), "Count of Prot", xlCount

With ActiveSheet.PivotTables("Protocols").PivotFields("Protocol Name")
    .Orientation = xlRowField
    .Position = 1
End With

ActiveSheet.PivotTables("Protocols").PivotFields("Protocol Name").AutoSort _
    xlDescending, "Count of Prot"

' Label each of our tables.
[A2].Value = "Source Addresses"
[D2].Value = "Destination Addresses"
[G2].Value = "Top Protocols, highest level only"
End Sub

Adding a Button for Flare

To make things even easier to use, I added a button on the main page. All it does is call the TopUsers macro. So you just copy the data in NM3, and click the button in Excel. Magically, the new sheet is populated with the table showing you the top users.

So to do this, I insert a picture. In my case I used an autoshape that looks like a rounded rectangle. But any picture will do, as long as you can assign a macro to it. To assign the macro, I just right clicked the shape and chose, Assign Macro. I also added some text to my rectangle like, "Copy Frame Summary in NM3 and then Click Here".

More to Come

While this is a simplistic expert, it should be possible to create more complicated ones. NM3 allows you to add columns to the interface. The columns you can choose are based on properties exposed the by the parsers. And you can go one step further by adding properties to the parsers. Stay tuned for a more complex example!

NPL – The Power Behind the Parsers

PaulELong — Thu, 05 Oct 2006 00:15:00 GMT

I always found it frustrating to use an application, only to find something didn’t work exactly like I expected it to. And in the world of network sniffers, it seems that I always find a problem with a particular protocol or some output that I wish displayed a little differently. With Network Monitor 3.0, you have the ability to modify the source which describes the protocols! This in turn allows you to control what shows up in the frame summary and frame details. And furthermore, allows you to define what is available for columns in the frame summary. Did chills just go up your spine?

The NPL language resembles C++ mixed with IDL’ish (interface description language). For more complicated changes, this may not be for the faint of heart. But as you might be a protocol geek, like me, the leap from understanding how protocols work to how NPL works may not be that big a stretch. And for making simple changes, like summary line modifications, I think it’s in the grasp of most network Admins.

So what can NPL do for me?

While I don’t intend this Blog entry to be a “How to Program in NPL in 5 minutes” I do want to give you an idea of the power in NPL, and perhaps peak your interest. In future Blogs I do plan to highlight some of the details of NPL, but this will just be a taste.

So here is a list of simple modifications you can make by changing the NPLs

· Fix a parser so it displays the network data correctly

· Create a new parser for your proprietary protocol

· Modify the summary description for your specific needs

· Add a new available column with some aggregated data

· Change the port a particular protocol uses

A common problem: “No silly, we do HTTP traffic on port 8888, not 80 or 8080!”

While changing port mappings for protocols could be something revealed in the user interface, we haven’t gotten that far in Network Monitor 3.0 yet. I expect we should address this specific problem on different fronts, i.e. a UI for each protocol, and some way to handle dynamic port allocations. And there are also some heuristics we can use to identify protocols as well. But today, there is a fairly simple way to modify the NPL script for protocols on non-standard ports.

NPL borrows a construct from C called a switch statement. And while it has an overloaded syntax, the basics are the same. A switch statement works by taking a specific data field and then “switching” on different cases depending on the data field’s value. So if the value is 100 you do one thing and if it’s 200 you do something else.

In the context of our problem, making HTTP traffic appear on port 8888, the switch statement is used to decide which protocol to use based on the TCP port number defined. So as you may imagine, TCP.NPL has a switch statement telling it what protocol to use next based on the protocol number defined in the TCP layer.

So here it is, well part of it:

switch(Port)

{

case 20:

case 21:

FTP Ftp;

case 23:

TELNET Telnet;

…

case 80:

case 8080:

Struct

{

Switch

{

Case AsciiString( FrameData, FrameOffset, 4 ) == "COPY":

Case AsciiString( FrameData, FrameOffset, 4 ) == "LOCK":

…

OK, now I know you are probably saying, FTP looks straight forward and so does Telnet, but the stuff below ports 80 and 8080 is starting to look hairy. Well, for our purposes we really don’t care about the info in the case statement. All we are going to do is add another port in the list after 80 and 8080. And here it is…

case 80:

case 8080:

case 8888:

Struct

{

The way the case statement works is that if any of the values (80, 8080, or 8888) match; it will execute the code directly underneath it until the next case. Technically in NPL you only have one statement per set of cases, and in this case the “Struct”.

Once you make the change and save TCP.NPL, just go to the Tools Menu and select Reload Parsers. Once the reload finishes you can reload your trace and now any traffic on port 8888 can be parsed as HTTP.

Can I get a column to display the TCP Sequence Number?

You can probably create a column to display anything you want. You simply define a property in NPL, and it will be available to you in the UI in the column chooser once you rebuild the parsers. A property is simply a variable that is not part of the traced data. So in this case we are going to create a property that stores the TCP sequence number.

[MyNewTCPSeqNumber]

UINT32 SequenceNumber;

The “UINT32 SequenceNumber;” was already there so all we added was the “[MyNewTCPSeqNumber]” line. The way the “[ ]” bracket section works it that it always applies the line right below/next to it. And you can enter multiple statements separated by commas. In this simple case you type a property name and the value of the property is set based on how SequenceNumber is displayed. The display of SequenceNumber is based on the default for UINT32, but I’ll leave the details of that for some other discussion.

So, you could get even fancier by using FormatString.

[MyNewTCPSeqNumber = FormatString(“Seq = %d”, SequenceNumber)]

UINT32 SequenceNumber;

Since the column name is already MyNewSequenceNumber, this is a bit of overkill, but I wanted to show you a simple example of FormatString. This way you can contrive any kind of complex data you want for your column data.

In TCP.NPL we already have defined a property called “TCPSeqNumber” for. But feel free to change this to something else, or add another property by using a comma to separate them.

[

MyNewTCPSeqNumber,

TCPSeqNumber

]

UINT32 SequenceNumber;

Conclusion

This should give you a feeling for the power that NPL can pack. If your interest is peaked and you want to learn more, you can start with the ParserLanguage.doc included with Network Monitor 3.0. I’m sure folks will find interesting ways to extend our parsers, as well as write their own. In fact there’s no reason you couldn’t write a parser for something other than network data, given you could get your data in the Network Monitor file format. The possibilities are endless.