Network Monitor : NPL

Windows 7 and ISA Remote Windows Sockets Parsers Available

PaulELong — Thu, 04 Jun 2009 19:17:00 GMT

If you don't already know, we have been updating the our parsers for Network Monitor on http://www.CodePlex.com/NMParsers every month. Most recently we have updated the Windows parser set to support Windows 7 protocol updates. In the June parser release on CodePlex we have support for Remote Windows Sockets (RWS) protocol, which is used to proxy TCP and UDP traffic from Winsock applications. So now with the new parser set you can decode this traffic into the upper level protocols that ride on top of RWS.

These parsers rely on Network Monitor 3.3, so please upgrade first if you haven't already. Please visit NMParsers on CodePlex and download the last parser set so you can get the most up to date parsing experience. Enjoy!

SMB2 Parser for NM3.1

PaulELong — Tue, 06 May 2008 22:12:00 GMT

We have decided to release an SMB2 parser for Network Monitor 3.1 (released July 07) to hold people over untill the beta for Network Monitor 3.2 releases in early June.

Where can I get the SMB2 parser?

You can download SMB2.NPL parser, along with SPARSER.NPL, CER.NPL, FCCS.NPL, SCNA.NPL and SMB.NPL (all supporting parsers) on http://connect.microsoft.com under the Network Monitor 3 project. If you’ve already signed up you’ll see it as one of your active projects. If you need to sign up you will need to create a passport account and join our project. Once you are in on the Network Monitor 3 project, click on the Downloads link on the left. You will see SMB2 Parser as one of the selections.

How do I use the new SMB2 parser?

Look at the article on using the SSL parser (http://blogs.technet.com/netmon/archive/2007/10/23/new-ssl-public-parser-available-how-to-deal-with-new-parsers.aspx) in the sections “Where do I stick it?” and “Working with NPL Parser path”. The instructions for installing the SMB2 parsers are the same.

Happy SMB2 parsing!

New SSL Public Parser Available: How to deal with new parsers

PaulELong — Tue, 23 Oct 2007 23:39:02 GMT

You’ve made some changes to your parsers and want to keep things organized. With the release of SSL.NPL, which we will also talk about today, it makes sense to understand how to work with parsers when you want to update or modify them. So I’d like to go over some strategies for keeping out of NPL H-E-double hockey sticks. If you are not familiar with this term, don’t worry, it is just a synonym for a type of file management night mare. It’s happened before, so we have learned some lessons and will attempt to help, but sometimes…some things…are in the user’s hands.

Where do I get it?

The SSL.NPL parser is available on our connect.microsoft.com site. http://Connect.microsoft.com is where we host our Netmon community. You can keep up to date, file bugs, and ask questions on our new forum (it’s still a newsgroup, but there’s now a web interface). From the connect.microsoft.com site, you should be able to get to the Network Monitor 3 project if you’ve already signed up. If not, just create a new account and then join the Network Monitor 3 project. Once you are on our site, you can click on the Downloads link on the left side. This will show you a list of files, one of which is the SSL.NPL parser.

Where do I stick it?

You could just plop the SSL.NPL file right on top of the old one in C:\program files\Microsoft Network Monitor 3\NPL, and things will work just fine potentially for the rest of you and your computer’s life. But there is a potential problem lurking.

In this case it’s the next update that may cause a problem. Not so much if we send out a new SSL.NPL, but more if you decide to change a parser yourself. It’s called forward integration, and it makes things complex. In this situation you have to figure out the difference you made and integrate into the new SSL parser, or vice versa if you prefer. Either way it’s an issue that not easy to resolve automatically and does require some user intervention.

But even in the replacing the SSL with new one, there are problems. On Vista, we should not be writing to the Program Files directory. So saving a file there using NM3 could end up shadowing it to some other directory (AppData\Local\VirtualStore\Program Files\Microsoft Network Monitor 3\NPL). So if you ever have a situation where you think you are updating the correct version of SSL.NPL in the NM3 editor, you may not be touching the invisible one. This has also caused problem in the past when trying to update from NM3.0. In any case we hope to address this for NM3.2.

Working with NPL Parser path

The solution is to store your NPLs in your own personal NPL directory using the Tools, Options dialog on the parsers tab. In NM3, you have the ability create a parser load order. This is a list of directories that are searched in order for each parser we encounter. When an NPL file includes another, we search each path in order and load the first NPL we encounter that matches. Even if the file is local to that NPL file, we still use the search order path.

We do create a personal NPL directory (documents\NM3\NPL), but it doesn’t automatically get added to your parser path. So it will search your local directory first and then each path listed in the options dialog in order (top to bottom). Add your local version first and then add the directory for versions of MS supplied files and finally add in the “C:\program files\Microsoft Network Monitor 3\NPL” directory so it searches that directory last. It won’t search the Program Files directory automatically once you add at least one directory in parser search path.

Once you’ve modified your path and added the new version of SSL.NPL, you’ll either need to restart NM3.1, or rebuild the parsers from the parser tab. Since we’ve already plugged a stub version of SSL into public build, no other modifications are need in the parsers to call the new SSL parser. In the case of a new parser, however, you would have to add this to sparser.npl.

This technique helps solve the problem by making sure you don’t clobber the original files. If we update a file, you can also check your local changes and forward integrate our changes in to your version. And if it’s Vista, we can make sure the versions of MS updates end up in an obvious place. Forward integration is a whole other topic, but there are difference tools which display file differences and make it easy to take changes from one file into another. But for simple changes this can be done by hand.

Improvements in NM3.2

There are things we can do in the future to make this more transparent to the user. And working more cooperatively with Vista is one of our goals for the next version. Fortunately this is not something most of us do all the time.

Using Columns and Properties

PaulELong — Thu, 01 Mar 2007 20:26:00 GMT

You might have had an occasion to add a new column in Network Monitor 3.0. But the list of available choices might be quite daunting. What you may not know, however, is that the list is derived from properties in the NPL script that makes up each parser. This means you can add any kind of information you want as a column.

This blog will attempt to show you some of the more useful columns you can add. But we will also explore the creation of properties in NPL for more advanced analysis.

The Skinny on Columns

It’s probably important to give some background about how columns work in NM3. A common misconception is that the column information is actually the data, but in reality it’s a string describing the data. That means that what’s in the Source Column is not the hex data for 192.168.0.1, but a string that describes it.

This is important because a property is just a variable in NPL that can change based on other factors. For instance the Source column can display the Hardware Address, IPv4 Address, or IPv4 Address based on the frame in question. But in each case, the property is still just a string representation.

Time Related Properties

For starters, let’s display what the default column layout is.

By default, NM3 show “Time Offset” as one of its columns. But it’s often useful to see either the Time of Day or the Time Delta (time from last packet). Well both of these items are available for addition. To add these in, right mouse click any of the column headers and select “Choose Columns…”.

Note that you can also save the current column layout as the default for new sessions, or restore the default column layout. Once you select the menu item, you will see a dialog like the one below.

On the left is a list of column headers you can add, and on the right, your currently enabled column headers. To add a column, select the column on the left and hit add. Or to remove, click on an item on the right and select remove. You can also order the columns using the Move Up/Move Down buttons. It is also possible to move the columns around from the Frame Summary by drag and dropping the columns around, so you don’t need to go to the Choose Frame Summary Columns dialog to do this.

Finding the Column to Add

The first problem you may encounter is that the list is huge, which is why I want to highlight some of the more useful headers you can add without NPL modification. But you still have to find the column, which you could do by just scrolling through the list. But there is a faster way! If you start typing a few of the letters, it should get you to that section more quickly. So, in this example I want to show you how to add the “Time of Day” and “Time Delta” columns. So first click in the Disabled Columns portion of the dialog, and start typing “T I m e” without the spaces, and you see that the items that start with Time appear. The first one you can see is “Time Delta”, so let’s add this to the list by clicking the “Add” button. Then also click on the “Time Of Day” and add it.

Once you click OK from the dialog, you should now see the new columns appear. The “Time Of Day” column will show you the time when the frame was captured. The “Time Delta” displays the time from the last frame. This is updated when a filter is applied, so if frames 1 and 2 are 1 second apart, and frames 2 and 2 are 1 second apart, a filter which only shows frames 1 and 3 will display a time delta of 2 seconds. For example given the following data:

Time Of Day	Time Delta	Frame	Time Offset
15:41:35.331	0.000000	1	0.000000
15:41:36.331	1.000000	2	1.000000
15:41:37.331	1.000000	3	2.000000

If filter that includes frames 2 and 3 are added, the resulting Rime Delta column would look as follows.

Time Of Day	Time Delta	Frame	Time Offset
15:41:35.331	0.000000	1	0.000000
15:41:37.331	2.000000	3	2.000000

Note: If you try to filter on the property for Time Delta in NM3.0, say to find all frames with a delta greater than 1 second, you won’t return any frames. We currently don’t have a way to properly apply this property in a filter. We hope to address this in future version of Network Monitor.

Other Useful Properties

So I wanted to provide a list of other useful properties and their descriptions. This should provide you a reference of some of the more popular column entries.

TCPDescription

It’s often useful to always see the TCP Description rather than the upper protocols. By default, NM3 shows the highest level protocol in the Frame Summary Description field. So by adding the TCPDescription property to the columns, you can always see this information. This is very useful for t-shooting strictly TCP problems.

Other Protocol Description Fields

As with TCP, you may want to see other protocol header rather than other upper level protocols. For instance SMBDescription or HTTPDescription, could be added as columns.

TCPAckNumber, TCPSequenceRange, and TCPFlags

These three properties are also great for t-shooting TCP issues. You may actually find this more useful than the TCP Description as the column information lines up the ACK/SEQ number for easy reference.

SourceNetworkAddress and DestinationNetworkAddress

The way the Source/Destination columns are populated today, you don’t always see the real IPv4 or IPv6 address. Instead we show this based on the following rules. If a higher numbered item exists, we show that item.

Alias for the IP address
Resolved Name of IP address
Real IP address
Hardware Address

So in cases where you’d like to see the real IP address and a resolved name exists, turning off the aliases doesn’t show you the real IP address. So you view the real IP address all the time by adding these columns.

SourceHardwareAddress and DestinationHardwareAddress

Like the IP address, the Source and Destination Hardware addresses don’t always show up because of the same rules above. So adding these columns will allow you to always see the Hardware Addresses.

Creating Your Own Properties

On occasion you might find it useful to create your own properties for viewing as a column in NM3. There may be a frame summary data field that you want to show up as a column. You might also want to combine a few data fields and create a column that combines that information. So let’s create a few example properties to show you how this is done.

IPv4 Identification: Adding just a simple property

In this case we’ll create a property for IPv4’s Identification field. This is the field that uniquely tags an IPv4 packet and can be used to associate the same frame from two different captures. We’ll start by opening ipv4.npl. In NM3, you can do this in the parsers tab by searching for the file in the Parser Files in the Object View. Once you find IPv4.npl, double click it to edit. Next, search for the field called Identification, it should look like this:

UINT16 TotalLength;
UINT16 Identification;

UINT16 FragmentFlags
{
…

Now we add in a property by placing it in square brackets above the data field we want to create a property for.

UINT16 TotalLength;
[IPv4Identification]
UINT16 Identification;

UINT16 FragmentFlags
{
…

Now save your changed NPL, and reload the parser. From the menu select Tools, Reload parsers. Once the build is complete you can load a trace with IPv4 traffic and add the column you just created.

TCP Port Pair: Adding a derived property with two data fields

In this example, we are going to get a bit fancier. We will create a property that shows both the source and destination port in Hex, along with the friendly port name. Remember that since a property is just a string, we can set it to anything we want. The property does not have to only represent one value.

So in this case we will edit TCP.NPL like we did IPv4.NPL before and look for this section of code.

UINT16 SrcPort = PortNameTable(this);
[Pair = Port, DestinationPort]
UINT16 DstPort = PortNameTable(this);

We’ll want to add in our property right before the last data field we will reference. In this case there is already a bracketed section above DstPort, but these directive sections can have multiple lines separated by a comma. So we will change this to

UINT16 SrcPort = PortNameTable(this);
[
    Pair = Port, DestinationPort,
    TcpPortDesc=FormatString(“%d (%s) -> %d (%s)”, 
    SrcPort, 
    PortNameTable(SrcPort),
    DestPort,
    PortNameTable(DstPort))
]
UINT16 DstPort = PortNameTable(this);

As you see we’ve called our new property TcpPortDesc and we are using FormatString, which is like printf in C, to format the way our property will look. We have 4 parameters for FormatString, the first two are the Source port and friendly name, and the second two are for the destination. We use PortNameTable to convert the number of the port to a descriptive string. PortNameTable is just a table that has already been defined for you in GlobalTables.NPL. Tables are simple structures that given one value, we convert and return a different value.

Once you add in this new column, as described before, you will see a column that displays as follows:

0x50 (HTTP(80)) -> 0xC0E6 (49382)

TCPPayload Info: Adding a property to display TCP payload as ASCII

Often you want to look at the TCP data as ASCII. For FTP conversations, the text reads like a story. But we must also work around a bug in NM3 which limits the text in a field to 0x103 bytes. So what we’ll do is limit the text when we collect it.

Here’s the original section of TCP.NPL we are going to modify.

//This maybe useful for TCP payload data filter for all TCP based protocols
//A possible filter maybe: Property.TCPPayload.Contains("*****")
//[TCPPayload = AsciiString(frameData, offset, TCPPayloadLength)]
[
DataFieldFrameLength = frameOffset + TCPPayloadLength,

In this case the statement for the entire payload is commented out. You can see that this property may be useful to enable if you want to search TCP Payloads using something like

Property.TCPPayload.Contains(“Anything”)

So we are going to enable this property and add another. Let’s change the code to this:

//This maybe useful for TCP payload data filter for all TCP based protocols
//A possible filter maybe: Property.TCPPayload.Contains("*****")
[
TCPPayload = AsciiString(FrameData, FrameOffset, TCPPayloadLength),
TCPPayloadDisp = AsciiString(FrameData, FrameOffset, 0x103),
DataFieldFrameLength = frameOffset + TCPPayloadLength,

This adds in two new properties. The first, TCPPayload, will allow us to do searching like described above. The first parameter is the data we are going to search in. When you use FrameData, it means the current frame information. The second parameter is the offset in the buffer, and another global property FrameOffset points to the current location in the frame. The final parameter is the length, and the property TCPPayloadLength holds that value for TCP.

But we can’t use TCPPayload for display because if the length is greater than 0x103 bytes, it won’t get displayed at all. For TCPPayloadDisp we fix this by explicitly setting the length.

It’s All About What You Want to See

Using columns for displaying useful information can save you time and help you t-shoot problems. Perhaps in future version of NM3 we’ll have a way to define standard column setups so you can switch from you TCP T-shooting setup to your SMB t-shooting column setup. Also we do plan to provide some organization to the properties to make it easier to find things in that long list.

NPL – The Power Behind the Parsers

PaulELong — Thu, 05 Oct 2006 00:15:00 GMT

I always found it frustrating to use an application, only to find something didn’t work exactly like I expected it to. And in the world of network sniffers, it seems that I always find a problem with a particular protocol or some output that I wish displayed a little differently. With Network Monitor 3.0, you have the ability to modify the source which describes the protocols! This in turn allows you to control what shows up in the frame summary and frame details. And furthermore, allows you to define what is available for columns in the frame summary. Did chills just go up your spine?

The NPL language resembles C++ mixed with IDL’ish (interface description language). For more complicated changes, this may not be for the faint of heart. But as you might be a protocol geek, like me, the leap from understanding how protocols work to how NPL works may not be that big a stretch. And for making simple changes, like summary line modifications, I think it’s in the grasp of most network Admins.

So what can NPL do for me?

While I don’t intend this Blog entry to be a “How to Program in NPL in 5 minutes” I do want to give you an idea of the power in NPL, and perhaps peak your interest. In future Blogs I do plan to highlight some of the details of NPL, but this will just be a taste.

So here is a list of simple modifications you can make by changing the NPLs

· Fix a parser so it displays the network data correctly

· Create a new parser for your proprietary protocol

· Modify the summary description for your specific needs

· Add a new available column with some aggregated data

· Change the port a particular protocol uses

A common problem: “No silly, we do HTTP traffic on port 8888, not 80 or 8080!”

While changing port mappings for protocols could be something revealed in the user interface, we haven’t gotten that far in Network Monitor 3.0 yet. I expect we should address this specific problem on different fronts, i.e. a UI for each protocol, and some way to handle dynamic port allocations. And there are also some heuristics we can use to identify protocols as well. But today, there is a fairly simple way to modify the NPL script for protocols on non-standard ports.

NPL borrows a construct from C called a switch statement. And while it has an overloaded syntax, the basics are the same. A switch statement works by taking a specific data field and then “switching” on different cases depending on the data field’s value. So if the value is 100 you do one thing and if it’s 200 you do something else.

In the context of our problem, making HTTP traffic appear on port 8888, the switch statement is used to decide which protocol to use based on the TCP port number defined. So as you may imagine, TCP.NPL has a switch statement telling it what protocol to use next based on the protocol number defined in the TCP layer.

So here it is, well part of it:

switch(Port)

{

case 20:

case 21:

FTP Ftp;

case 23:

TELNET Telnet;

…

case 80:

case 8080:

Struct

{

Switch

{

Case AsciiString( FrameData, FrameOffset, 4 ) == "COPY":

Case AsciiString( FrameData, FrameOffset, 4 ) == "LOCK":

…

OK, now I know you are probably saying, FTP looks straight forward and so does Telnet, but the stuff below ports 80 and 8080 is starting to look hairy. Well, for our purposes we really don’t care about the info in the case statement. All we are going to do is add another port in the list after 80 and 8080. And here it is…

case 80:

case 8080:

case 8888:

Struct

{

The way the case statement works is that if any of the values (80, 8080, or 8888) match; it will execute the code directly underneath it until the next case. Technically in NPL you only have one statement per set of cases, and in this case the “Struct”.

Once you make the change and save TCP.NPL, just go to the Tools Menu and select Reload Parsers. Once the reload finishes you can reload your trace and now any traffic on port 8888 can be parsed as HTTP.

Can I get a column to display the TCP Sequence Number?

You can probably create a column to display anything you want. You simply define a property in NPL, and it will be available to you in the UI in the column chooser once you rebuild the parsers. A property is simply a variable that is not part of the traced data. So in this case we are going to create a property that stores the TCP sequence number.

[MyNewTCPSeqNumber]

UINT32 SequenceNumber;

The “UINT32 SequenceNumber;” was already there so all we added was the “[MyNewTCPSeqNumber]” line. The way the “[ ]” bracket section works it that it always applies the line right below/next to it. And you can enter multiple statements separated by commas. In this simple case you type a property name and the value of the property is set based on how SequenceNumber is displayed. The display of SequenceNumber is based on the default for UINT32, but I’ll leave the details of that for some other discussion.

So, you could get even fancier by using FormatString.

[MyNewTCPSeqNumber = FormatString(“Seq = %d”, SequenceNumber)]

UINT32 SequenceNumber;

Since the column name is already MyNewSequenceNumber, this is a bit of overkill, but I wanted to show you a simple example of FormatString. This way you can contrive any kind of complex data you want for your column data.

In TCP.NPL we already have defined a property called “TCPSeqNumber” for. But feel free to change this to something else, or add another property by using a comma to separate them.

[

MyNewTCPSeqNumber,

TCPSeqNumber

]

UINT32 SequenceNumber;

Conclusion

This should give you a feeling for the power that NPL can pack. If your interest is peaked and you want to learn more, you can start with the ParserLanguage.doc included with Network Monitor 3.0. I’m sure folks will find interesting ways to extend our parsers, as well as write their own. In fact there’s no reason you couldn’t write a parser for something other than network data, given you could get your data in the Network Monitor file format. The possibilities are endless.