MessageAnalyzer

Network Capture is Dead!

Paul E Long — Mon, 04 Mar 2013 21:48:27 GMT

Long Live Capture! In this continuing series of differences between Message Analyzer and Network Monitor, we’ll explore the trace capture experience. I reference the iconic cliché because Message Analyzer (note not Network Analyzer) is about exploring any kind of structured message data. Capturing falls in line because we don’t just capture network traffic, but any kind of ETW (Event Tracing for Windows) events. Let’s explore.

Event Tracing for Windows

ETW has been around forever, well at least since Windows 2000. This is the standard event messaging system built into the OS used by components to provide diagnostic info. So, Message Analyzer is a kind of stethoscope for the Windows OS. While Message Analyzer only works on Windows 7 and above, the plan is to be able to read any ETL log. Certainly there are other ways of capturing this data, but our focus is to provide simple templates for capturing that a user can modify, save and share. And also, that you can watch this data live as it happens.

Going back to the main point, ETW providers populate the right side which are registered with Windows. Selecting a provider gives you provider specific configuration which varies. I won’t go into details here but instead offer our Network Trace Scenario help documentation which has lots of great info. But you can add as many providers as you want to a scenario, configure it how you want and save it. And once you save the scenario, it appears in your Trace Scenario templates.

NDIS, Firewall, HTTP Proxy Provider in Box

I mentioned you can the list all of providers installed. There are ones for USB, Bluetooth, and many others that are not network specific. These are here whether you install Message Analyzer or not. However, we do install 3 of our own providers. The Network Monitor way of capturing was through an NDIS Filter driver. We’ve upgraded this, in a sense, to provide ETW messages for Message Analyzer. You can see this by expanding the stack, which is done by clicking on the blue or green cubes on the left hand side of the row. This shows multiple ETW fragment messages which form into our NdsiProvider message. And above that comes your typical network stack. (If you don’t already know, you can also explore this stack in full by clicking on the ‘+’ icon on the left.)

The Firewall is another great place you can capture from now. It provides a different inspection point into the stack. At this layer IPSec traffic is decrypted and you have access to Loop Back traffic. If you have a SQL server and client on the same machine, you can now capture that traffic!

Additionally, we updated the Firewall capture scenario so show when a message has be discarded. With this and a provider to list out all the firewall rules, you can now understand when the firewall is involved in blocking your traffic. And of course we’ve wrapped this up into a new scenario template you can select.

HTTP Proxy is the third provider. It can capture HTTPS traffic from your browser. This makes a very efficient and lean HTTP capture machine.

You’ll notice that even visually, as you move up the stack, you capture less information. The closer you get to the source, the more efficient you capturing becomes. Your computer has already done the heavy lifting of ‘parsing’ the message to get it to the application, why have us do it again?

Another unique thing about our providers is that we expose some advanced, non ETW, settings. For instance, each of the providers support filtering that is done before we parse. The NDIS provider can be configured to look for a specific IP address. The Web Proxy provider can look at a specific HOST. This type of filter is much quicker because it’s one quick check in the provider, rather than the parsing involved when a trace filter is used. So this allows for a high performance way to filter out data on busy machines.

Advanced Configuration

Finally there are some advanced configurations that relate to the ETW engine. If you find that you are dropping messages (which is not reported in Beta 2) you can change the buffer settings. ETW documentation might help more in this regard, but at this point I just want to point out it exists.

Mix and Match

So once you get the hang of various providers, you can combine them together. You can get all the data in one session and then use Grouping, Quick Filtering or an alternate viewer to see what you want and how it’s connected. Then save your trace scenario and share it with your colleagues.

Of course you can still add a Trace Filter (previously called a Capture Filter), which throws out traffic that doesn’t match. A major difference here is that the message numbering still increments for those that are not captured. If you have a filter of UDP and there are 5 UDP message, then 20 TCP messages, the UDP message that follows will have a message number of 26.

Starting another New Session

So, now you want to start another capture session? Today when you enter the back stage page from a running session, we default to showing you the current session configuration. You can press the arrow next to the session info to show other sessions or start a new one.

Evolution of Capture

We want to make capturing traces easier. And with all the new streams of data ETW provides, won’t it be wonderful to configure a scenario for a more novice user, and then share the trace scenario with them? Also, by targeting the data you need, you put less stress on the machine and result in a trace which is more compact. There’s still some more interesting work to be done here, but we are off to a great start.

Microsoft Message Analyzer Beta 2 is released (build 5950)!

Paul E Long — Fri, 15 Feb 2013 15:52:00 GMT

Install the Beta 2 version from here: https://connect.microsoft.com/site216. You’ll need to be a member of our connection.

This release adds a range of new functionality and resolves a number of bugs:

· IntelliSense UI for filter creation – As one of the most requested features, Filter IntelliSense is now available for exploring protocol message hierarchies to find the fields you need to build filter expressions. The capabilities are vastly improved compared to Network Monitor, now displaying protocols, messages, fields, structures, properties, annotations and more!

· Quick filter - Quick filtering makes it easy to create a time window in which to view trace results! Unlike BSV, it filters messages in memory after loading them instead of during import. Just select the traces you want, adjust the time slider as needed, and you are done. It’s that easy.

· Capture firewall discard events – This feature allows you to discover how the firewall is affecting network traffic. New messages tell you when traffic is blocked and associated IDs point to the specific firewall rule responsible for dropping the message.

· OPN Viewer – You can right click on any field and select Go to Definition to view the field’s OPN definition. This feature provides the equivalent functionality of the NPL Viewer in Network Monitor 3.4.

· Parsing REST Protocols – This feature enables you to diagnose and analyze RESTful web services. RESTful web services are one of the fastest growing network areas.

· Performance improvements:

o Message Analyzer startup time has improved by over 50%.

o Sorting on selected column has improved by 60%.

o Grouping has improved by 30%

o Parsing after the initial load has improved by up to 15%, depending upon the protocol type.

Message Analyzer also presents exciting graphic viewer features that are still under development, but we would like to share them with you now to get your initial feedback:

· Gantt viewer – Do you need to see a bird’s eye view of your message traffic? Message Analyzer now includes a highly customizable Gantt Viewer that provides easy-to-use navigation, zooming, and the ability to drill down into further details, as necessary.

· Console viewer – provides an interactive command-line interface for filtering, sorting, grouping, and viewing messages collections.

Microsoft is providing the Beta 2 release to give you the opportunity to provide feedback so we can continue to make improvements on the new features as we drive towards our upcoming RTM.

Please install Message Analyzer Beta 2, take it for a spin, and send us your feedback!

Pivoting on Trace Data Using Grouping

Paul E Long — Fri, 25 Jan 2013 21:45:17 GMT

The single most versatile feature in Message Analyzer is “Grouping”. It’s basically a replacement for the conversation tree where the conversation tree is just one kind of grouping. Slicing of data has never been better. It’s like your Ginsu knife deal just got better.

Grouping by Module

A great first example is grouping by Module. Just right click the module column in summary grid and select Group.

Each group is displayed in the analysis grid with a count of the matching messages in the parenthesis. If you have multiple levels of groups, the number indicates the number of subgroups. At the top, you’ll notice that a grouping box has been added for Module. You can remove it with the red X, or move it to change the order of grouping. This view has conveniently collated the traffic.

And since we reassemble and associate request/response as operations, the list is concise and complete. The TCP group will contain all the TCP specific handshake stuff and unidentified traffic, but none of the SMB2 related fragments. Keep in mind that the Module is determined by the top level. This is a general rule about columns as there is a tree underneath with many different values. So even though there might be TCP in the tree, the top level takes precedence and is displayed.

Where’s the Conversation View?

This question has two answers, because it’s really two questions. One could be, where’s that tree control on the left side? The other more precise question is, how can you dice data like the Network Monitor 3.x conversation tree?

As for the control, we are working on it. The new embedded tree has some advantages, like it takes up less space. But when you want to see related traffic and drive traffic from the tree the separate control is better. For now the control is still on the design table.

For the second part of the question, providing a grouping that represents the conversation view is easy, though with some differences. We can map the Process ID/Network/Transport type view by using the ProcessId field of ETW and some properties we’ve created to expose the Network and Transport conversations as strings. The process ID is buried down in the ETW layer, where all messages from our new providers start. By right clicking and selecting ProcessId, you can quickly add it as a grouping.

For the Network and Provider properties, you have to go to the Column Chooser. In fact you need to add them as columns first, and then right click and add as grouping. In the future I’m sure we can remove some steps.

Here’s the Network property which exists at multiple modules:

And here is the Transport property, again in multiple modules:

One difference concerning the Network/Transport properties and Network Monitor conversations is that the properties don’t define the hierarchy. They only provide a string to describe the port definition. Also there is no conversation ID anymore. Also, if there is tunneled traffic, the last property wins again. So only the top layer is exposed.

Once you are finished, you can right click each Grouping box and collapse all.

Then you can start expanding Processes and Network parents to see the structure.

Another huge benefit of the tree being in the grid is that filtering now affects the tree. How many times have you wanted the tree to be filtered? No longer do you need a sharp eye to pick out a specific IPV4 address and related TCP connections. Now the grouping tree is shown based on the current filter so you can apply an IPv4.Address==192.168.1.13 and see only parts of the tree that involve that single client address.

Changing Group Order

As I mentioned previously you can move groupings around. Select and drag a grouping box to another location and re-pivot your data.

Transport is at the end:

Transport moved to the middle:

Group by Anything

And now, this is where you should go out and play with grouping. Group by Diagnosis and see how many messages are affected by a diagnosis and what kinds there are. Group by destination or source and see who is getting the largest cut of the messages. Group by HTTP.ContentType and see types of objects being requested by your browser. And group by *FileName, (SMB2.FileName and SMB.FIleName), to see what traffic is associated with which file for SMB traffic. And of course you can save your groups by using Manage Columns, “save column layout as…”, which includes your groupings. Let grouping become a normal part of your analysis and embrace the power of this new feature.

Filtering with Message Analyzer

Paul E Long — Thu, 10 Jan 2013 19:55:03 GMT

I want to point out some differences between Message Analyzer and Network Monitor in a short series of blogs. In this first one, let’s discuss filtering. There are some nice shortcuts that make it easier to type in filters and some differences from Network Monitor. Our goal with Message Analyzer was to keep the flexibility and power user’s expect of filter creation, while providing more help and shortcuts. So while there are certainly differences with how filters are created, the premise is the same.

Filtering Shortcuts

With a better Intellisense still being implemented on our side, there are some discovery issues for these features. However, you can still take advantage these features which I’ve arranged below as a chart. Keep in mind that there is an example in the filter library for most of these.

Example	Feature	Description
TCP.Rst	Partial Paths	TCP.Flags.Rst is the formal full path. But you only have to put the first and last entity name to make a valid, yet more inclusive filter. This might not be as fast as a more precise filter, but the extent of which depends on the filter.
*port==135	Wildcards	You can also place a wild card before an element to filter any message which matched “port”. In this case UDP and TCP will be included, as well as any entity called port. It’s also important to note that port is a special way to reference both the source and destination. Again this is not as precise and performs worse than the example above.
*address==FF-FF-FF-FF-FF-FF	Wildcards	This is similar to above, but an important point is that what is on the left side of address might not be obvious. The hardware address could come from wireless or Ethernet. An IP address might be via the firewall or NDIS driver. The IPv4 layer doesn’t exist in the firewall driver, though there is an IPv4 type address there. Remember that Address is the IPv4 type, but IPv4 is the protocol/module name. This is really important when creating capture filters because IPv4.Address in the firewall will not match. Also important is that the right side can take on any valid literal form. So if you enter an IPv4 address on the right side, we will find those address that match that can return an IPv4 literal type. Again, the address is a special way to reference both the source and destination addresses.
HTTP.Uri contains “msn” TCP contains “Microsoft”	Contains	You can use contains to search a string or all the raw data. You can also search a specific protocol for a string. You can control case sensitivity, but by default it’s case insensitive. For instance ‘TCP contains “bLah” caseSensitive’ would search for “bLah” matching case. Look at the library to see other options for encoding types.
HTTP.Headers[“Host”] == “www.msn.com”	Maps	A big difference in Messages Analyzer is that we created a new type to handle HTTP data. Using a Map is a better data type to describe HTTP and others, but this makes the filtering the syntax different. An advantage is that we can now reference the HTTP fields using the exact match of the HTTP field name. Previously we had to remove dashes which could be confused for subtraction. For instance in Network Monitor 3.x, Content-Type was written as HTTP.Response.ContentType. But now you can Type HTTP.Response.Headers[“Content-Type”] which is more predictable and consistent. On the other hand we group HTTP request and responses as operations. And for an operation we also expose fields which lift some information from the request or response. For example HTTP.ContentType = “jpeg”. Intellisense will make this easier to explore.
Tcp.port != 80 SMB.FileName ~= “”	Existence	This is a Dusey J A feature where math and practicality collide. If you’ve used Network Monitor a lot, you’ll know there’s a difference between tcp.port != 80 and !(tcp.port == 80). This mathematical irony seems to tickle the OCD in all of us. See below for more detailed info.

Existence Example: TCP.Port != 3389 vs. SMB.FileName ~= “”

There are two possible outcomes a user wants. However, a confused user might try and use the same filter for two different situations.

Example 1: Getting Rid of Traffic

One example is that you want to get rid of TCP traffic by a port, commonly used to remove noise, like RDP traffic. A user could mistakenly type TCP.Port != 3389, for instance. But as you see below, the result is not correct. With Network Monitor we automatically assume you want to include only TCP traffic. Behind the scenes we apply a filter like (TCP && TCP. Port != 3389).

For Network Monitor, if you wanted to get rid of traffic, you type “!(tcp.Port == 3389)”. And of course this could be confusing since these two filters seem mathematically equivalent.

With Message Analyzer, the two statements are equivalent. And we no longer assume you want to look for traffic where the TCP.Port fields exists. Tcp.port != 3389 is the same as !(tcp.port == 3389).

Example 2: Displaying Data where the Field Exists

For this example I want to find messages where the SMB File Name is not blank. With Network Monitor, the default action is to only return those fields with the FileName field already there. So something like Property.SMBFileName != “”, returns what you want.

However with Message Analyzer, we don’t assume to check for existence and the result is not exactly what we want. In this example you see we return other messages like the TCP session setup. The filter is more explicit without the check for existence and since TCP doesn’t even have a matching entity, the message matches the filter because there it is true that the smb2.filename is not blank.

Now, we use a new operator, “~=”, which will additionally check for existence. You can see below that we’ve limited the capture to SMB traffic, because in only these messages does the filename entity exists.

Rather than continuing to violate mathematical laws, and to reduce our OCD medicine intake, we’ve added this new operator, ~=. This includes the check for existence and doesn’t return messages which don’t have the field in the first place. We believe this more explicit notation will provide less confusion and more clarity moving forward.

Right Click Add as Filter

We understand this feature doesn’t work perfectly now and we are working to implement it completely. In future versions, the experience for right clicking something in the UI and adding as a filter should be improved. But still, even in this current implementation, it is a helpful way to learn how to specify entities to filter on.

More to Come

There are even more complex things you can do with the filtering language that I will leave to a later post. Hopefully with these small tips, your filtering experience will be better. Please remember to provide us any feedback on our Connect site where we are hosting the Message Analyzer Beta.

Meet the successor to Microsoft Network Monitor!

Paul E Long — Tue, 18 Sep 2012 00:45:13 GMT

It’s a very exciting week for me and my team! This week I’m attending the SNIA SDC 2012 conference in Santa Clara, CA and this is where we will announce Message Analyzer. There are so many new features and aspects to discuss, but for now I’ll leave you with the official announcement:

Microsoft Message Analyzer has been released to the public, available here:

https://connect.microsoft.com/site216 (you’ll have to join the Message Analyzer and Network Monitor program to see the downloads and access other parts of or our site.)

As you might guess from the name, Message Analyzer is much more than a network sniffer or packet tracing tool. Key capabilities include:

Integrated "live" event and message capture at various system levels and endpoints
Parsing and validation of protocol messages and sequences
Automatic parsing of event messages described by ETW manifests
Summarized grid display – top level is “operations”, (requests matched with responses)
User controlled "on the fly" grouping by message attributes
Ability to browse for logs of different types (.cap, .etl, .txt) and import them together
Automatic re-assembly and ability to render payloads
Ability to import text logs, parsing them into key element/value pairs
Support for “Trace Scenarios” (one or more message providers, filters, and views)

We are providing this beta release to give you an opportunity to let us know what you like and don’t like and where we need to focus our energy as we drive towards a mid-2013 RTM date.

Please install, take it for a spin, and send us your thoughts! There are “Report Issue” and “Community” buttons built into the ribbon, and we have a new blog here: http://blogs.technet.com/messageanalyzer.

(To capture at the NDIS and Firewall layers without running as admin, you must log off and back on after installation to pick up the necessary credentials. Please do this!)

Have a ball!

[update: adding a picture]

Coming Soon!!!

Paul E Long — Sat, 15 Sep 2012 21:17:20 GMT

Some things brewing in here…come back soon!