Long Live Capture! In this continuing series of differences between Message Analyzer and Network Monitor, we’ll explore the trace capture experience. I reference the iconic cliché because Message Analyzer (note not Network Analyzer) is about exploring any kind of structured message data. Capturing falls in line because we don’t just capture network traffic, but any kind of ETW (Event Tracing for Windows) events. Let’s explore.
ETW has been around forever, well at least since Windows 2000. This is the standard event messaging system built into the OS used by components to provide diagnostic info. So, Message Analyzer is a kind of stethoscope for the Windows OS. While Message Analyzer only works on Windows 7 and above, the plan is to be able to read any ETL log. Certainly there are other ways of capturing this data, but our focus is to provide simple templates for capturing that a user can modify, save and share. And also, that you can watch this data live as it happens.
Going back to the main point, ETW providers populate the right side which are registered with Windows. Selecting a provider gives you provider specific configuration which varies. I won’t go into details here but instead offer our Network Trace Scenario help documentation which has lots of great info. But you can add as many providers as you want to a scenario, configure it how you want and save it. And once you save the scenario, it appears in your Trace Scenario templates.
I mentioned you can the list all of providers installed. There are ones for USB, Bluetooth, and many others that are not network specific. These are here whether you install Message Analyzer or not. However, we do install 3 of our own providers. The Network Monitor way of capturing was through an NDIS Filter driver. We’ve upgraded this, in a sense, to provide ETW messages for Message Analyzer. You can see this by expanding the stack, which is done by clicking on the blue or green cubes on the left hand side of the row. This shows multiple ETW fragment messages which form into our NdsiProvider message. And above that comes your typical network stack. (If you don’t already know, you can also explore this stack in full by clicking on the ‘+’ icon on the left.)
The Firewall is another great place you can capture from now. It provides a different inspection point into the stack. At this layer IPSec traffic is decrypted and you have access to Loop Back traffic. If you have a SQL server and client on the same machine, you can now capture that traffic!
Additionally, we updated the Firewall capture scenario to show when a message has been discarded. With this and a provider to list out all the firewall rules, you can now understand when the firewall is involved in blocking your traffic. And of course we’ve wrapped this up into a new scenario template you can select.
HTTP Proxy is the third provider. It can capture HTTPS traffic from your browser. This makes a very efficient and lean HTTP capture machine.
You’ll notice that even visually, as you move up the stack, you capture less information. The closer you get to the source, the more efficient you capturing becomes. Your computer has already done the heavy lifting of ‘parsing’ the message to get it to the application, why have us do it again?
Another unique thing about our providers is that we expose some advanced, non ETW, settings. For instance, each of the providers support filtering that is done before we parse. The NDIS provider can be configured to look for a specific IP address. The Web Proxy provider can look at a specific HOST. This type of filter is much quicker because it’s one quick check in the provider, rather than the parsing involved when a trace filter is used. So this allows for a high performance way to filter out data on busy machines.
Finally there are some advanced configurations that relate to the ETW engine. If you find that you are dropping messages (which is not reported in Beta 2) you can change the buffer settings. ETW documentation might help more in this regard, but at this point I just want to point out it exists.
So once you get the hang of various providers, you can combine them together. You can get all the data in one session and then use Grouping, Quick Filtering or an alternate viewer to see what you want and how it’s connected. Then save your trace scenario and share it with your colleagues.
Of course you can still add a Trace Filter (previously called a Capture Filter), which throws out traffic that doesn’t match. A major difference here is that the message numbering still increments for those that are not captured. If you have a filter of UDP and there are 5 UDP message, then 20 TCP messages, the UDP message that follows will have a message number of 26.
So, now you want to start another capture session? Today when you enter the back stage page from a running session, we default to showing you the current session configuration. You can press the arrow next to the session info to show other sessions or start a new one.
We want to make capturing traces easier. And with all the new streams of data ETW provides, won’t it be wonderful to configure a scenario for a more novice user, and then share the trace scenario with them? Also, by targeting the data you need, you put less stress on the machine and result in a trace which is more compact. There’s still some more interesting work to be done here, but we are off to a great start.
Amaziiiiiiiiiiiiiiing work, so good that few person know it yet, it is like I have X-Ray vision while everybody is in the fog.
Thanks to all the team, this is life changer.
Amazing. I am going to install it today and try it out with different event providers. This is going to change everything related to tracing in windows.
Very good post! Wasn't aware of such functionalities.
This week it's the second time I discover network capture tools for newbie, after Debookee a network capture for iPhone and mobiles. It's a really good news for non-techie people who will be able to look at the network side without fears of jumping into bits and bytes.
Check out the Message Analyzer Usage Scenario documentation in the TechNet Library for lots of supportive information in using this awesome tool !!
Hey Paul, looks great!
However, when I go to download at connect.microsoft.com/.../Downloads, all I see are test suites and Netmon parsers. Did you remove the MA download?
Concur with the last comment - no MA download visible anywhere?
any update on the download issue? is anyone monitoring this board?
Any updates to NMApi in the future? It seems this tool uses the same library as Network Monitor.
Baron, whiteknight, ronvo,
Sorry for not replying earlier. As Paul mentioned in a previous post, you'll need to join our connection to see the download. Please check out the Directory and make sure you are part of the "Message Analyzer, Network Monitor and Protocol Test Suites" Product. There are several different products you could join which will present different downloads.
You should be able to see the MA download afterwards. Sorry for the inconvenience!
Looks awesome, but I would hesitate in saying that “Network Capture is Dead”, when indeed it is one of the most powerful methods of troubleshooting. Now I will say that Network Monitor was a very weak tool compared to something like Wireshark, and adding the features you have to Message Analyzer certainly will help troubleshoot on a specific system. But sometimes you may still need a Network Capture, actually taken on the network, to point you in the right direction.
Long Live Network Capture! :)
Just to be clear, the statement "Network Capture is Dead" is an allusion to the Who's Long Live Rock. It supposed to mean a re-inventing of term Network Capture. So Network Capture is not going away, but rather what we know as capturing is changed by the way we include more things and more ways to capture with Message Analyzer.
Sorry if there is confusing by the blog title.
Hi Paul, I'm currently troubleshooting some network issues on a hyper-v cluster and teaming. Is there any documentation at which level the NDIS capture filter sets in? I need to capture the traffic at the physical NIC level and be sure what I see is what the NIC sees without any chance some other NDIS driver or the vSwitch sets in to drop or change packets.