by Ning Kuang and Cheng Chang

As the successor to Network Monitor, Microsoft Message Analyzer (MMA) provides new parsing capabilities with superior enhancements. With the use of the new Open Protocol Notation (OPN) language upon which the parsers are built, Message Analyzer dramatically extends protocol modeling capabilities, network traffic diagnostics, and the analysis features of Network Monitor. In this blog, we introduce some of the most important parsing improvements since Network Monitor, which includes performance enhancements, automatic reassembly, and protocol validation and diagnosis.

Parser Coverage

The Message Analyzer release is provided with 175 pre-written OPN Parsers. This enables Message Analyzer to parse most of the core public networking protocols and commonly used Microsoft protocols. The parser list is included in the OPN Parser Package Contents Addendum in the Message Analyzer Operating Guide.

More Information

To learn more about how to manage OPN Parser packages, see Managing Microsoft OPN Parser Packages.

ETW Event Parsing

In addition to the pre-written OPN Parsers, Message Analyzer can automatically generate parsers for manifest-based Event Tracing for Windows (ETW) components that generate ETW events. This means that approximately 700 additional parsers can be automatically generated, which greatly expands the parser base. This capability enables you to analyze events from Windows components that are instrumented with ETW. Further information about parsing ETW events will be provided in a future blog.

More Information

To learn more about ETW events and providers, see the ETW Framework Tutorial and System ETW Provider Configuration Settings in the Message Analyzer Operating Guide on TechNet.

To learn more about how ETW event parsers are created, see the Event Traces section of the Message Analyzer Tutorial on TechNet.

Requesting New Parsers

Microsoft is continuing to develop and release new OPN Parsers on a regular basis. These will be made available periodically through the download service that you can access from the Message Analyzer Start Page. However, if you have a protocol for which you would like to request a new parser, please submit a request here. You will need a Microsoft account to log in to the Connect site to make your request.

More Information

To learn more about how to synchronize your existing OPN Parser packages for automatic updates, see Managing Microsoft OPN Parser Packages in the Message Analyzer Operating Guide on TechNet.

Parser Groups

Parser grouping in Message Analyzer is a new approach that facilitates the logical organization of parser families. If Message Analyzer has fewer parsers loaded, it usually means a smaller footprint. This enables you to improve performance and efficiency by having the flexibility to retain only the parser groups that you require for the troubleshooting task at hand, while removing those that you do not require. When you do need the removed parsers, you can simply reinstall them from the Message Analyzer Start page.

OPN Parser Packages

For ease of management, parsers are grouped into different packages that relate to specific areas of functionality, such as applications, devices, communications, support, and so on. These packages are described in following table.

Name

Description

Example

Core Networking

Contains parsers for most commonly used public network protocols.

This is the basic foundation for protocol parsing functionality and most other parsers depend on this group.

DHCP.opn, DNS.opn, HTTP.opn, IPV4.opn, IPV6.opn, SMTP.opn, SSL.opn, TCP.opn, UDP.opn, Wifi.opn;

Infrastructure

This group does not contain actual parsers. It Contains modules that provide support functions, enumerations and types for all other parsers and modules.

DTYP.opn, ERREF.opn, WindowsReference.opn

Device and Log File

Contains device traffic and text log file parsers.

(The .config file contains information on how to parse the log files)

usb3.opn, AzureAppFabric.config,

Cluster.config,

SambaSysLog.config

Public

Contains parsers for public protocols (not included in the core networking group)

FTP.opn, iSCSI.opn,

NFS.opn, RMCP.opn

Microsoft Common

Contains parsers for common Microsoft and public protocols.

Microsoft protocols depend on this group.

LDAP.opn, KerberosV5.opn,

AuthIP.opn, DCOM.opn

Microsoft File Sharing

Contains parsers for Microsoft Windows file sharing and branch cache protocols.

FSRVP.opn, SMB.opn, SMB2.opn

Microsoft Remote Desktop

Contains parsers for Microsoft Windows Remote Desktop communication and management protocols.

RDPBCGR.opn, RDPEDYC.opn , RDPEGFX.opn, RDPEGT.opn

Microsoft Identity and Security

Contains parsers for Microsoft Windows identity, authentication, authorization, and security protocols.

NRPC.opn, SAMR.opn

Microsoft Other

Contains parsers for other, less common Microsoft Windows protocols, to support special requirements.

XCEP.opn, WKST,opn

A detail list of OPN Parser groups and their contents is specified in the OPN Parser Package Contents Addendum on TechNet.

Choosing Parser Packages

If you use Message Analyzer with certain groups of OPN Parsers only, you can remove any unnecessary parser packages for better performance. To remove an OPN Parser package, click the Settings tab on the Message Analyzer Start Page, scroll down to the OPN Parsers section, identify the unneeded package, and then click the X icon to the right of the package to Uninstall this OPN package. The Delete OPN package message then displays and prompts you for the deletion, as shown in the following figure.

clip_image002[4]

To reinstall an OPN Parser package, restart Message Analyzer and then click the Downloads tab on the Start Page. On the Downloads tab, identify the package you want to download, and then click the server download icon to the right of the package. In the Item Download Options dialog that displays, select the Download once and don’t automatically update option and then click OK to start the download.

Note Due to a known issue, you might need to restart Message Analyzer in order to use this package after downloading. Also, there are dependencies between OPN Parser groups. When you uninstall an OPN Parser package upon which other OPN Parser packages depend, you are prompted to also uninstall the dependent OPN Parser packages. For details, see http://technet.microsoft.com/en-us/library/dn281833.aspx

OPN Parser Language

Parsers are developed with the OPN language. OPN is a domain-specific language that enables developers to describe protocol data, architecture, and behavior. OPN is a more expressive and powerful language with greater capabilities than the Netmon Parser Language (NPL). It therefore enables developers to create more powerful and complete parsers.

One of the key differences with respect to NPL is that OPN can simulate the behavior of a protocol by using full-state models. This not only enables the automatic validation of message format and data constraints, but also allows developers to determine if a sequence of messages is conforming to the specified behavior of a protocol. With the OPN language, you have the ability to describe and implement reassembly functions, grouping messages into higher-level operations, and you can track data constraints that span across time.

More Information

To learn more about the OPN Language, download the OPN Programming Guide v1.

Automatic Reassembly

While a Network Monitor user can choose to reassemble manually, Message Analyzer can support automatic reassembly on both live and static (archived/imported) traces, with OPN’s power to simulate protocol behavior.

Reassembly usually applies to transport protocols such as TCP, IPv4, and IPv6, but can also apply to HTTP and SMB2 messages. When data is transferred on a network, it can be fragmented and rebuilt by the Transport and Application layers, which can make for difficult reading. To make the data easy to understand, Message Analyzer reassembles the network traffic by putting the packets back together and displaying the entire packet in the right order.

Note: For TCP reassembly, if retransmits happen at the same time, you may find discontinuous frames reassembled to a packet and a single frame with a diagnosis message following it.

For example, you might have TCP traffic in a Network Monitor trace that looks like the following:

clip_image004[4]

When Message Analyzer reassembles similarly fragmented TCP traffic, it will look like the following, with fragments reassembled and included within an expandable top-level message row:

clip_image006[4]

Note that the default message number order might be different when comparing Message Analyzer to Network Monitor. In Network Monitor, all message numbers are sequential, as they align with the network transporting sequence; and this makes message fragment association a little more difficult. In Message Analyzer, the display order is adjusted to align with the protocol communication sequence, making it easier to understand protocol sequences through the association of message fragments with a particular source message.

Validation and Diagnosis

Because OPN can simulate protocol behavior, Message Analyzer can diagnose network and protocol-related error issues. In addition to these capabilities, OPN enables Message Analyzer to validate whether message field values and sequences of a particular protocol align with its definition, as indicated by the protocol’s technical specification.

Diagnosis Messages

Message Analyzer can detect error conditions and display visualizations in top-level Analysis Grid messages, where you can click an error icon to display inline descriptions that contain meaningful error information. Each diagnosis message comes with a DiagnosisType and a DiagnosisLevel, as well as a description of the error issues.

There are four DiagnosisTypes that are represented in the Analysis Grid with different icons, as follows:

· Application – can indicate a network communication issue.

· Validation – indicates that messages do not align with their protocol definitions.

· Parsing – indicates a parsing failure when invalid data is decoded.

· InsufficientData – indicates that data was lost, for example, when Message Analyzer is grouping messages as an operation or when performing data reassembly.

There a three DiagnosisLevel indications that can apply to each DiagnosisType. These are used to specify error severity and include the Error, Warning and Information levels.

Diagnosis messages can display in both Analysis Grid message rows or in the Diagnostics tool window. The following is an example of the four different diagnosis message types, as displayed in the Diagnostics window:

clip_image008[4]

More Information

To learn more about diagnosis messages, including how to filter for them in Message Analyzer, see the Diagnosis Category section in Filtering Trace Results.

Operations

Message Analyzer can group related message sequences such as a request/response pair, as a single operation encapsulated in a top-level message row of the Analysis Grid, even if the pair is not contiguously retrieved by Message Analyzer. Because grouping enables you to see request and response messages encapsulated in a single message row as an operation, Message Analyzer improves your data analysis process and helps you to quickly assess your data at a high level.

To undo the encapsulation of messages in operations and return them to their original chronological sequence, you can click the Hide Operations button in the Viewpoints group on the Ribbon of the Message Analyzer Home tab, to display the operation messages in their original chronological order, similar to the way they are displayed in Network Monitor.

For example, see the SMB2 operations that are indicated by the Analysis Grid message rows with blue-cubed icons in the following figure:

clip_image010[4]

Note: Operations usually apply to application layer protocols that communicate through the pairing of request and response messages; for example HTTP, SMB, DNS, LDAP, RPC, and others. Similar to reassembly, an operation helps you to better visualize protocol sequences and logic by collapsing the message pairs into a single expandable top-level message row in the Analysis Grid. However, in most cases, this results in the message numbers displaying non-sequentially. Because Message Analyzer attempts to display the protocol logical sequence rather than the original network traffic sequence, your data analysis perspective is enhanced.

More Information

To learn more about Operations and Viewpoints, see Applying and Managing Viewpoints and Viewpoints: OSI Model and APSTNDP.

Encrypted Traffic Visibility

Parsing encrypted traffic is always a challenge. However, by leveraging ETW to provide inspection points that capture at the Firewall and HTTP Proxy layers, Message Analyzer can intercept encrypted traffic. With these capabilities, protocols such as HTTPS, IPsec, and SMB/SMB2 can be correctly decoded.

Conclusion

Message Analyzer provides broad protocol parsing coverage with ETW support and enables you to achieve better performance with OPN Parser package management. With the new and powerful OPN language at the core of Message Analyzer, it offers a more accurate and comprehensive parsing and diagnosis experience, with features such as automatic reassembly, validation, and operations.

We are committed to bringing you a more powerful and accurate parsing experience. We would like to hear from you about your experiences, concerns, and issues! Please visit and join the Microsoft Connect site and provide us with your feedback.

More Information

To learn more about how OPN supports Message Analyzer in the PEF architecture, see the PEF Architecture Tutorial in the Message Analyzer Operating Guide on TechNet.