Making sense of Exchange Logs using ExLogAnalyzer

Early 2008 we have posted a blog entry with a VB script that generates some pre-canned reports that are based on message tracking logs. The script has proven to be useful in understanding Microsoft's Exchange work load and guide some design decision for Exchange 2010. This script was developed by Todd Luttinen, Principal Program Manager at Microsoft.

During the development of Exchange 2010, we needed to extended our log analysis beyond just message tracking and to answer a variety of questions that assist with design decisions. This exposed a bottle neck with having a single script that has all the parsing and analyzers bundled together.

This resulted in the creation of ExLogAnalyzer by Victor Boctor, Principal Architect at Microsoft. ExLogAnalyzer was developed in C# with the following goals:

• Separation of syntax and semantics.
• Multi-Server support (process log files that span multiple servers). Log events across servers are processed in chronological order.
• Multi-Log Type support (process / cross reference logs of different log types to produce a single report). Log events across log types are processed in chronological order.
• Provide an extensibility model to support rapid development and distribution of extensions (to support new log types) and analyzers (to encapsulate reporting logic).
• Ability for the community to develop their own analyzers or even extensions.
• Support for Exchange 2007 / 2010 log types.

The main shift in this model, compared to the previous script, is that ExLogAnalyzer is built as a framework that can be used to analyze Exchange as well as possibly any other log format. New log types are supported via plugins called "extensions". Extensions are responsible for doing all the parsing and converting of log lines into events, where each event triggers a method and passes all the pre-parsed information as the event arguments. The specific reports are also implemented as plugins known as "analyzers", where each analyzer handles the events it is interested in and does the appropriate accounting and report generation (typically in CSV format). Implementing each analyzer in isolation (rather than one script that answers multiple questions) makes it much simpler to develop, understand and distribute such analyzers. Such extensions and analyzers can also be easily shared given the plugin model. The following simple diagram summarizes the architecture of this tool:

The ExLogAnalyzer is now released to the community with the following extensions / analyzers available out of the box:

• Message Tracking Log
• MsgTrkTopSendersByDeliverLogAnalyzer - Generates the top 1000 senders based on mailbox deliveries. Messages to the internet are not counted.
• MsgTrkTopSendersBySubmitLogAnalyzer - Provides an analysis of the sender load distribution based on number of messages sent from their mailboxes.
• MsgTrkTopRecipientLogAnalyzer - Generates the top 1000 recipients based on mailbox deliveries. Messages to the internet are not counted.
• MsgTrkMessageSizeDistributionLogAnalyzer - Provides an understanding of the message size distribution.
• MsgTrkRecipientNotFoundLogAnalyzer - Discover and summarize recipients for which "Recipient Not Found" error was generated.
• MsgTrkMailflowVisualizerLogAnalyzer - Generates a directed graph showing the server being analyzed and all the inbound / outbound mail flow paths.
• MsgTrkComponentLatencyPercentileLogAnalyzer (E14) - Analyzes the latencies of the different components and determines the latencies experienced by the specified percentiles of messages.
• MsgTrkDuplicateDeliveryLogAnalyzer - Analyzes the sources for duplicate deliveries to Store. Note that end users don't see such duplicates.
• MsgTrkEventFrequencyLogAnalyzer - Provides an understanding of the distribution of the event + source combinations.
• MsgTrkEventTimeDistributionLogAnalyzer - Provides an understanding of the event distribution over time with a per hour resolution.
• MsgTrkExpandLogAnalyzer - Analyzes the distribution list expansion load on the system.
• MsgTrkReceiveLogAnalyzer - Analyzes the distribution of the sources for the messages received by a server or a set of servers.
• Smtp Receive Log
• SmtpReceiveWorkLoadLogAnalyzer - Analyzes the SMTP receive work load over time while tracking tarpitting, client time outs, etc.
• SmtpReceiveDelayedAckLogAnalyzer (E14) - Analysis of delayed ack performance over time. This report provides an overview of the redundancy that is achieved for legacy systems via delayed ack.
• SmtpReceiveFormatterLogAnalyzer - Re-writes the logs with each session in a separate file, it also reformats the log so that the common session information is included in the header, hence, making the session details more readable.
• SmtpReceiveSeparatorLogAnalyzer - Re-writes the logs with each session in a separate file while maintaining the exact log format.
• SmtpReceiveSessionIndexLogAnalyzer - Provides a summary of all sessions processed within the provided logs.
• Connectivity Log
• ConnectivityWorkLoadLogAnalyzer - An analyzer that samples the connections over time. This analyzer generates a CSV file per source (e.g. SMTP or MAPI).
• ConnectivityStatsLogAnalyzer - An analyzer that provides the frequency of sessions, failed and DNS failures per source + destination combination.
• ConnectivityFormatterLogAnalyzer - Re-writes the sessions as a file per session, moved all the common session information to the header to make the sessions more readable.

Sample Reports

Following are some samples to provide a feel of the outputs of some of these analyzers.

Mail Flow Visualizer (demonstrated possible visualization using directed graphs):

Message Size Distribution:

SmtpReceiveFormatterLog (log re-writing for splitting sessions and making them more readable):

# Session Id: 08CBDCECE3DDF231
# Start Time (local): 2009-07-28T11:07:46.922
# End Time (local): 2009-07-28T11:07:46.953
# Start Time (UTC): 2009-07-28T18:07:46.922Z
# End Time (UTC): 2009-07-28T18:07:46.953Z
# Disconnect Type: Local
# Connector Id: MyServer\MyServer_CrossForest
# Local End Point: 157.54.7.153:25
# Remote End Point: 157.54.71.39:4183

0000000,+,,
0000000,*,None,Set Session Permissions
0000000,*,SMTPSubmit SMTPAcceptAnyRecipient SMTPAcceptAuthenticationFlag SMTPAcceptAnySender SMTPAcceptAuthoritativeDomainSender BypassAntiSpam BypassMessageSizeLimit SMTPAcceptEXCH50 AcceptRoutingHeaders AcceptForestHeaders AcceptOrganizationHeaders SMTPAcceptXShadow,Set Session Permissions
0000000,>,220 MyServer E14 Cross Forest,
0000000,<,EHLO otherhost.otherforest.microsoft.com,
0000000,>,250-MyServer.redmond.corp.contoso.com Hello [157.54.71.39],
0000000,>,250-SIZE 10485760,
0000000,>,250-PIPELINING,
0000000,>,250-DSN,
0000000,>,250-ENHANCEDSTATUSCODES,
0000000,>,250-AUTH,
0000000,>,250-8BITMIME,
0000000,>,250-BINARYMIME,
0000000,>,250-CHUNKING,
0000000,>,250-XEXCH50,
0000000,>,250 XSHADOW,
0000000,<,XSHADOW 3333YTkxYjEtYzE1OC00NDcxLWI4OTktMDA2NDI5YmVmZWRlQFRLNUVYMTRNTFRXNjUxLndpbmdyb3VwLndpbmRlcGxveS5udGRldi5taWNyb3NvZnQuY39t,
0000000,>,250 q7rdaFIdKk3NNRTbjRsjrQ==,
0000000,<,MAIL FROM:<sender@contoso.com> SIZE=25477 XSHADOW=70136df4-c89b-4700-9654-b642c4eb78bb,
0000000,*,08CBDCECE3DDF231;2009-07-28T18:07:46.922Z;1,receiving message
0000000,<,RCPT TO:<receiver@contoso.com> ORCPT=rfc822;receiver2@contoso.com,
0000000,>,250 2.1.0 Sender OK,
0000000,>,250 2.1.5 Recipient OK,
0000000,<,XEXCH50 1136 2,
0000000,>,354 Send binary data,
0000015,>,250 2.0.0 XEXCH50 OK,
0000015,<,BDAT 25477 LAST,
0000031,>,250 2.6.0 <DB82FD8C490D4F43ACE766C04B23A7050F0F12@someserver.otherforest.contoso.com> [InternalId=16796908] Queued mail for delivery,
0000031,<,XQDISCARD 50,
0000031,>,251 OK, no discard events,
0000031,<,QUIT,
0000031,>,221 2.0.0 Service closing transmission channel,
0000031,-,,Local

Top Senders by Submit (analysis yielding CSV - full report has top 1000):

Sender Distribution by Submit (analysis yielding CSV):

Distribution Group Expansion Analyzer (analysis yielding CSV):

Getting started

• Download ExLogAnalyzer from here.
• Checkout the Power Point slide deck included in the download for more details about ExLogAnalyzer.
• Use ExLogAnalyzer and its distributed sample analyzers to analyze your logs.
• Develop your own analyzers. Visual Studio and Visual C# Express Edition are the recommended tools. However, you can use Notepad or your favorite editor, given that ExLogAnalyzer detects and compiles the analyzer CSharp files at runtime.
• Provide us with feedback about ExLogAnalyzer, sample analyzers or the development process.
• Share your analyzers or ideas for useful new analyzers with the Exchange community.

- Victor Boctor