Template: How to take Network traces between clients and Exchange servers.
In the last couple of years we have noticed that due to the complexity of companies and networks support must collect more and more network traces. This is to help people collect the correct data the first time.
You will need a network tap device or software that can collect network traffic. There are two widely used capturing software’s: Microsoft Network Monitor (netmon) or Wireshark.
Below is an example using netmon between a windows XP/Vista/7 client and Exchange 2003/2007.
1. Depending on how exchange is setup you will need to know where the effected user’s mailbox is homed and if that server is a cluster. The client will always attempt to connect to the Active node were store is running.
2. Install netmon on your client. (if not already installed)
Note: I do not give the link for downloading netmon but go to https://download.microsoft.com and search “netmon” the latest version will come up. Also download the correct version depending on if the OS is 32bit or 64bit.
3. Install netmon on your Exchange server. (if not already installed)
Note: if you are using ISCSI targets over Ethernet, installing a network capturing software can cause a “blip” on the wire because it’s adding a driver to the network stack. This could cause loss of disks for a second and worse case cause database corruption. If this is an issue, install netmon on your passive node first and then failover to that node. If you have JBOD or Fiber Channel this is a non-issue.
4. On the client be sure that outlook is closed and the process is not still running in Task Manager.
5. At this point go into a command prompt Start > Run type CMD click OK.
6. In to command prompt you will have to navigate to the install folder netmon is installed. Type cd "c:\Program Files\Microsoft Network Monitor 3" (default install location)
7. To start the network trace type nmcap.exe /network * /capture /file ExchangeTrace-client.chn:100M
Note: This will allow netmon to collect a network trace from all adapters and save them in 100 MB sequential files. This will make it easier to upload and review.
8. Go to the Exchange server and do Steps 5. 6 and 7. Change the name of the file to ExchangeTrace-server.chn:100M
9. On your client start outlook and reproduce the issue.
10. Now we need to stop the network traces. In the command prompt press CTRL+C. Remember to do it on both client and server.
11. Note the actions taken during the reproduction of the issue with time stamps. This allows the person reviewing the network traces to better understand why specific actions happened in the trace.
For Exchange 2010 Outlook clients connect to the Client Access Server for Mailbox requests and the Mailbox server for Public Folder requests. Also, with CASarray you will need to capture on all Client Access Servers or use a Host file record to bypass the load balancer and have a 1:1 network trace.
Notes to the above content
As we saw above, the command to start a capture on all adapters and make 100MB sequential files.
nmcap.exe /Network * /Capture /CaptureProcesses /File ExchangeTrace.chn:100M
Now here is how to do the same but add a filter since maybe we know we only want traffic to and from the server and client but want to drop everything else from the trace.
In the below example the client IP is 10.10.50.40 and the server IP is 10.1.75.20
nmcap.exe /Network * /Capture "(ipv4.Address == 10.10.50.40 and ipv4.Address == 10.1.75.20)" /CaptureProcesses /File ExchangeTrace.chn:100M
If you are familiar with netmon filtering you can see what is in the parentheses is basic netmon display filtering. You can do other filters like (tcp.port == 80) but be cautious using filtering. We may filter out other requests that could help understand why the issue is happening, such as Kerberos or LDAP requests and much more.
For high performance network capturing on an exchange server or heave load client we can use the following command.
nmcap.exe /Network * /Capture /File ExchangeTrace.chn:100M /MinDiskQuotaPercentage 10 /UseProfile 85D454AF-5EED-4d4d-8BAB-588631A33429 /DisableConversations
This command will stop the capture if the disk we are saving our output to reaches 10% free space and uses the Pure Capture profile thus not having to parse the packets as the are recoded.
Also consider reading Reducing Dropped Frames with Network Monitor 3.4
Below is how to take a network capture on an Apple client or a Linux/Unix client.(subject to change on TCPdump version)
In a terminal you can use TCPDump to collect network traffic and netmon is able to review the cap files created.
Apple OSX 10.5.x
Sudo tcpdump –t en0 –s 0 –C 100 –w ~/Desktop/netcap.cap
To stop tracing press Control+C
Apple OSX 10.6.x
sudo tcpdump -i en0 -s 0 -B 10485760 –C 100 -w ~/Desktop/netcap.pcap
To stop tracing press Control+C
Linux/Unix
tcpdump -i any -s 0 -B 10485760 –C 100 -w /home/{user}/netcap.pcap
To stop tracing press CTRL+C