Hi there,

 

In this blog post, I would like to talk about a network trace analysis scenario where we were requested to find out if excessive ARP requests were legitimate or not.

 

Here are more details about the problem:

 

Problem description:

==================

Some Windows Vista clients experience network connectivity problems. After the client gets connected to the network, it is disconnected after a short while like 1 minute or so. After some initial troubleshooting, it was found out that the connectivity problem occurred because the switch port was disabled. And our customer’s network team stated that the switch port was disabled because excessive ARP requests were received from the client in a very short amount of time (like getting 10+ ARP requests within 10 milliseconds from the same source MAC etc).

 

How we did the troubleshooting:

=============================

We collected a network trace to better understand the root cause of the problem from one of those client machines. But the problem was that we were not able to see what the client would do after resolving those MAC addresses by ARP protocol because the switch was shutting down the port due to the mentioned security setting.

 

It’s also very difficult to associate an ARP packet to a process on a Windows machine by using the methods mentioned in my earlier post. So I decided to use the method 5 mentioned in the same post). I requested the customer to disable that security setting on a switch port where the problem was seen. So that we would be able to see what the client would do after resolving the MAC addresses via ARP protocol. We collected a new network trace after making such a configuration change and got to the bottom of the problem afterwards:

 

a) In the new network trace, it was clearly seen again that the client was sending excessive ARP requests (20 ARP requests sent for different IP addresses within the same millisecond!)

 

Wireshark filter:

arp.src.proto_ipv4 == 10.0.80.1 and arp.opcode==0x1

 

No.     Time        Source                Destination           Protocol Info

326437 30.192068   HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.202?  Tell 10.0.80.1

326438 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.165?  Tell 10.0.80.1

326440 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.83?  Tell 10.0.80.1

326441 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.59?  Tell 10.0.80.1

326442 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.220?  Tell 10.0.80.1

326443 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.215?  Tell 10.0.80.1

326451 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.28?  Tell 10.0.80.1

326453 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.60?  Tell 10.0.80.1

326454 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.127?  Tell 10.0.80.1

326455 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.81?  Tell 10.0.80.1

326458 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.29?  Tell 10.0.80.1

326462 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.98?  Tell 10.0.80.1

326465 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.65?  Tell 10.0.80.1

326467 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.129?  Tell 10.0.80.1

326468 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.37?  Tell 10.0.80.1

326470 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.239?  Tell 10.0.80.1

326471 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.15?  Tell 10.0.80.1

326472 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.26?  Tell 10.0.80.1

326473 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.235?  Tell 10.0.80.1

326475 0.000000    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.77?  Tell 10.0.80.1

 

- There're 20 ARP requests sent in a row in the same millisecond

 

b) Then I decided to check a few actions that the client would take after resolving the MAC address. I started with the first one (10.0.80.202)

 

Wireshark filter:

((arp.src.proto_ipv4==10.0.80.202 and arp.dst.proto_ipv4==10.0.80.1) or (arp.src.proto_ipv4==10.0.80.1 and arp.dst.proto_ipv4==10.0.80.202)) or (ip.addr==10.0.80.202 and ip.addr==10.0.80.1)

 

=> The client first resolves the MAC address of 10.0.80.202 about 2 minutes before the problem:

 

No.     Time        Source                Destination           Protocol Info

  62994 2.518411    HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.202?  Tell 10.0.80.1

  62996 0.000000    HewlettP_11:21:6b     HewlettP_10:11:91     ARP      10.0.80.202 is at 00:23:7d:13:2f:6b

 

=> Then the client establishes a TCP session to TCP port 5357 and exchange some data with that system:

 

  62997 0.000000    10.0.80.1            10.0.80.202           TCP      59327 > wsdapi [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=2

  63000 0.000000    10.0.80.202           10.0.80.1            TCP      wsdapi > 59327 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=8

  63001 0.000000    10.0.80.1            10.0.80.202           TCP      59327 > wsdapi [ACK] Seq=1 Ack=1 Win=65700 Len=0

  63008 0.000000    10.0.80.1            10.0.80.202           TCP      59327 > wsdapi [PSH, ACK] Seq=1 Ack=1 Win=65700 Len=223

  63009 0.000000    10.0.80.1            10.0.80.202           TCP      59327 > wsdapi [PSH, ACK] Seq=224 Ack=1 Win=65700 Len=658

  63013 0.000000    10.0.80.202           10.0.80.1            TCP      wsdapi > 59327 [ACK] Seq=1 Ack=882 Win=65536 Len=0

  63133 0.046800    10.0.80.202           10.0.80.1            TCP      wsdapi > 59327 [ACK] Seq=1 Ack=882 Win=65536 Len=1460

  63134 0.000000    10.0.80.202           10.0.80.1            TCP      wsdapi > 59327 [PSH, ACK] Seq=1461 Ack=882 Win=65536 Len=908

  63136 0.000000    10.0.80.1            10.0.80.202           TCP      59327 > wsdapi [ACK] Seq=882 Ack=2369 Win=65700 Len=0

 

=> Then the client goes into idle state in that TCP session for about 115 seconds. And then it resolves the MAC address of 10.0.80.202 once more (most likley the the ARP cache entry was deleted so source TCPIP stack needed to re-resolve the MAC address:

 

326437 115.073211  HewlettP_10:11:91     Broadcast             ARP      Who has 10.0.80.202?  Tell 10.0.80.1

326483 0.000000    HewlettP_11:21:6b     HewlettP_10:11:91     ARP      10.0.80.202 is at 00:23:7d:13:2f:6b

 

=> Right after resolving the MAC address of the target system once more, the client immediately forcefully closes the TCP session with a TCP RST segment:

 

326484 0.000000    10.0.80.1            10.0.80.202           TCP      59327 > wsdapi [RST, ACK] Seq=882 Ack=2369 Win=0 Len=0

 

c) After doing the same check for the other IP addresses that the client was trying to resolve the MAC addresses for, I saw that the behavior was the same (the client was connecting to another target at TCP port 5753 and was exchanging some data with that system and after staying idle for about 2 minutes, was re-resolving the MAC address and was forcefully closing the TCP session with a TCP RST.

 

SUMMARY:

=========

1) After analyzing the network trace as described above, it was more clear why the client was sending soo many ARP requests within the same milliseconds to various other targets:

 

To forcefully close its existing TCP sessions established to TCP port 5753 at those target machines.

 

2) The next question was that if that activity was legitimate or not. After doing some more research on the related protocol, I came up with the following results:

 

The activity on these TCP sessions (which were forcefully closed by the client) was generated by WSDAPI protocol. Also the data within the TCP segments confirmed that the activity was legitimate (seeing meaningful XML requests and responses in the traces as documented below)

 

http://msdn.microsoft.com/en-us/library/ee886312(VS.85).aspx Get (Metadata Exchange) HTTP Request and Message

http://msdn.microsoft.com/en-us/library/ee886311(VS.85).aspx GetResponse (Metadata Exchange) Message

 

This protocol seems to be used for discovering printers, Web cameras, video systems and manipulating them remotely. More details could be seen at the following articles:

 

http://msdn.microsoft.com/en-us/library/aa385800(VS.85).aspx About Web Services on Devices

http://blogs.msdn.com/dandris/archive/2007/11/09/wsdapi-101-part-1-web-services-from-10-000-feet.aspx

 

3) Even though it wasn’t very clear why the client was closing many WSDAPI sessions at the same time, it was now clear that  it was a legitimate activity and our customer wanted to take it further from that point on.

 

Hope this helps

 

Thanks,

Murat