Mfartura's blog : Basic

Real Case: .Net Apps - "No connection could be made because the target machine actively refused it" (basic)

Marcelo Fartura — Sat, 04 Nov 2006 06:39:00 GMT

Recently we’ve been receiving (by coincidence) some escalations to troubleshoot the same issue. It’s about .Net apps returning the following exception:

SocketException::.ctor() 10061:No connection could be made because the target machine actively refused it

The cases I’ve been working with show different stack traces but the main point here is that exception is originally occurring when the application is trying to make use of a TCP socket to establish a connection with a remote machine. For all the cases escalated to us so far, the problem has ended up being simply what the error message states, i.e the remote machine, or something in between, refuses the connection attempt. This post is just to show how we’ve been able to troubleshoot these problems and isolate them to network related issues.

Let’s go through two different scenarios and see step by step how to troubleshoot them. In the first scenario we have an IIS server (which we will call from now on WEB1 – IP:10.0.0.1) hosting a .Net application which is intermittently failing to make a web services call to another IIS server (let’s call it WEB2) located in a remote environment. The problem has already been isolated to the point when the .Net app in the WEB1 makes a call to a web service hosted in the server WEB2.

The two last frames on the stack for this first scenario would be the following ones:

System.Net.Sockets.TcpClient.Connect(String hostname, Int32 port)

System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)

The first step, as always, is understand the infrastructure involved in the communication between WEB1 and WEB2. Making it short, WEB2 is the virtual name used to mask a round robin DNS rotation between 10 other web servers and between the WEB1 and WEB2 there are at least 2 firewalls, one at the WEB1 location and another one at the WEB2 location (classic environment where the WEB servers are located in the DMZ provided by both firewalls on their respective locations).

Next step is to generate a network trace from the server WEB1 (it would be also nice if we could generate a trace simultaneously from the WEB2, however first in this scenario we don’t have access to the WEB2 and even if so we would need to take a trace from all the 10 web servers because we don’t know at this point to which the WEB1 is trying to establish a connection when the problem happens. So we went ahead, installed the Microsoft Network Monitor tool in the WEB1 started collecting the network trace and waited until the problem happens. In the mean time while the problem did not happen, we just got informed about the other 10 IP addresses of the 10 web servers hereby called simply by WEB2.

Ok, the problem happened, so now it’s time to dive in the network trace:

All the know-how we need now it’s a basic knowledge about how TCP connections happen. At the moment of the problem the trace shows the following suspect frames:

2 {TCP:2, IPv4:1} 10.0.0.1 192.168.0.1 TCP TCP: Flags=.S......, SrcPort=4973, DstPort=HTTP(80), Len=0, Seq=3411621893, Ack=0, Win=16384 (scale factor 0) = 16384

3 {TCP:2, IPv4:1} 192.168.0.1 10.0.0.1 TCP TCP: Flags=.S..A..., SrcPort=HTTP(80), DstPort=4973, Len=0, Seq=1096790416, Ack=3411621894, Win=0 (scale factor 0) = 0

4 {TCP:2, IPv4:1} 10.0.0.1 192.168.0.1 TCP TCP: Flags=....A..., SrcPort=4973, DstPort=HTTP(80), Len=0, Seq=3411621894, Ack=1096790417, Win=16616 (scale factor 0) = 16616

5 {HTTP:3, TCP:2, IPv4:1} 10.0.0.1 192.168.0.1 HTTP HTTP: HTTP Payload

Tcp: Flags=....A..., SrcPort=4973, DstPort=HTTP(80), Len=1, Seq=3411621894 - 3411621895, Ack=1096790417, Win=16616 (scale factor 0) = 16616

6 {HTTP:3, TCP:2, IPv4:1} 10.0.0.1 192.168.0.1 HTTP HTTP: HTTP Payload

Tcp: [ReTransmit #5]Flags=....A..., SrcPort=4973, DstPort=HTTP(80), Len=1, Seq=3411621894 - 3411621895, Ack=1096790417, Win=16616 (scale factor 0) = 16616

7 {HTTP:3, TCP:2, IPv4:1} 10.0.0.1 192.168.0.1 HTTP HTTP: HTTP Payload

Tcp: [ReTransmit #5]Flags=....A..., SrcPort=4973, DstPort=HTTP(80), Len=1, Seq=3411621894 - 3411621895, Ack=1096790417, Win=16616 (scale factor 0) = 16616

8 {TCP:2, IPv4:1} 4.79.185.20 10.0.0.1 TCP TCP: Flags=..R....., SrcPort=HTTP(80), DstPort=4973, Len=0, Seq=1096790417, Ack=843163868, Win=0 (scale factor 0) = 0

The frame #2 represents the server WEB1 starting a TCP 3 way hand shake with the server whose the IP address is 4.79.185.20 at the port TCP 80 (remember that this is a web service call) which at this point is the one representing our server WEB2 (there are 10 possible IP addresses depending on which one the DNS round robin will provide through the name resolution). The 3 way hand shake has been successfully accomplished through the frames #2,#3 and #4 (the sequence SYN, ACK SYN, ACK has occurred). After that something interesting happens: The WEB1 starts sending packets (starting on frame #5) of 1 single byte (notice that difference between the sequence number results in 1 -> 3411621895 - 3411621894 = 1) instead of making the complete POST to the web service.

The WEB1 receives no acknowledge from WEB2 to the packet sent at the frame #5, so it sends it again (that’s why the frame #6 and #7 are being considered retransmissions) in the frame #6, again the WEB2 doesn’t ack that, so WEB1 keeps resending that 1 byte packet in the frame #7 and then finally the WEB2 responds to that but with a RESET instead. The packet sent by the WEB2 at the frame #8 in unexpected and will cause the application to handle that as the exception mentioned in the beginning of this article.

The question is: Why the server WEB1 keeps sending a 1 byte length packet after the hand shake?

Before answering this question, you might make a mistake and think that the WEB2 is legitimately resetting the connection with the WEB1 since WEB1 has not behaved as expected (which would be send a POST to the web service and not the 1 byte packets) and this would take you to the assumption that there is something wrong with the server WEB1 and lead you to completely wrong troubleshooting line… Didn’t you see what’s wrong yet? Ok, it’s really a little tricky but the problem here happens at the hand shake (event though is was successful one)… Do you see now? No? Ok, look again to the 3 frames that compose the hand shake, more precisely to the frame #3. Notice that the WINDOW size is 0 (Win=0) which basically means the server WEB2 can not receive anything larger than 0 bytes… Interesting, isn’t it? So after receiving the 0 bytes length window size advertised by the WEB2, the WEB1 sends a 1 byte packet in attempt to make the WEB2 to advertise a new valid WINDOW size, which doesn’t happen after the first attempt, so the WEB1 keeps trying until the WEB2 resets the connection.

So the problem has been isolated to be at the WEB2 end. Another question remains: Why is the problem intermittent? After all, if this behavior of advertising an invalid window size is always happening, the problem should also always happen. It’s time to take a new network trace…

On a new network trace, we have both situations: When it works and when it doesn’t work. We already know what happens when it doesn’t work, so now let’s see what happens when it works:

2 {TCP:2, IPv4:1} 10.0.0.1 192.168.0.2 TCP TCP: Flags=.S......, SrcPort=4961, DstPort=HTTP(80), Len=0, Seq=843591585, Ack=0, Win=16384 (scale factor 0) = 16384

3 {TCP:2, IPv4:1} 8.7.82.20 10.0.0.1 TCP TCP: Flags=.S..A..., SrcPort=HTTP(80), DstPort=4961, Len=0, Seq=1446804074, Ack=843591586, Win=8760 (scale factor 0) = 8760

4 {TCP:2, IPv4:1} 10.0.0.1 192.168.0.2 TCP TCP: Flags=....A..., SrcPort=4961, DstPort=HTTP(80), Len=0, Seq=843591586, Ack=1446804075, Win=16560 (scale factor 0) = 16560

5 {HTTP:3, TCP:2, IPv4:1} 10.0.0.1 192.168.0.2 HTTP HTTP: Request, POST /WebSrv/QueryWebSrv.asmx

6 {HTTP:3, TCP:2, IPv4:1} 192.168.0.2 10.0.0.1 HTTP HTTP: Response, HTTP/1.1, Status Code = 100

Ok, now we don’t see the same window size problem and 3 way hand shake is ok and everything is fine. However our WEB1 server is not connecting to the same server WEB2 as now the server WEB2 is being represented by a different IP address – 192.168.0.2 as oppose to 192.168.0.1 that we’ve see before – due to the round robin balance system in place. So, it answers our second question: The problem actually happens always when the WEB2 is the 192.168.0.1 server, however since there is a round robin balance in place, the server WEB2 will not always be the one whose the IP address is 192.168.0.1.

The solution of problem? Well, after isolating the problem we contacted the people in charge of the WEB2 end and they generated new traces on their end and found out there was a firewall in between the servers which was misconfigured regarding the IP 192.168.0.1. They reconfigure the firewall and this fixed the problem for our first scenario.

The second scenario is a IIS server (from now one WEB1 IP: 10.0.0.1) reporting the same error as above, however now the stack, even though the exception happens at the same place, is a little different as below:

System.Net.Sockets.TcpClient.Connect(String hostname, Int32 port)

Microsoft.AnalysisServices.AdmdCleintXmlaClient.GetTcpClient(ConnectionInfo connectionInfo)

The difference here is just that in this scenario our .Net application being hosted by IIS is calling a SQL Server (IP: 172.16.0.1) running the Analysis Services as opposed to a web service. The architecture of this scenario is a little bit easier to work with since the problem has already been isolate to one specific SQL Server when it’s accessed from any web server.

By following the logic as before, we will first generate a network trace from the WEB1 server (again, we don’t have control over the other end – the SQL server, as it would be useful to also generate a network trace from that server). By looking at the trace we see the following:

7447 {TCP:534, IPv4:533} 10.0.0.1 172.16.0.1 TCP TCP: Flags=.S......, SrcPort=20531, DstPort=2383, Len=0, Seq=655839663, Ack=0, Win=16384 (scale factor 0) = 16384

7448 {TCP:534, IPv4:533} 172.16.0.1 10.0.0.1 TCP TCP: Flags=..R.A..., SrcPort=2383, DstPort=20531, Len=0, Seq=0, Ack=655839664, Win=0 (scale factor 0) = 0

7467 {TCP:535, IPv4:533} 10.0.0.1 172.16.0.1 TCP TCP: Flags=.S......, SrcPort=20531, DstPort=2383, Len=0, Seq=655839663, Ack=0, Win=16384 (scale factor 0) = 16384

7468 {TCP:535, IPv4:533} 172.16.0.1 10.0.0.1 TCP TCP: Flags=..R.A..., SrcPort=2383, DstPort=20531, Len=0, Seq=0, Ack=655839664, Win=0 (scale factor 0) = 0

7554 {TCP:538, IPv4:533} 10.0.0.1 172.16.0.1 TCP TCP: Flags=.S......, SrcPort=20531, DstPort=2383, Len=0, Seq=655839663, Ack=0, Win=16384 (scale factor 0) = 16384

7556 {TCP:538, IPv4:533} 172.16.0.1 10.0.0.1 TCP TCP: Flags=..R.A..., SrcPort=2383, DstPort=20531, Len=0, Seq=0, Ack=655839664, Win=0 (scale factor 0) = 0

What we see here is the WEB1 making successive attempts to connect to the port TCP:2383 (the default port for non named instances of Analysis Services running in SQL Server 2005) and the SQL Server is resetting all of them. We realized that there are two routers between the server WEB1 and the SQL Server. We can’t identify if the resets are coming from the SQL or any of the routers since the MAC address in the reset packets will always be the one of the last router in the network path between the servers. However by doing closer look at the reset packets, we notice that the TTL has been decremented by 2 (ttl=126) as below:

+ Ethernet: Etype = Internet IP (IPv4)

- Ipv4: Next Protocol = TCP, Packet ID = 14341, Total IP Length = 40

+ Versions: IPv4, Internet Protocol; Header Length = 20

+ DifferentiatedServicesField: DSCP: 0, ECN: 0

TotalLength: 40 (0x28)

Identification: 14341 (0x3805)

+ FragmentFlags: 0 (0x0)

TimeToLive: 126 (0x7E)

NextProtocol: TCP, 6(0x6)

Checksum: 9687 (0x25D7)

SourceAddress: 10.205.100.146

DestinationAddress: 10.205.100.200

- Tcp: Flags=..R.A..., SrcPort=2383, DstPort=20531, Len=0, Seq=0, Ack=655839664, Win=0 (scale factor 0) = 0

SrcPort: 2383

DstPort: 20531

SequenceNumber: 0 (0x0)

AcknowledgementNumber: 655839664 (0x271751B0)

+ DataOffset: 80 (0x50)

+ Flags: ..R.A...

Window: 0 (scale factor 0) = 0

Checksum: 65170 (0xFE92)

UrgentPointer: 0 (0x0)

The TTL being decremented by two basically means this packet has been routed twice and since we know there are two routers between the servers, we can conclude the resets are really being originated at the SQL server and not by any of the routers. Why is the SQL Server resetting these connection attempts? The answer for this question is also the solution for the problem and it is that the port TCP 2383 is the one used by SQL server 2005 but our SQL Server is a SQL Server 2000 which has the default instance for the Analysis Services running in the port TCP 2725 instead.

We confirmed that by running a “netstat –b” at the SQL Server. Here is the abbreviated version of the output:

TCP svp020064:2725 SVP060005:42802 ESTABLISHED 2344

TCP svp020064:2725 SVP060002:58478 ESTABLISHED 2344

TCP svp020064:2725 SVP060001:2064 ESTABLISHED 2344

We now use the command “tasklist” to identify which process has the PID 2344. The abbreviated version of the output is below:

Image Name PID Session Name Session# Mem Usage

============ ======= ============= =========== ==========

msmdsrv.exe 2344 Console 0 734,136 K

The solution is to change the client application so it connects to the right port which is the TCP 2725 instead of TCP 2383.

/3GB, /PAE, AWE what??? (basic)

Marcelo Fartura — Thu, 14 Sep 2006 17:35:00 GMT

I've been following the discussions about these options for a long time. When it seems that everybody agrees on something, somebody else comes with a different understanding and the discussion starts all over again... The questions at the beginning of each thread about this subject are often the same:

When should I use /3GB option, and when should I not use it?
Why should I have /PAE option in place?
What about /PAE and /3GB? Should they be used together? Or, can't they be used together?
Are there any performance or reliability implications when using these options?
Where AWE come from?

I understand you don't have to be an expert to perfectly understand these options and when they apply. Let's try applying the KIS (Keep It Simple) technique here, and see how all these options work in a very basic, or simple, way. But first let's make sure some basic concepts, very important to understand those options, are covered:

A very simple vision of Processes, Applications and Virtual Memory
Imagine a classroom with any number of students. This classroom have a set of minimum resources like chairs, desks and notebooks for the students, a whiteboard etc.

The Processes are what is represented by the classroom to the students in this example, or a space shared by them and their resources. The space occupied by the students and the resources within the classroom is limited, as it is the Virtual Address Space of a Process, or it's Virtual Memory. While we measure the space limit of a real classroom in square feet, we do for a process and its virtual address in GBytes with the difference that classrooms can be any size and a process virtual address space is always 4GBytes large when working with 32 bit architecture - 2^32 = 4G.

The Applications will be the students in the classroom. You can have multiple students sharing the same classroom, so you can have multiple Applications hosted by the same process, or in other words, sharing the same virtual address space. Actually, for instance when a process is created to host a Win32 app, it will also host at least the application responsible for providing the interface to the executive (ntdll.dll) and one of the applications which provide the Win32 APIs (kernel32.dll). So, when we start up a very simple application, its process will also host at least 2 additional other applications.

The Virtual Address Space of each process is divided in two regions: The User Mode where the mentioned applications will run and the Kernel Mode which will be a representation of an address space that is exclusively accessed by the OS kernel. By default the distribution is 50/50 or 2GBytes being available to all User Mode apps sharing the same address space, or being hosted by the same process, and 2GBytes for the kernel. Back to our example, suppose our classroom has its available space reduced to half of the total size. The students can not access the other half.

Now, direct to the point:

/3GB
The /3GB boot.ini option or 4GT Ram tuning as it's called, will change the way the virtual address space is divided between the kernel and the user mode application on each process. As said before, by default, when not using /3GB option, the division is 50/50. When using the /3GB though, the division becomes 25/75 so the Kernel will reduce its virtual address space to 1GB and the address space used by User Mode apps will earn the additional 1GB so becoming 3GB. Let's take a look at the picture below for a better illustration on what happens when we're not using the /3GB option:

The 4 billion 32 bit addresses in hex format will go from 0x00000000 to 0xFFFFFFFF. When not using the /3GB option, the line which separates the Kernel and User Mode addresses is placed in the middle, at the address 0x7FFFFFFF in a way that the User Mode addresses are from 0x00000000 to 0x7FFFFFFF and the Kernel Mode addresses start at 0x80000000. Now, let’s look on what happens when we do have the /3GB option in place:

As we can see the line was lowered and the addresses scheme for both User and Kernel Mode changed. Now, the addresses used by User Mode apps are from 0x0000000 to 0xBFFFFFFFF and the ones used by the Kernel are from 0xC0000000 to 0xFFFFFFFF.

However the fact that the address space accessible by the User Mode applications is now larger might not represent any advantage to the applications. The applications will need to be aware of this larger address space. They need to be designed to make use that. The applications need to be compiled with a special link option called IMAGE_FILE_LARGE_ADDRESS_AWARE otherwise they won't use the additional 1GB of virtual memory. The tool dumpbin.exe from the Visual Studio can tell, when using the flag /HEADERS (dumpbin.exe /HEADERS), if the application has this link option enabled or not.

In the other side, the 1GB the Kernel needed to give up from, might be missed because the memory reduction affects key memory areas like, for instance, the file system cache (it directly impacts the paged and non-paged pool sizes and drastically reduces the number of system PTEs - but this is outside the scope of this article) and this may impact the reliability and performance of the entire system.

The conclusion is that the only general recommendation for the usage of /3GB option is do not use it unless it's required by the vendor of the application the sever is dedicated to. There are, for instance, specific recommendations for Domain Controllers, SQL Servers, Exchange Server etc, but no general rule for using it. Also, make sure the rules of your server do not conflict regarding the /3GB recommendation. For instance, if you have a SQL Server and File Server running in the same box, you will need to deal with the conflict that for a SQL Server the use of /3GB is recommended however for a File Server it is recommended to do not use it.

/PAE
Now that we know where the 4GBytes limit comes from (32-bit procs -> 2^32 = 4G) we also understand that the OS kernel itself will not be able to see beyond that either. Something needed to change at the processors in order to be able to address beyond the 4GBytes. The Physical Address Extension (PAE) is an extension provided by intel (the 6th feature, or the bit 6 = 1 in the feature flags of the processor) to enable 32-bit processors to support more than 4GBytes of RAM. Looking at that in a very simple way, it makes the processors be able to address up to 64GBytes through providing a 36-bit physical addressing mode. So, with 36 bits available for physical addressing, the process is able to use 2^36, or 64 billion addresses, instead of the original 4 billion.

The OS kernel also needs to be aware of this since it will need to change its address translation scheme (PAE Kernel will introduce a third level - PDPE - in the address translation) to proper handle the memory beyond the 4GB. The /PAE option at the boot.ini will make the PAE kernel (ntkrnlpa.exe for single proc machines or ntkrnlpamp.exe for multiproc machines - SMP) to be loaded, instead of the regular one (ntoskrnl.exe).

Notice that PAE has no effect at the processes Virtual Address Space as they are still 4GB large regardless of the amount of physical memory in the system. So referencing back our classroom example, we could say that there are several classrooms (or processes) in our building (or running in the OS) and that the PAE will provide more space to the building

AWE
We know that PAE will make the processor be able to address 36 bits and the OS, if the correct kernel gets loaded (through the /PAE option), will be able to see beyond 4GB of physical memory. However we did not mention anything about how this will impact the application behaviors. The application are, by default, still on that same classroom with 2GB (or 3GB if 4GT is enabled) even with the PAE enabled. The Address Windowing Extension (AWE) is a set of extension APIs that the application need to use in order to make use of the same 32-bit large pointers to access physical memory beyond 4GB. In other words, if the application was not developed to take benefit of that, even on systems with way more memory than 4GB they will still be limited to 2GB (or 3GB if 4GT is enabled). The SQL Server is as example of application that uses AWE.

Putting all together:

The /3GB option is to be used whenever, and only, the application vendor requires or recommends it. The applications must to be compiled with the IMAGE_FILE_LARGE_ADDRESS_AWARE in order to take advantage of it and it has no necessary relationship with the amount of physical memory the system has available. However if the system has more than 4GB of physical RAM and it's not 64-bit, you will need to use the /PAE option to make the OS aware and use beyond the 4GB limit.

If you have the /PAE and /3GB used together, the system will limit the physical memory usage to 16GB regardless of the amount of memory beyond that (this is due to the exhaustion of kernel resources to do both addressing large amounts of memory and reduce it's virtual space to 1GB, but this is subject to another article :-)).

If you have more than 4GB of physical RAM, be aware that just enabling PAE will not make the applications to actually make usage of any additional memory unless they are properly designed for that (they must use the AWE special APIs for this).