I have been working with a customer recently who has a print server that has had its spooler crashing after a 3rd Party service running on it was locking up and freezing and falling over.
One of the signs it was about to fall over was running the following command against the server and seeing lots and lots of Close_waits. This was observed by running the
Netstat –an command. Example output below.
So what do the “State” actually mean. And what is the significance of Close_Wait.
Understanding the TCP sequence of steps for socket closing
As the TCP conversation is a ports and sockets sequence, to understand how to troubleshoot it and carry out root cause analysis. This was an excellent blog that explains this tcp socket conversation very well
Also see below for the explanation of the different states sockets can enter into as part of that conversation.
CLOSED Indicates that the server has received an ACK signal from the client and the connection is closed
CLOSE_WAIT Indicates that the server has received the first FIN signal from the client and the connection is in the process of being closed
So this essentially means that his is a state where socket is waiting for the application to execute close()
A socket can be in CLOSE_WAIT state indefinitely until the application closes it. Faulty scenarios would be like filedescriptor leak, server not being execute close() on socket leading to pile up of close_wait sockets
ESTABLISHED Indicates that the server received the SYN signal from the client and the session is established
FIN_WAIT_1 Indicates that the connection is still active but not currently being used
FIN_WAIT_2 Indicates that the client just received acknowledgment of the first FIN signal from the server
LAST_ACK Indicates that the server is in the process of sending its own FIN signal
LISTENING Indicates that the server is ready to accept a connection
SYN_RECEIVED Indicates that the server just received a SYN signal from the client
SYN_SEND Indicates that this particular connection is open and active
TIME_WAIT Indicates that the client recognizes the connection as still active but not currently being used
So the explanation for a close_wait situation is as below;
CLOSE is an operation meaning "I have no more data to send." that is the
client/server has chosen to treat CLOSE in a simplex fashion. The user who CLOSEs
may continue to RECEIVE Until he is told that the other side has CLOSED also. Thus,
a program/application could initiate several SENDs followed by a CLOSE, and then
continue to RECEIVE until signalled that a RECEIVE failed because the other side has
CLOSED. We assume that the TCP will signal a user, even if no RECEIVEs are
outstanding, that the other side has closed, so the user can terminate his side
gracefully. A TCP will reliably deliver all buffers SENT before the connection was
CLOSED so a user who expects no data in return need only wait to hear the
connection was CLOSED successfully to know that all his data was received at the
destination TCP. Users must keep reading connections they close for sending until
the TCP says no more data.
Adjusting Registry Settings
Registry keys to look at which can sometimes help to configure and adjust this conversation.
TCP Connection States and Netstat Output
This entry makes more ports available.
Reducing this value from its default setting of 240 seconds will make ports expire sooner. This parameter determines the length of time that a connection stays in the TIME_WAIT state when it is being closed. While a connection is in the TIME_WAIT state, the socket pair cannot be reused. This is also known as the 2MSL state because the value should be double the maximum segment lifetime on the network. See RFC 793 for more details.