Blog du Tristank : Networking

ISA Server 2006 TCP Retransmits

tristank — Wed, 14 Oct 2009 04:24:20 GMT

Health Checks

I perform ISA Server Health Checks for Premier Support (via Premier Field Engineering) as part of my role.

I’ve seen something a few times recently that I thought it might be helpful to call out, while poking around in the Performance Monitor TCPv4 counter area.

The Problem

In short: Lots of TCP retransmissions per second.

Like, lots. More than 1% is annoying; any more than 5% and you pretty surely have a problem.

Recently, I’ve been seeing 20%.

That’s right, kids, according to Perfmon’s statistics, one in five TCP packets requires retransmission.

If your ISA Server seems like it might be a bit slow, and you haven’t looked yet, go look. I’ll wait. You’re interested in the TCPv4 object, specifically the Segments/sec and Segments Retransmitted/sec counters.

What I’ve seen looks like this:

The green area is TCPv4\Segments/sec. The red area is TCPv4\Segments Retransmitted/sec. They’re using the same scale.

Notice that the retransmission figures track with the overall volume.

This 20% figure has been seen across Intel and Broadcom server NICs, so I don’t think it’s specific to either vendor.

Fixing It

In at least one of the places I found this, a simple driver upgrade to the latest version available looked like it fixed the problem.

Otherwise, it could indicate a NIC issue, or a hardware issue with the switch.

If you find yourself in this situation, and do resolve it, please do post details in the comments section below.

PL15W2SP.DLL vs Firewall Client

tristank — Wed, 19 Aug 2009 14:37:00 GMT

As I possibly misspelled or misremembered it, the PL15ws2p.dll (possible sic) file was installed as a Winsock Layered Service Provider on a couple of boxes at a customer site.

Coincidentally, these machines were Windows Server 2008 machines where we couldn’t get the Firewall Client to work properly.

We found that there was a third party LSP using:

NETSH WINSOCK SH CA > catalog.txt

And then opening catalog.txt in notepad. The properties of the Pl15ws2p.dll indicated that it was a signed DLL from American Power Corporation or similar (APC or ACP; one of those no-notes half-hours), and that it was used in some sort of management capacity.

But only one of the machines had this APC software installed on it, and the other didn’t… perhaps it got left behind when it was being uninstalled? The search engines didn’t seem to know much about it.

Either way, next step was clear:

NETSH WINSOCK RESET

To return the Windows Sockets provider list to its shiny defaults, and reboot the computer.

After that, the Firewall Client wasn’t working (which we expected).

A Repair from Not-Called-Add-Remove-Programs-Any-More-Now-It’s-Programs-And-Features-Silly fixed that up.

Cool, huh? Remember: when nothing makes sense and the configuration looks good, perhaps LSPs are to blame?

Now if only I could get my stupid Huawei 3G modem working on my Win7 laptop again (“Device attached to the system is not functioning”… thaaanks).

ISA Server 2006 on Windows Server 2008: Nup

tristank — Thu, 04 Jun 2009 18:52:46 GMT

Yuri’s blog explains some of the detail. But there’s slightly more subtlety to it, which I’ll try to snake-oil in front of you here:

Can I install ISA 2006 on 32-bit Windows Server 2008 ?

No, it only runs on Windows Server 2003. Okay, so technically, it also runs on Windows 2000, but if you’re installing it like that now, you should check the calendar. Windows 2000 is old, man.

Why not ISA Server 2006 on Windows 2008?

Whenever I asked that, people mumbled about TCP/IP stack changes. Sounds plausible, so I let it slide.

Well can I install ISA 2006 on 64-bit Windows Server 2008 ?

No. Wait – sort of, not really. Do you count virtualization?

What do you mean?

Hyper-V or an SVVP-validated platform. (Details on security. And the inimitable “Jim Harris” apparently pretending to be Jim Harrison. Giggle.)

Er, if I do count virtual machines?

Yes. You run it in a 32-bit Windows Server 2003 guest.

Isn’t that cheating?!

No. Well, maybe. Sorry, did you have a point there?

What about Windows Server 2003, x64 Edition?

Installing ISA on it? No. It’s 32-bit only and uses kernel-mode software; you can’t mix and match 32-bit with 64-bit k-mode drivers. Hint: I just helped you study for 070-351.

What about Service Pack 2?

X64 Edition?

Yes!

No.

You’re not being helpful.

Oh really? Your eyes are the wrong shape.

The next version of ISA Server, called Forefront Threat Management Gateway (TMG, or, I guess, Timmy to his friends (yep, I’m betting the G ends up semi-silent)), is available in its initial release in the Windows 2008 Essential Business Server thingo, which is 64-bit only.

The next standalone (i.e. non-EBS-integrated) release is currently available in Beta form, and runs exclusively on Windows Server 2008, x64 edition.

That was more helpful.

You still look funny.

Hey, why don’t your links open in new windows?

Because I think it’s nice for the reader to be able to choose whether an informational link should appear in the current frame or a new tab (or a new window).

Sometimes (probably quite often on this blog), you’ll be done with the content at the current page you’re reading, and just want to replace it with something else. Forcing a new window isn’t polite in the age of tabbed browsing.

Let the user choose.

I agree, that’s so wise. You’re like, amazing.

I know.

IIS WebDAV Security Advisory

tristank — Tue, 19 May 2009 09:49:20 GMT

Today, an IIS 5.0 to 6.0 security advisory was released:

Vulnerability in Internet Information Services Could Allow Elevation of Privilege

http://www.microsoft.com/technet/security/advisory/971492.mspx

If you’re using WebDAV on any version prior to 7.0 (where it was completely rewritten, and released as an add-on module after ), you’ll want to read the advisory, and take appropriate action.

Mitigating factors are listed in the advisory.

On the ISA Server Security Update

tristank — Wed, 15 Apr 2009 03:57:59 GMT

Rambling my way to a point

One of my most favourite “Favorites” (read: “he snarled”) in recent weeks has been the ISA Server Product Team’s Build Numbers post.

They helpfully list the version numbers of each ISA Server, um, version, along with a link to the most recent hotfix for that version. That’s so helpful.

But: In most cases, you had to use the self-service hotfix feature to get that hotfix. Which is better than calling someone, but still not quite one-click conweenyence.

And there was some useful stuff fixed in each – you can do the research (hint: research is typically along the lines of “isa server hotfix site:support.microsoft.com” in whatever search engine you use).

Back to the security update: if you look at the file list for the security updates, they look a lot like the file lists for the recent hotfixes.

(Aside from a little while ago: nice that we’re again using KB articles for file information and not just “you should read the bulletin” placeholders. Makes it easier to reliably find file version information in the one place. No idea who changed it in the first place, but my blunt message to you: that was suboptimal.)

I know you love short versions, Glenda

So, long story short, by applying the security update, you’re getting the most recent build of those binaries for your ISA Server.

Just one caveat: remember that with this patch, you’ll need to reapply it if you make any significant installation-level changes to ISA later (see the bulletin for that).

Antivirus software on ISA Server

tristank — Thu, 09 Apr 2009 02:44:00 GMT

There are two major classes of Anti Virus software (yes, I know I used one word above, it’s called SEO, okay?) that can be used on an ISA Server computer:

ISA-integrated antivirus scanning products
Regular desktop/server antivirus products

The first category is the cooler of the two, and typically involves a Web Filter and/or an Application Filter. It’s been designed to work with ISA Server, and will likely scan HTTP streams while ISA is processing them.

The second category is more common – a desktop or server antivirus product is installed on the ISA Server. That’s probably a good idea from a Defense In Depth perspective.

But if you’re using the second category (or it’s just part of your server build), did you know that there are a set of exclusions we recommend you should use?

The ISA Server product team did some great work in pulling together a set of recommendations for when Antivirus is used on ISA Server. Have a read, have a think, and then check whether yours is implemented correctly. If it isn’t, outages, poor performance and other issues might arise.

And (sorta getting into the ramble here) have you ever noticed that Support people tend to make uncomfortable noises about Antivirus products when you mention they’re installed (if not outright suggesting that you disable and/or uninstall them straight-off)? Well, that’s because when they’re not configured in a way that doesn’t interfere with the operation of other software, they really have, statistically, experientially, and commonly, been known to cause problems.

It’s almost a cliche to be asked to remove AV software while troubleshooting a problem – but the cliche came from somewhere to begin with. Configuring the AV as recommended is an excellent way of minimizing that risk.

IAG – now available for Hyper-V

tristank — Thu, 29 Jan 2009 12:47:03 GMT

Of all the things I could be doing right now, blogging is the one that won. Feel special? Procrastination, but with a helpful bent.

IAG SP2 is now a VHD for Hyper-V

Your mission, Jim, is to make that into a song.

The most interesting “wow” moment I had today was reading that IAG (Intelligent Application Gateway - that’s that Whale SSL thingo) is now available without accompanying hardware.

Previously (as I understand it) IAG 2007 was only available on a hardware appliance of sorts.

Now, at least as far as the Technet Deity is concerned, IAG 2007 SP2 is licensable as a Hyper-V Virtual Machine, if you don’t want to go for the hardware.

The VHD includes IAG 2007 SP2 (I’m downloading the trial now, to get up to Mischief) and ISA Server 2006 (for the firewalling capabilities), running on Windows Server 2003.

I’m something of a noob to IAG, so, um, if you want to ask something, go hit them up instead.

But yay, can’t wait to try it out.

Home Hyper-V Networking Gotchas

tristank — Tue, 13 Jan 2009 15:12:56 GMT

Before the holidays, I bought myself an early present: a new quad-core box with 4GB RAM, which I was going to use for a home Hyper-V lab, so that I could run a bunch of 64-bit VMs as well as the 32-bit staples I’ve been using for years (SBS 2003, and a separate ISA Server box).

I’d had Windows Server 2008 installed on my Virtual Server host for a while, and use it with Routing and Remote Access (RRAS)’ NAT to provide a simple internet gateway for a segment of my internal network.

Lesson #1: Core Quad Q8200s don’t support VT (that’s Hyper-V, kids)

There was a 1300Mhz FSB Q8200 available for the same price as a Q6600, and I figured that I couldn’t go wrong with that. Surely, I thought, all Intel CPUs since the Core2 Duos support Hyper-V?

Well, no, said Intel, and thanks for your money (stupidty tax, I seem to pay a lot of it). The one Quad core chip that doesn’t support Hyper-V is the one I bought. Q8200 is being phased out (I read somewhere), so this mistake should be easily avoidable in the future. Or now, by how-you-say smarter people.

Lesson #2: When you Hyper-V-ify a Parent Partition, It’s Sort Of A Client Too (aka “You may need to set stuff like RRAS up again with the new virtualized network adapters”)

What I mean by this is that when I got the Right CPU and installed Hyper-V, I was without Internets.

To cut a long and boring troubleshooting story short: the physical network adapters I’d configured in RRAS were no longer the Right Network Adapters.

I set up new virtual networks for each physical adapter (one Internet, one Local), and then had to set up RRAS again, because it didn’t think there were any new interfaces to set up – it was quite happy only seeing the old ones, thank you very much.

After checking both virtual adapters were visible in the Network Connections interface, and that they had the right IPs assigned, I rechecked my Windows Firewall settings and ran a port probe to confirm only ports I knew I wanted open were open (RRAS Basic Firewall doesn’t exist any more in 2008, so be careful with dual-homing where the Internet is attached to one of your adapters).

The disconnect here was that I was assuming the parent partition would see the physical hardware – it does, it just doesn’t use it directly any more, it looks like it uses the virtualized setup instead, at least to some extent.

Lesson #3: Hyper-V and DHCP didn’t like each other when the physical host became the parent partition

My RRAS server had (to this point) been my DHCP server for the internal network. This was all fine, and seemed to be working okay (or had my lease durations just not expired yet?), except for the new virtual hosts I created today.

There’s some lore floating around on the forums that worked for me – the bit that worked was manually adding a REG_MULTI_SZ called IPAddress to the likeliest-looking adapter interface in the registry, because Hyper-V setup for whatever reason doesn’t do that.

The DHCP server wouldn’t bind to the physical adapters (or even show them in the Bindings interface), presumably because IPv4 and IPv6 was unbound from them (interesting, hey?) and also wouldn’t show me either of the virtual adapters, which I guess is due to the lack of a static IP address on either of them.

Now, though, my setup’s working nicely, everything more or less as it was before, only virtualized. And thus, you know, more sexy.

What does it mean when there's no "broken page" icon in IE8?

tristank — Wed, 10 Sep 2008 10:59:09 GMT

I was just catching up on some of my RSS feeds, and noticed that one of the pages I was at didn't have a broken page icon, but wasn't working quite right (some broken javascript in the photos area, I'm guessing... I'll investigate that next).

I wondered what that meant, so fired up Fiddler2 to have a look.

The Headers collection didn't include the compatibility header (X-UA-Compatible: IE=EmulateIE7 or similar):

HTTP/1.1 200 OK
Proxy-Connection: Keep-Alive
Connection: Keep-Alive
Content-Length: 77144
Via: 1.1 MYPROXY
Date: Wed, 10 Sep 2008 07:42:24 GMT
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/6.0
X-AspNet-Version: 2.0.50727
Cache-Control: private

But the META tag was present (link):

<!doctype html public "-//w3c//dtd xhtml 1.0 transitional//en"
"http://www.w3.org/tr/xhtml1/dtd/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head><meta http-equiv="X-UA-Compatible" content="IE=7" /><title>
Spied: New Mazda3

So: the page is being told to render in IE7 Standards Mode (forced, as opposed to IE=EmulateIE7, which would behave as IE7 did). This makes the toggle compatibility mode button moot, because the site has chosen their mode explicitly.

Wonder if that's the problem... Time to investigate with the developer toolbar, I think...

(Update: nup, document mode didn't fix it - Script Debugging needed to be un-disabled in IE, and then the debugger showed me it was happening in motiongallery.js. I've lost interest now :) )

That Memory Leak Revisited

tristank — Wed, 25 Jun 2008 08:59:22 GMT

While searching for memory leaking troubleshooting techniques that could be applied to 64-bit Windows (for the DHCP Server memory leak I found I had the other day), I stumbled across the answer to my problem in an internal tool (weird that I missed it from a web search the first time, but c'est la vie).

A Windows Server 2008-based DHCP server that is configured in a workgroup environment may consume too much memory

http://support.microsoft.com/default.aspx/kb/949530

And that's my problem! One REG command (and one restart of the DHCPServer service) later, I'm waiting to see how it went, but it all looks promising, based on that article. Neat-o.

Windows Server 2008 Diagnostics Off The Cuff

tristank — Fri, 20 Jun 2008 14:15:47 GMT

A word of caution to those of you that like endings: this isn't over yet.

I'm running a rather sad and noisy X64 desktop as a server at home. Once a proud warrior, actually, no, wait, it was never any good. It's just a Virtual Server host (it's not quite Hyper-V capable; next one will be). SBS 2003, an IIS and an ISA Server all exist(ed) happily in there at one point. (Did I mention I virtualized my work desktop machine the other day? So liberating!)

I blatted Windows Server 2008 onto it at RTM, and it's been happily puttering along doing the RRAS internet access and Virtual Server thing for me ever since.

Until Recently

But I've had to reset it from unresponsive-no-mouse-no-capslock situations on about four occasions over the last two weeks, and as the problem wasn't getting any better, so I figured I'd take a look at what I could do to try to diagnose it.

My guess was that I had a kernel-mode memory leak (a user mode memory leak shouldn't ever trash the box to that extent), but it didn't seem to correspond with any driver upgrades or software installations... something else had changed, sometime.

Perfmon (the new, shiny version) or more specifically the Reliability Monitor confirmed my suspicions:

(happy, everything-used-to-be-so-nice side on the left, then the gradual decline due to Disruptive Shutdowns towards the right). Note the quite-regular interval of red things on the bottom row. (Does it happen more when I'm at home, he wondered?)

Preparation:

As I had a theory in mind, I thought I'd create a Perfmon BLG (log file with lots of counters in it; lots of people seem to like CSV, but BLG is faster, and I'm never going to be opening it in Excel anyway).

How to do that? Things have changed: now, I create a "Data Collector Set", it seems. Oh yeah, reading manuals and/or following basic instruction: not my thing.

I created a new one based on the System Performance collector set, which matches my needs nicely because it contains all the Process counters and Memory counters. Between that lot, I should easily be able to spot a memory leak.

Started the collector set, and made a mental note to check in tonight.

Tonight:

After a little fiddling, I worked out that the animated "Data Collection In Progress" screen wasn't generating a report, and that I'd have to stop the data collector set to view it. Right on!

So, one stopped data collector set later, the Reports view is what I'm interested in.

Remember your training - you're interested in patterns that have slopes or steps. One counter leapt out at me, which I moused over and found was....

Process (_Total) Pool Nonpaged Bytes

So, yep, there's a memory leak, and it's in one or more of the objects tracked by Process counters. So let's add the Pool NonPaged Bytes counters for <All Instances> (so I can see all the processes).

So Add all them, and there's a counter that matches the slope, but at a different scale. Click it in the display to select it, and it's SVCHOST#10. Hide all the other counters I've just added (multi select, right click, hide all), and then right-click it and choose Scale Selected Counter.

Whop! Matches the curve almost exactly.

So, now I know it's a service host, but I don't know which one (they all look alike to me). I assume it's probably still running, too. How do I find that out now?

Easy: Add the "ID Process" counter for svchost#10 (#9 pictured, artistic license)

And then click the counter in the list to see the value it has (the plotted line is flat across the graph, meaning it didn't change at any point). I get PID 1348.

TASKLIST /SVC tells me everything I need to know (well, not everything obviously, but enough to take corrective action).

Yep - it's the DHCP Server instance of SVCHost that's apparently leaking NPP, a kernel resource.

Why!? And why now!?

The graph tells me the times at which this happened, but the Event Logs are very, very quiet around then. So I'll need to use tracing or logging or some other technique to actually track down the cause of the problem.

I right-clicked the SVCHOST instance with PID 1348 and chose Create Dump File (awesome feature, mentioned that before), for archival/root cause purposes - it may well not be possible to see the cause of the leak after the fact from a hangdump, but it's worth grabbing just in case - and then restarted the DHCP Server Service.

Taskman memory use dropped by about 100MB straight away. This is not a busy network, and NPP isn't typically used as cache by user mode programs (he giggled (in a manly way)). Something weird is going on there.

I restarted my performance logging, and I'll check in again tomorrow to see if there's any further indication of a memory leak (I haven't done anything to fix it, so I assume there will be). Now, time to look for logging and diagnostic options...

A word on Perfmon in Windows Vista and 2008: USE IT!

If you're doing any level of performance analysis of Perfmon logs, you need to try out the new, improved Perfmon in Vista. It runs rings around the old one. It's fantastic (at least by comparison). It's worth the cost of the upgrade alone. Seriously, if you do any sort of work with perfmon logs, try doing it on a Vista box and see whether your life is 1000% easier! I'm not saying it's perfect, but by comparison with the last version in XP/2003...

The Cat's Out Of The Bag: ISA Server will become ForeFront TMG

tristank — Wed, 09 Apr 2008 09:21:00 GMT

So, we all know that ISA 2006 doesn't work on Windows Server 2008. Massive architectural changes to the IP stack, blah blah, etc, etc.

People (uh, yeah, just "people") have been asking about what's to become of ISA Server for a while:

"There's no ISA 2008 announced!" they'd scream.

"This surely means the end of one of the best product lines Microsoft has produced!" might have also been heard (in a somewhat muffled way).

"Won't Tristan be out of a job?" one person wailed, unconvincingly.

Well, that's right - the plan at this point is that there is no ISA Server 2008.

(pause for effect, teeth-gnashing, gasping, horror to subside)

As of the next version, Internet Security and Acceleration Server is ForeFront Threat Management Gateway!

(Now you're going to tell me that ISA was a perfectly good name and not at all unwieldy...)

See our well-formed Press Release for details!

Microsoft also today announced the name of its next-generation network edge security product, Forefront Threat Management Gateway. Forefront Threat Management Gateway is the future version of Microsoft ISA Server and will extend the capabilities of ISA Server 2006 with new features and security technologies, designed to help provide multiple-threat protection, simplified management and secure connectivity, and will be built on Windows Server 2008. More details about Forefront Threat Management Gateway will be available later this year.

Excellent! So, key takeaway: we are working on a successor. The product isn't going away!

There's an early beta available from here, though it's downloading at a slow trickle for me right now (got excitement?).

So, go forth and, you know, Manage Threats! In the future!

Post-SP2 TCP Offload Fix

tristank — Mon, 17 Mar 2008 06:31:00 GMT

I've mentioned Chimney before. Now, a new Windows Update fix for TCP Offload, which turns it off.

It was on by default in Windows Server 2003 SP2, so if your NIC supported Offload, or RSS, or that other thing I can never remember, it was enabled.

But: we (PSS we) typically turn it off as a first troubleshooting step for any network-related issue -

a) because we know from experience that several drivers seem to do interesting things with it installed (that's a nice way of saying update your drivers),

b) because several of our drivers do interesting things with it (if you're going to choose to use it, check for recent-model tcpip.sys hotfixes), and

c) because we want to be able to see TCP traffic in a network capture for troubleshooting purposes.

Off-unless-opted-in brings parity with Windows Server 2008.

"Stacking" NTLM Authentication

tristank — Tue, 11 Mar 2008 09:30:49 GMT

This question came up today (well, actually, it was about four weeks ago I started typing this, but bear with me), and it's been a little while since I've rambled about authentication protocols, so let's enjoy a nice, calm discussion on a ~~Monday~~ Tuesday arvo.

The request was something like:
In a Web Publishing scenario, can I do NTLM at the ISA Server and NTLM at the Exchange server too?

No

And the answer is - well, no.

There's no way for the client browser to distinguish between the ISA Server (first) saying 401 WWW-Authenticate: NTLM , and then the IIS Server saying 401 WWW-Authenticate: NTLM.

Because it appears to be a repeated authentication sequence when the connection is already authenticated from IE's perspective (and IE doesn't think it's talking to a different server), IE assumes there's been an auth failure (why else would the server challenge again?).

So, lots of authentication prompts are going to happen. The solution (as described) is not workable.

But

With ISA 2006 and its amazingly-useful-how-did-we-ever-live-without-them Authentication features:

What you could do is Integrated Windows Authentication at the Exchange server (i.e. allow Kerberos), and use protocol transition at the ISA Server, from whatever form of authentication you can accept from a client to Kerberos Credential Delegation (or even another protocol, depending on the auth method used by the listener).

So

The question itself was a "no", but the question almost always isn't actually the question. That one's for free.

Special note: I worked really hard on the headings for this post. I hope it was appreciated.

MaxUserPort - what it is, what it does, when it's important

tristank — Tue, 11 Mar 2008 09:17:57 GMT

What can we say about MaxUserPort that hasn't already been said? Not a lot, it would seem. He's a beautiful dancer, perhaps? Ahh, such gentle humour, and nary a kitten drowned anywhere.

But TCP port shenanigans are fairly frequently misunderstood, so let's talk about the very basics of MaxUserPort.

NB: This is all pre-Vista behaviour - applicable from NT4 through to Windows Server 2003, including all the little NT-flavoured stops on the way.

MaxUserPort controls "outbound" TCP connections

MaxUserPort is used to limit the number of dynamic ports available to TCP/IP applications.

It's never going to be an issue affecting inbound connections. MaxUserPort is not the right answer if you think you have an inbound connection problem.

(I don't know why, I just know it is. Probably something to do with constraining resource use on 16MB machines, or something.)

To further simplify: it's typically going to limit the number of outbound sockets that can be created. Note: that's really a big fat generalization, but it's one that works in 99% of cases.

If an application asks for the next available socket (a socket is a combination of an IP address and a port number), it'll come from the ephemeral port range allowed by MaxUserPort. Typically, these "next available" sockets are used for outbound connections.

The default range for MaxUserPort is from 1024-5000, but the possible range is up to 65534.

When You Fiddle MaxUserPort

So, why would you change MaxUserPort?

In the web server context (equally applicable to other application servers), you'd usually need to look at MaxUserPort when:

- your server process is communicating with some type of other system (like a back-end database, or any TCP-based application server - quite often http web servers)

And:

- you are not using socket pooling, and/or

- your request model is something like one request = one outbound TCP connection (or more!)

In this type of scenario, you can run out of ephemeral ports (between 1024 and MaxUserPort) very quickly, and the problem will scale with the load applied to the system, particularly if a socket is acquired and abandoned with every request.

When a socket is abandoned, it'll take two minutes to fall back into the pool.

Discussions about how the design could scale better if it reused sockets rather than pooling tend to be unwelcome when the users are screaming that the app is slow, or hung, or whatever, so at this point, you'd have established that new request threads are hung waiting on an available socket, and just turn up MaxUserPort to 65534.

What Next? TcpTimedWaitDelay, natch

Once MaxUserPort is at 65534, it's still possible for the rate of port use to exceed the rate at which they're being returned to the pool! You've bought yourself some headroom, though.

So how do you return connections to the pool faster?

Glad you asked: you start tweaking TcpTimedWaitDelay.

By default, a connection can't be reused for 2 times the Maximum Segment Lifetime (MSL), which works out to 4 minutes, or so the docs claim, but according to The Lore O' The Group here, we reckon it's actually just the TcpTimedWaitDelay value, no doubling of anything.

TcpTimedWaitDelay lets you set a value for the Time_Wait timeout manually.

As a quick aside: the value you specify has to take retransmissions into account - a client could still be transferring data from a server when a FIN is sent by the server, and the client then gets TcpTimedWaitDelay seconds to get all the bits it wants. This could be sucky in, for example, a flaky dial-up networking scenario, or, say, New Zealand, if the client needs to retransmit a whole lot... and it's sloooow. (and this is a global option, as far as I remember).

30 seconds is a nice, round number that either quarters or eighths (depending on who you ask - we say quarter for now) the time before a socket is reusable (without the programmer doing anything special (say, SO_REUSEADDR)).

If you've had to do this, at this point, you should be thinking seriously about the architecture - will this scale to whatever load requirements you have?

The maths is straightforward:

If each connection is reusable after a minimum of N (TcpTimedWaitDelay) seconds
and you are creating more than X (MaxUserPort) connections in an N second period...

Your app is going to spend time "waiting" on socket availability...

Which is what techy types call "blocking" or "hanging". Nice*!

Fun* KB Articles:
http://support.microsoft.com/kb/319502/
http://support.microsoft.com/kb/328476