So you went out and got yourself a new server and it came with TOE functionality, and now you're playing Windows 2008 which has TCP offload enabled but you just want to know if its actually offloading traffic. Here's the only way I know of finding what traffic is offloaded without setting breakpoints in the debugger.
First off to check if TCP offload is enabled:
C:\>netsh int tcp show globalQuerying active state...
TCP Global Parameters----------------------------------------------Receive-Side Scaling State : enabledChimney Offload State : enabled <-----Receive Window Auto-Tuning Level : normalAdd-On Congestion Control Provider : ctcpECN Capability : disabledRFC 1323 Timestamps : disabled
To turn it on/off (does not require a reboot)
netsh int tcp set global chimney=disabled
netsh int tcp set global chimney=enabled
So how do we see if traffic is offloaded? You run netstat -nt, the 't' dumps their current offload state. I used findstr just to grab the offloaded connections.
C:\>netstat -nt | findstr /i offloaded TCP 126.96.36.199:445 10.5.17.2:1369 ESTABLISHED Offloaded TCP 10.100.44.52:445 188.8.131.52:4741 ESTABLISHED Offloaded TCP 10.100.44.52:49157 184.108.40.206:2444 ESTABLISHED Offloaded TCP 10.100.44.52:49157 220.127.116.11:2255 ESTABLISHED Offloaded TCP 10.100.44.52:49157 18.104.22.168:54620 ESTABLISHED Offloaded TCP 10.100.44.52:49157 22.214.171.124:50442 ESTABLISHED Offloaded TCP 10.100.44.52:49157 126.96.36.199:1191 ESTABLISHED Offloaded TCP 10.100.44.52:49157 188.8.131.52:58308 ESTABLISHED Offloaded TCP 10.100.44.52:49449 184.108.40.206:1025 ESTABLISHED Offloaded
Windows 2003 its a bit different:
Netsh int ip set chimney DISABLED
Netsh int ip set chimney ENABLED
Want to know more about Scalable Networking?
Thanks for this article. Is there a way to tell if it is running prior disabling it on 2k3.
Is the TCP offload compatible with Windows Firewall in W2008? In 2003 its either firewall or offload, but not both.
Yes they work together, at least they do inside the MSFT walls.
I have problem windows 2008 64bit, it shows when you do netstat -t enabled but not in the broadcom Configuration, still showing Unlicensed. this was working when i had the windows 2003 installed. can you plaese give me some idea how to get work.
Thush, probably the driver you are using is not supported with 2k8 or does not have the offlaoding feature turned on. If the OS shows that offload is enabled, then for all intents and purposes, its enabled. I'd try to find out of the driver your using is supported in 2k8.
This information can save hours of trying with no luck for many on Windows 2008 and HP Proliant servers: even the latest Broadcom driver (v.4.4.15, signed by Hewlett Packard 2008.05.14, named as version 9.20 in HP support site), used in network adapters NC373i (built into most of HP Proliant servers), NC370T, NC380T and in other so called "multifunction NIC" by HP, DOES NOT support TCP offload under W2008. When HP support got official request why TCP offload is not working they answered: NC373 (Broadcom 5708) hardware supports TCP offload and this is why server specs. say "TCP offload enabled", but software under W2008 not yet, you need to wait for the future driver. Under W2003 SP2 software does support TCP offload but it is not stable and we recommend to disable it.
These Broadcom TCP ofload adapters will soon become outdated and still no driver. Customers will stop believing in Broadcom future products like 10G chip (BCM57710), no use even for the best chip if drivers are not working properly. For Microsoft, I would like to pay attention that drivers for some network adapters for Windows are already more than 100MB in size, it may look to the average user that under Vista/W2008 more code is needed for proper network adapter driver than was in whole Windows 98. Maybe the chip vendors even so big as Broadcom just failing to write so complex driver code. They are not Microsoft... something universal for network is needed in OS like the driver for USB flash keys - driver is provided by MS and hardware vendors provide just hardware.
Update: HP Proliant Broadcom Multifunction network adapter drivers version 9.30 (v.220.127.116.11) released. Tested - no changes. TCP Offload not supported under Windows 2008. In overall, time for HP to correct all QuickSpecs for Broadcom based NC so called multifunction series and put a note that TCP offload is not supported under W2008. BTW, in our datacenter iSCSI offload with this chip is not stable under heavy load if IP latency is a bit higher (for example if some switch with higher latency is in between - like Cisco 6509 - then Broadcom iSCSI offloader drops connections. W2003 dynamic disks tolerate this but W2008 dynamic disk goes offline immediatelly and in damaged state, if iSCSI connection is broken under heavy writes, for example DPM2007 is doing backup). Use this "multifunction" chip only as simple network adapter, for iSCSI use Microsoft software iSCSI initiator only - it uses a bit more CPU than hardware offload but no stability problems. Executive summary: NC373, NC380 - these Broadcom 5708/6 based premium priced multifunction network cards actually can only do one simple network adapter functiion (IPv4 only, for IPv6 no support even for check-sum). After I spent so many hours only to find that the problems are not solved and they are quite hidden, it becomes interesting why HP is placing such a network chips with too low driver quality into almost all Proliant servers. To end this story with the funny info, maybe HP love Broadcom because of the high top customer care standard :)), read here:
Wrong focus. Instead of these activities, proper drivers would do better success for keeping the top customers.
As far as I know, HP virus throttling conflicts with CHIMNEY on W2K3, I saw an advisory pertaining to it on HP site. So it's probably not an issue with the network card driver itself but with the virus throttling filter driver.
This blog is very powerful - even HP responded. Now a note is put in the QuickSpecs that TCP/IP offload is supported in Windows 2003 only (where it is not recommended for use due to stability problems, because of that it is disabled automatically by Windows update). In Windows 2008 TCP/IP offload IS NOT SUPPORTED, no way to enable it. So, at least for now, idea to use hardware TCP offload on Windows will not work for masses, and its not clear who is guilty - Broadcom, failing to write proper drivers, or Microsoft, requiring so complex drivers. Solution for 1Gbps users: buy faster processors with more cores and use software iSCSI. Solution for 10Gbps users: NONE. Unfortunatelly, you need approx. 1GHz of processor clock for 1Gpbs going through TCP/IP stack, and even more clock for iSCSI special check-sums. Only offload hardware can provide iSCSI on 10G levels. HP just released BL495c server with built-in 10G Broadcom adapters with hardware iSCSI on the chip. Dell released 10G network adapter with the same Broadcom 57711 chip "three in one" at 10G (LAN, TCP offload, iSCSI offload). But same story - hardware "can do" everything, but no drivers for Windows yet, neither iSCSI nor TCP/IP offload. This situation decreases the value of Windows server for the users. Lets hope Microsoft will put a focus on this issue before the time Windows 2008 R2 will be released. If anyone can please forward this to the right MS person: please make a work-group with the biggest network chip provider to help them write a proper iSCSI and TCP offload drivers for most popular Broadcom hardware offload chips, for drivers to be included on W2008 R2 disk. Offloads must be made stable and enabled by default. Then users will get a payback of their costly Windows license upgrade in the form of better network speeds, lower server processor utilization, and better capacity for applications (this means better power efficiency of Windows server versus the competition).
Status Update: HP released "HP Proliant networking software", dated 4Feb2009, with HP network configuration utility (NCU) 9.50 and Multifunction driver 4.6.17 inside. After installation, offload in Proliant G5 servers can be activated as described here, just in addition, one more check box also must be enabled in NCU advanced settings. Currently available only for IPv4. iSCSI hardware offload also works OK with 1Gbps adapters. Good!
I also tested 10Gbps connection offload in DL495c blade server (Broadcom 57711 chip+Flex10). TCP offload works (!), but iSCSI offload option not yet available.
I think Broadcom is now doing a long awaited progress. If iSCSI offload will be fixed for 10Gbps chip SOON, this will actually make a 10Gbps+iSCSI a very interesting "unified fabric" solution. Lets hope it will not be one more year until 10G iSCSI offload drivers for Broadcom 57711 are released. 1G servers became so old while waiting for the drivers.... Dear Broadcom, 57711 chip has excellent description but it can be of commercial interest only if proper driver is available, so speed-up please with iSCSI driver delivery.
To end with funny non-technology note, I hear one politician talking over radio: "I will not participate in this economy crisis as I do not like it. And I invite everyone to do the same."
Algis, it's like you have a blog inside my blog :)
You are right. I noticed that for some reason when I post something here changes actually happen. So maybe some important people and lot of others were reading it - despite that you made the post one year ago, if you enter "TCP offload windows 2008" on Google search and press "I am feeling lucky" you will get this page.
Sorry for posting here updates on my developing TCP offload enabling story - inspired by your post :), I spent more than a year fighting with hardware vendors asking them to turn TCP offload on in servers and chips that were advertised and sold with this feature to me.
In the future this thing will be even more important as speeds go to 10Gbps. Lets hope users will have no need to know all this and starting with Windows 2008 R2 it will work in optimised way by default, at least with most popular hardware.
This is my last comment on your blog :) Saying that, I would like to thank you for this post.
Hi, I was wondering if there are any Kernel Debug commands that can be utilized to identify if SNP / TOE is enabled on a target system via live debug or memory dump?
I have made the required changes in the registry for the
but once I close the reedit the entries are gone. WHat am I doing wrong? PLease advice.
Is there a similar command that can disable the iSCSI offload without manually doing it from the BACS ?
We use automated deployment and so would like to have all the servers with iSCSI offload set to disabled by default ...