Mark Russinovich’s technical blog covering topics such as Windows troubleshooting, technologies and security.
A few weeks ago a poster with the handle dloneranger reported in the 2CPU forums that he experienced reduced network throughput on his Vista system when he played audio or video. Other posters chimed in with similar results, and in the last week attention has been drawn to the behavior by other sites, including Slashdot and Zdnet blogger Adrian Kingsley-Hughes.
Many people have correctly surmised that the degradation in network performance during multimedia playback is directly connected with mechanisms employed by the Multimedia Class Scheduler Service (MMCSS), a feature new to Windows Vista that I covered in my three-part TechNet Magazine article series on Windows Vista kernel changes. Multimedia playback requires a constant rate of media streaming, and playback will glitch or sputter if its requirements aren’t met. The MMCSS service runs in the generic service hosting process Svchost.exe, where it automatically prioritizes the playback of video and audio in order to prevent other tasks from interfering with the CPU usage of the playback software:
When a multimedia application begins playback, the multimedia APIs it uses call the MMCSS service to boost the priority of the playback thread into the realtime range, which covers priorities 16-31, for up to 8ms of every 10ms interval of the time, depending on how much CPU the playback thread requires. Because other threads run at priorities in the dynamic priority range below 15, even very CPU intensive applications won’t interfere with the playback.
You can see the boost by playing an audio or video clip in Windows Media Player (WMP), running the Reliability and Performance Monitor (Start->Run->Perfmon), selecting the Performance Monitor item, and adding the Priority Current value for all the Wmplayer threads in the Thread object. Set the graph scale to 31 (the highest priority value on Windows) and you’ll easily spot the boosted thread, shown here running at priority 21:
Besides activity by other threads, media playback can also be affected by network activity. When a network packet arrives at system, it triggers a CPU interrupt, which causes the device driver for the device at which the packet arrived to execute an Interrupt Service Routine (ISR). Other device interrupts are blocked while ISRs run, so ISRs typically do some device book-keeping and then perform the more lengthy transfer of data to or from their device in a Deferred Procedure Call (DPC) that runs with device interrupts enabled. While DPCs execute with interrupts enabled, they take precedence over all thread execution, regardless of priority, on the processor on which they run, and can therefore impede media playback threads.
Network DPC receive processing is among the most expensive, because it includes handing packets to the TCP/IP driver, which can result in lengthy computation. The TCP/IP driver verifies each packet, determines the packet’s protocol, updates the connection state, finds the receiving application, and copies the received data into the application’s buffers. This Process Explorer screenshot shows how CPU usage for DPCs rose dramatically when I copied a large file from another system:
Tests of MMCSS during Vista development showed that, even with thread-priority boosting, heavy network traffic can cause enough long-running DPCs to prevent playback threads from keeping up with their media streaming requirements, resulting in glitching. MMCSS’ glitch-resistant mechanisms were therefore extended to include throttling of network activity. It does so by issuing a command to the NDIS device driver, which is the driver that gives packets received by network adapter drivers to the TCP/IP driver, that causes NDIS to “indicate”, or pass along, at most 10 packets per millisecond (10,000 packets per second).
Because the standard Ethernet frame size is about 1500 bytes, a limit of 10,000 packets per second equals a maximum throughput of roughly 15MB/s. 100Mb networks can handle at most 12MB/s, so if your system is on a 100Mb network, you typically won’t see any slowdown. However, if you have a 1Gb network infrastructure and both the sending system and your Vista receiving system have 1Gb network adapters, you’ll see throughput drop to roughly 15%.
Further, there’s an unfortunate bug in the NDIS throttling code that magnifies throttling if you have multiple NICs. If you have a system with both wireless and wired adapters, for instance, NDIS will process at most 8000 packets per second, and with three adapters it will process a maximum of 6000 packets per second. 6000 packets per second equals 9MB/s, a limit that’s visible even on 100Mb networks.
I caused throttling to be visible on my laptop, which has three adapters, by copying a large file to it from another system and then starting WMP and playing a song. The Task Manager screenshot below shows how the copy achieves a throughput of about 20%, but drops to around 6% on my 1Gb network after I start playing a song:
You can monitor the number of receive packets NDIS processes by adding the “packets received per second” counter in the Network object to the Performance Monitor view. Below, you can see the packet receive rate change as I ran the experiment. The number of packets NDIS processed didn’t realize the theoretical throttling maximum of 6,000, probably due to handshaking with the remote system.
Despite even this level of throttling, Internet traffic, even on the best broadband connection, won’t be affected. That’s because the multiplicity of intermediate connections between your system and another one on the Internet fragments packets and slows down packet travel, and therefore reduces the rate at which systems transfer data.
The throttling rate Vista uses was derived from experiments that reliably achieved glitch-resistant playback on systems with one CPU on 100Mb networks with high packet receive rates. The hard-coded limit was short-sighted with respect to today’s systems that have faster CPUs, multiple cores and Gigabit networks, and in addition to fixing the bug that affects throttling on multi-adapter systems, the networking team is actively working with the MMCSS team on a fix that allows for not so dramatically penalizing network traffic, while still delivering a glitch-resistant experience.
Stay tuned to my blog for more information.
I know that this isn't related to the sound issue.
But can I make a request for myself and others:
Many many people, including me and many others (googling for "TrustedInstaller.exe" will show you!) are finding on Vista that this process totally hogs the CPU and disk for minutes at a time even when no activity is going on at Vista (typically at startup, rendering the desktop performance very poor for minutes after bootup).
Could Mark investigate this?
> I am having this Vista network slow down too
No, you're not.
The symptoms don't match in the slightest. If you aren't playing music, and you've disabled MMCSS then this issue can't possibly apply to you. Also, the audio issue doesn't affect what speed you're connected at, just how many packets you can process.
Also, the speeds you're getting are waaaaaaay lower than usual- or almost even worst-case scenarios for the audio throttling problem (unless all your packets are _meant_ to be around 62 bytes, that is)
There's something else going on in your system, I'm afraid. Try netstat -s and looking for packet loss symptoms
This issue can have a heavy influence on systems with 100MBit network connections too!
My database application usually has a network throughput of 1800 packets/s. Starting Media Player will decrease this to 800 packets/s!
Database is connected by ODBC via TCP/IP to SQL-Server. It's the same effect using Sybase/Anywhere database instead.
Quicktime and MusicMatch player showing this effect on startup, on WMP you have to play something to get it.
Copying files on the network is not effected by this issue on my 100MBit connection.
So what if I use Winamp to play MP3s on my Vista machine. Will I get hit by the network throttle problem too?
MS is a first class, leading edge software company.
The real problem here is the extreme difficulty
of designing highly complex software that works effectively. I am sure that MS will sort it. There were many difficulties with xp and it needed 2 service packs.
Thank you Mark for the usual interesting and informative article.
Could it be that the reason why this problem appears to apply only to Vista and not to XP is the redesign of the Vista audio stack? The new audio stack in Vista has a lot more components, including something called (from memory) audiodg.exe, and seems to do more of its processing in user mode than XP does. This would explain why lengthy processing in a DPC (which pre-empts any user mode code) would cause audio playback (or recording) to stutter.
Anyway, I agree with other posters that throttling network bandwith in this way is a horrendous hack on Microsoft's part, be they or be they not a 'leading edge software company', and thank you Mark for your article.
what a disgraceful train wreck. While I appreciate the post and it not being covered up, I am really sick of post after post talking about these sorts of issues cast as feature related. It is absurd to the point of negligent. As an Ultimate owner I cannot even copy a file from my desktop hard drive over USB to the USB backup drive while listening to music! All while running 4GB of ram, dual core,great graphics card, and the list goes on.
It is not acceptable to release what amounts to garbage and charge for it. These things should have been fixed immediately! Not nearly a year after release.
Most users do not care what the reason is, new features, rewrites etc. what they care about is basic funtionality that has been working for multiple generations not working any longer. There is no excuse or reason that will make that acceptable.
This is all coming from a long time Microsoft customer and one who understands it is a complex business. It is not complex, however, to understand that certain basic functionality should run without issue, especially when it has been generally available since 1995.
Say again? This affects network transfers, how on earth would it affect your USB transfer?
I would think throttling should be dynamic, in the sense that the NDIS driver only needs to be notified if Windows Media notices that it gets close to getting into trouble. The amount of throttling should never be fixed even within the one system.
If they can't fix the sound problems, Microsoft should give us a copy of XP since it "Just Works", unlike Vista that does nothing but waste our time.
Users expect multimedia applications, including music and video players, to offer a seamless playback experience. However, demand for the CPU by other concurrently running applications, like antivirus, content indexing, or even the mail client, can result in unpleasant hiccups. To provide a better playback experience, Windows Vista introduces MMCSS to manage the CPU priorities of multimedia threads.
The funniest part of all of this? Pro Audio applications that actually register with MMCSS still perform very badly under Vista. As a result, many of us who need to stream audio at low latency have dropped Vista and returned to XP. Low latency audio in Vista is glitchy as hell, despite MMCSS. So there are no winners here. From a pro audio editing/mixing perspective, Vista is an abortion of an OS. The moving of most of the audio stack from kernel mode to user mode has a lot to do with the problem, IMHO. Microsoft's reasoning was that a bad audio driver can cause a BSOD. My goodness, the carnage caused by this move makes a BSOD seem like a walk in the park.
I too am having the same problems like chris , not playing audio or streaming video , brand new core2 duo 3gh 2 gig ram running vista ultimate 64bit OS.
I have tried doing transfers from a xp machine to vista of mainly a image files. I always get 5% utilisation on the gigabit network , Vista reports 8.44MB/sec , but if I dump to the xp box i can get speeds of 40MB/sec.
Previosuly i could get speeds of 22MB/sec when i was using xp with a lower spec system.
I have just tried , swapping out gigabit network switch for a different switch and connecting with new purchased cables , same result.
I should mention i'm doing all this on a scsi u320 drive system and its not a primary drive that i dump to , on either systems.
I have run all the fixes, changed Mb nics even disable my bluetooth adapter and uninstalled my antivirus program , i still get 5%-9% utilization .
Anyone else got any ideas on what to do bedides going back to xp ?
Has someone tested the behavior with SP1?
Is it really solved?
Regardless of whether or not there is a multimedia/throttling issue going on. Anyone with a 1 gig router will suffer horribly as I do with internal file transfer speeds that will not under any circumstance go beyond 100Mb/s (89Mb/s to be exact).
This is unnaceptable for any OS as I can imagine if it was a 100Mb router my internal speeds would be around 8Mbs.
And this holds true on 6 different routing devices and not only in Vista 32bit, also in Vista 64 bit.
The whole network throughput thing is not singularily caused by media, there is a deeper more pervasive problem at the kernel core that needs to be addressed and re-worked all the way up to the tcpip.sys driver to get the throughput problem addressed and fixed.
Say that five times fast.
If this issue isn't addressed soon, I will change to linux as MS has proven beyond a doubt that they have no interest in delivering a tested proven product to its customers and the dollar is their bottom line.