Mark Russinovich’s technical blog covering topics such as Windows troubleshooting, technologies and security.
A few weeks ago a poster with the handle dloneranger reported in the 2CPU forums that he experienced reduced network throughput on his Vista system when he played audio or video. Other posters chimed in with similar results, and in the last week attention has been drawn to the behavior by other sites, including Slashdot and Zdnet blogger Adrian Kingsley-Hughes.
Many people have correctly surmised that the degradation in network performance during multimedia playback is directly connected with mechanisms employed by the Multimedia Class Scheduler Service (MMCSS), a feature new to Windows Vista that I covered in my three-part TechNet Magazine article series on Windows Vista kernel changes. Multimedia playback requires a constant rate of media streaming, and playback will glitch or sputter if its requirements aren’t met. The MMCSS service runs in the generic service hosting process Svchost.exe, where it automatically prioritizes the playback of video and audio in order to prevent other tasks from interfering with the CPU usage of the playback software:
When a multimedia application begins playback, the multimedia APIs it uses call the MMCSS service to boost the priority of the playback thread into the realtime range, which covers priorities 16-31, for up to 8ms of every 10ms interval of the time, depending on how much CPU the playback thread requires. Because other threads run at priorities in the dynamic priority range below 15, even very CPU intensive applications won’t interfere with the playback.
You can see the boost by playing an audio or video clip in Windows Media Player (WMP), running the Reliability and Performance Monitor (Start->Run->Perfmon), selecting the Performance Monitor item, and adding the Priority Current value for all the Wmplayer threads in the Thread object. Set the graph scale to 31 (the highest priority value on Windows) and you’ll easily spot the boosted thread, shown here running at priority 21:
Besides activity by other threads, media playback can also be affected by network activity. When a network packet arrives at system, it triggers a CPU interrupt, which causes the device driver for the device at which the packet arrived to execute an Interrupt Service Routine (ISR). Other device interrupts are blocked while ISRs run, so ISRs typically do some device book-keeping and then perform the more lengthy transfer of data to or from their device in a Deferred Procedure Call (DPC) that runs with device interrupts enabled. While DPCs execute with interrupts enabled, they take precedence over all thread execution, regardless of priority, on the processor on which they run, and can therefore impede media playback threads.
Network DPC receive processing is among the most expensive, because it includes handing packets to the TCP/IP driver, which can result in lengthy computation. The TCP/IP driver verifies each packet, determines the packet’s protocol, updates the connection state, finds the receiving application, and copies the received data into the application’s buffers. This Process Explorer screenshot shows how CPU usage for DPCs rose dramatically when I copied a large file from another system:
Tests of MMCSS during Vista development showed that, even with thread-priority boosting, heavy network traffic can cause enough long-running DPCs to prevent playback threads from keeping up with their media streaming requirements, resulting in glitching. MMCSS’ glitch-resistant mechanisms were therefore extended to include throttling of network activity. It does so by issuing a command to the NDIS device driver, which is the driver that gives packets received by network adapter drivers to the TCP/IP driver, that causes NDIS to “indicate”, or pass along, at most 10 packets per millisecond (10,000 packets per second).
Because the standard Ethernet frame size is about 1500 bytes, a limit of 10,000 packets per second equals a maximum throughput of roughly 15MB/s. 100Mb networks can handle at most 12MB/s, so if your system is on a 100Mb network, you typically won’t see any slowdown. However, if you have a 1Gb network infrastructure and both the sending system and your Vista receiving system have 1Gb network adapters, you’ll see throughput drop to roughly 15%.
Further, there’s an unfortunate bug in the NDIS throttling code that magnifies throttling if you have multiple NICs. If you have a system with both wireless and wired adapters, for instance, NDIS will process at most 8000 packets per second, and with three adapters it will process a maximum of 6000 packets per second. 6000 packets per second equals 9MB/s, a limit that’s visible even on 100Mb networks.
I caused throttling to be visible on my laptop, which has three adapters, by copying a large file to it from another system and then starting WMP and playing a song. The Task Manager screenshot below shows how the copy achieves a throughput of about 20%, but drops to around 6% on my 1Gb network after I start playing a song:
You can monitor the number of receive packets NDIS processes by adding the “packets received per second” counter in the Network object to the Performance Monitor view. Below, you can see the packet receive rate change as I ran the experiment. The number of packets NDIS processed didn’t realize the theoretical throttling maximum of 6,000, probably due to handshaking with the remote system.
Despite even this level of throttling, Internet traffic, even on the best broadband connection, won’t be affected. That’s because the multiplicity of intermediate connections between your system and another one on the Internet fragments packets and slows down packet travel, and therefore reduces the rate at which systems transfer data.
The throttling rate Vista uses was derived from experiments that reliably achieved glitch-resistant playback on systems with one CPU on 100Mb networks with high packet receive rates. The hard-coded limit was short-sighted with respect to today’s systems that have faster CPUs, multiple cores and Gigabit networks, and in addition to fixing the bug that affects throttling on multi-adapter systems, the networking team is actively working with the MMCSS team on a fix that allows for not so dramatically penalizing network traffic, while still delivering a glitch-resistant experience.
Stay tuned to my blog for more information.
Great article Mark, thank you for giving a clear explanation of the problem.
is it the reason that explain why my games are A LOT less performant with windows Vista? does playing audio in games and having Teamspeak (or ventrilo at the same time) reduce the graphic and procesor output for my games because audio is at higher priority (especially in MMORPG where the network is important)?
is there a way to put this value back to normal?
Running XP I just tried playing an MP3 file, a video file and downloading several files from the internet all at the same time. The audio and video files played perfectly and there was no slowdown in my network speed.
"mechanisms employed by the Multimedia Class Scheduler Service (MMCSS), a feature new to Windows Vista"
Seems to me Microsoft tried to "fix" something that wasn't broken.
Very useful to know. I've spent a good bit of time trying to figure out why my file transfers were so slow. During all of that debugging, I probably had a music file running in the background...
Serge: Games performance is probably more related to video driver issues. My understanding is that the new Vista video driver model is theoretically just as fast as the XP video driver model, but it requires a lot of stuff to be re-written by the driver developers for best performance. Until it all gets re-written, they're using pieces from the XP driver (changing Vista's input to what the XP driver expects, then changing the XP driver's output to what Vista expects), and the result is that some tasks are done less efficiently.
On my laptop, I could hardly get any video files to play without glitching. CPU usage would go up to 100% and stay there, even though the media file played just fine with only 20% CPU usage on XP. Then by accident I was running another program that was incompatible with Aero and forced Vista to switch into the XP-style video mode. Suddenly my video started playing back smoothly at 20% video usage, just like XP.
I don't own Vista yet, but the first thing I thought of when I started reading this post was "games". Good question...
A sincere thank you for explaining the issue in detail and not “suger coating” it—you might renew my faith in Microsoft.
Monday, August 27, 2007 5:28 PM by Richard McBeef
It won't effect your internet speed; just your LAN speed.
my first thought was to disable the MMCS service, but the Windows Audio service is dependent on it.
So I ran regedit, and changed the key
Just remove MMCS from that key in the list, and set MMCS to disabled in services, then reboot.
As soon as I rebooted I was able to copy files at 40mb/s+ while listening to audio
Why is XP not having this problem then? This seems like unnecessary feature of Vista.
I've spent countless hours for couple months trying to figure out why copying files over network was going at 5MB/s
It seems to me that Vista has things backwards here. Rather than ensuring that music playback can be performed within its realtime constraints, and having programmable facilities within the kernel to ask for that, it's arbitrarily degrading other bits of the system "just in case" they might interfere. The dependency is incorrect. It ought to be the scheduler making these kinds of decisions - not the sound subsystem.
It smells more like a nasty hack - as if last-minute testing showed that there were glitches in playback with the latest kernel, so quick fixes needed to be bludgeoned in.
"While DPCs execute with interrupts enabled, they take precedence over all thread execution, regardless of priority, on the processor on which they run"
Am I reading this correct: The DPCs can all run on core #1 and multimedia can run on the other cores? -> no interruption of multimedia?
Serge: No, the MCSS service is only used by multimedia applications like WMP (actually that's probably all that uses it ATM).
You can revert to old behavior by right clicking My Computer, clicking Manage, finding the Services entry on the left pane and finding the MCSS service mentioned and stopping it. To make this setting permanent, you have to go into properties and set the startup type to Disabled. However it will not make your games run any faster. (Hmm, Courtney notes it's dependent on the Windows Audio user-mode stack and offers a workaround... I didn't know that. Thanks Courtney!)
Games are slower overall because of all the added functionality and features in Vista... even with many services, eyecandy, and programs disabled I still find programs run better in XP (and even better in Linux!). For example, BioShock runs horribly on my computer in Vista with input and audio lag making it unplayable. This didn't surprise me too much as my computer didn't meet the minimum specs in the CPU department. But, in XP, it ran quite acceptably. I was even able to up the details settings without performance degradation.
I would recommend gamers to stick to Windows XP for the time being, unless you have a DirectX 10 video card and plan on playing DirectX 10 games (and you really, REALLY need DirectX 10 for some reason). Also note that DirectX 10 cards will not be able to take advantage of new DirectX 10.1 features in Vista SP!. You'd need a new card for those.
Richard: As noted in the article, you have to have quite high bandwidth (over 100mbits/s) to be able to notice any slowdown. Also there was something broken they fixed... audio stuttering or jittering in WMP and other media applications. Now if they only did the same thing for games I might switch over to Vista permanently.
Doug: Unless you're speaking of LAN transfers on a 1gbps network, I doubt that was your problem.
I have noticed problems when running on an IPv4 network. IPv6 drivers are enabled by default for every network adapter, meaning every time you try to connect to a remote computer/website, Vista tries using IPv6 first. If your network doesn't support it, this only results in wasted time. You can disable this protocol by finding the moved "Network Connections" folder (go to Network Control Panel and click on whatever side entry relates to showing network adapters) and then right clicking Properties on desired adapters and unchecking the IPv6 protocol entries.
Andrew: XP can have this problem if enough apps are suching up the CPU. Some multimedia apps, like Winamp, solve the problem themselves by boosting their own priority (Winamp runs at High by default) but this can be dangerous if done to High or above because the application is not stable because then it may freeze the entire system (fortunately Winamp is stable. I've never had problems in that area).
Whenever I have an application which is running a tad pokey, especially games, I boost it to Above Normal using Process Explorer or Task Manager (cmon Mark, fix Procexp so it can replace Taskmgr when UAC is enabled). This works with most programs (a few games go wonky though).
barrkel: I think all programmers do this, It's called implementing the required and requested features with the least amount of work. We're a lazy people. ;)
zzz: I'm guessing you could minimize DPCs and interrupts on one core by putting only threads essential to music playback there, but most likely this would make the core underutilized. And just like with memory, you don't want to leave it unused because you just end up wasting time which you could be saving by filling it with work.
An excellent explanation Mark.
Maybe I am oversimplifying things, but you mention it limits to 10,000 packets. Wouldn't the obvious solution be to increase this number (or make it configurable). Obviously if you make it too high, your songs can skip. If you have a multi-core CPU (as most will be in the next few years), then it would seem to me to be unnecessary. Perhaps Windows could detect whether this optimisation is really helpful.
Great post Mark!
MVP Windows Server - Admin Frameworks
"I have noticed problems when running on an IPv4 network. IPv6 drivers are enabled by default for every network adapter, meaning every time you try to connect to a remote computer/website, Vista tries using IPv6 first. If your network doesn't support it, this only results in wasted time."
Actually, the time wasted should be very little. Unless the hostname has an AAAA record associated with it, the system shouldn't use IPv6 to try and communicate with the host.
The only time it becomes a major delay, is when the remote host supports IPv6, and either the local or remote host isn't properly configured for IPv6. Then, it has to time out, then switches to IPv4.
IIRC, Teredo is on by default in Vista, which means (provided the teredo relay isn't down and firewall isn't blocking) you have functioning IPv6 on your end enough to be useful.