Last time I talked about buffer overflow errors that you might see in Filemon traces. Now I’ll turn my attention to the same errors, but in Regmon traces. Recall that a buffer overflow error in this context is not a security hole, but a way for the system to tell an application that there’s more data available in response to a query the application has made than can fit in the application’s output buffer. A commenter on the previous post pointed out that a better error for this case would be “buffer too small” or “more data available”, and I agree. There is in fact a STATUS_BUFFER_TOO_SMALL error code, but it’s used in situations where no data is copied to the caller’s buffer, whereas STATUS_BUFFER_OVERFLOW is used when some, but not all, available data has been copied.

Buffer overflow errors in Regmon traces are relatively common. The documentation for RegQueryValueKey says:

If the buffer specified by lpData parameter is not large enough to hold the data, the function returns ERROR_MORE_DATA and stores the required buffer size in the variable pointed to by lpcbData. In this case, the contents of the lpData buffer are undefined.

If lpData is NULL, and lpcbData is non-NULL, the function returns ERROR_SUCCESS and stores the size of the data, in bytes, in the variable pointed to by lpcbData. This enables an application to determine the best way to allocate a buffer for the value's data.

There are two programming approaches commonly taken to reading variable-length Registry data. The first is to simply pass in a NULL pointer on the first call and if the call succeeds to allocate a buffer for a second query. The other approach is for an application to first try and use a static buffer of a size that the programmer expects will usually be large enough to receive the stored data. If the system reports ERROR_MORE_DATA, which is the Windows API equivalent of the native STATUS_BUFFER_OVERFLOW error, the application dynamically allocates a buffer of the required size and re-executes the query. The second approach has the advantage of avoiding a second query in most cases.

Given these approaches you should never see two buffer overflow errors in a row that query the same data. At least that’s what you’d expect, but on many occasions I’ve seen two buffer overflow errors followed by a third query of the same data that succeeds. Here’s an example I see on a Windows XP SP2 system that results when I open My Computer:



I decided to investigate this particular case as I was writing this blog posting. First, I started SoftICE and set a breakpoint on NtQueryValueKey. When the breakpoint triggered I single-stepped in SoftICE to the return instruction, at which point the status of the query is stored in the EAX register. I then cleared the first breakpoint, and set a conditional breakpoint on the return instruction for the case that the result is STATUS_BUFFER_OVERFLOW (0x800000005):

bpx _NtQueryValueKey+0346 IF eax==80000005

Then I opened My Computer and this is the stack at first buffer overflow breakpoint:



A look at the parameters Shlwapi passes into RegQueryValueExA revealed a NULL buffer pointer. I stepped back out into Shlwapi’s code and saw that if RegQueryValueExA returns ERROR_SUCCESS, which is the error code to which RegQueryValueExA translates STATUS_BUFFER_OVERFLOW if the caller uses a NULL buffer pointer, Shlwapi dynamically allocates a buffer of the required size and makes the call again:



Shlwapi makes one call into RegQueryValueExA, but two entries get added to Regmon's trace, one with a buffer overflow and one that's successful. Stepping into Advapi32’s implementation of RegQueryValueExA showed me why the second call results in two Registry queries in the Regmon trace. NtQueryValueKey returns a data structure storing several different pieces of information regarding the Registry value, including its type, length and title index:

typedef struct _KEY_VALUE_PARTIAL_INFORMATION {
ULONG TitleIndex;
ULONG Type;
ULONG DataLength;
UCHAR Data[1]; // Variable size
} KEY_VALUE_PARTIAL_INFORMATION, *PKEY_VALUE_PARTIAL_INFORMATION;

However, RegQueryValueExA returns only the actual data into the caller's output buffer. RegQueryValueExA (and RegQueryValueExW) therefore uses a 144-byte stack-allocated buffer to query the value. This is the place in Advapi32 where RegQueryValueExA makes that call:



The first instruction in the disassembly is RegQueryValueExA loading the address of NtQueryValueKey into the ESI register. 144 is represented as 0x90 in hexadecimal, which you can see loaded into the EDI register and then passed as the fifth parameter to NtQueryValueKey, which corresponds to the output buffer size.

The additional value information included in the KEY_VALUE_PARTIAL_INFORMATION structure requires 12 bytes, so if the data being read is larger than 132 bytes NtQueryValueKey returns STATUS_BUFFER_OVERFLOW. If the caller’s buffer is large enough to store the data RegQueryValueExA allocates a buffer large enough to hold it and then executes the call to NtQueryValueKey again. After getting the data RegQueryValueExA copies it to the caller’s buffer.

When I saw the double-buffer overflows in the Regmon trace I thought it was evidence of poorly-written code, but my investigation showed that the pattern will be present in an application that reads Registry values larger than 132 bytes in size. What I don’t know is why the Windows API developers picked 132-bytes as the magical buffer size they expected would hold most Registry data.

On an unrelated node, I’m going to be in Orlando this week speaking at TechEd and delivering a Windows internals seminar with Dave Solomon. If you are attending either please let me know you read the blog. If you can’t make TechEd you can still catch my Understanding and Fighting Malware: Viruses, Spyware and Rootkits presentation the afternoon of Tuesday, June 7, via live web cast. Sign up here: http://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032274949&Culture=en-US

Originally by Mark Russinovich on 6/4/2005 8:20:00 PM
Migrated from original Sysinternals.com/Blog
# re: Buffer Overflows in Regmon Traces

(Sorry for the off-topic post, but can you please use PNG or GIF for your screenshots instead of JPEG? They'll look a *lot* better for flat-color images and text, and they'll usually compress better too. The JPEG artifacts and blurry text hurt my eyes.)

6/6/2005 2:50:00 AM by Anonymous
# re: Buffer Overflows in Regmon Traces

Pity the Webcast becomes hopelessly broken up as you go in to it.

I downloaded it and used Windows 2000 and Media Player 9 to view it (fully updated and patched).

By 45min in I can't see what you're pointing at. Only some squares over your moving mouse pointer were getting updated.

Does it actually work for others?

6/15/2005 7:04:00 AM by Anonymous
# re: Buffer Overflows in Regmon Traces

I should say that it sounds good and only the full screen blue slides seem to work.
(It starts off well, breaks up after 20mins).

I was going to show this to my staff as a good guide to malware and your tools.

But it becomes a right mess on screen.

Shame really.
No chance you could get MS to fix it? ;-)

6/15/2005 7:08:00 AM by Anonymous
# re: Buffer Overflows in Regmon Traces

Now tried your webcast on another PC running XP sp2 and Media Player 10.

Oh dear, the same video corruption is present (sound is ok).

So unless its a corruption in my copy of the downloaded .wmv file it
looks like MS have messed up your presentation video.

6/15/2005 8:34:00 AM by Anonymous
# re: Buffer Overflows in Regmon Traces

Unfortunately, there's no way to improve the quality, sorry.

6/15/2005 9:29:00 AM by Mark Russinovich
# re: Buffer Overflows in Regmon Traces

What software did you use to make the webcast?

I'd like to know what to avoid using in the future. ;-)

6/16/2005 10:36:00 AM by Anonymous