When customers first hear about being able to enable extra JET Blue or ESE Database performance counters via adding a "Squeaky Lobster" registry value, they often think it must be a joke or ask you to repeat it. And invariable the question comes up ... _why_ "Squeaky Lobster"? Various lore or conjecture has surfaced around it ... that it was a developer's child's toy or that is just thought up after lunch one day. Not exactly ... and retrieving the information was a little tricky as the initial checkin was 2 years before I started ...
The story starts with a certain developer, let's call him, "Andrew Goodsell", for the time being, but I call (or called) him, "boss", and I can assure you those explanations are not quite right. If you were to get incriminating information about your boss, you might wonder what is the proper time to blog about it? Right after you stop working for him I think is the right answer to that ... and as of today I have transferred to the Exchange organization, while "Andrew" has not. I am as they say, unsupervised. ;-) Andrew, is a very sharp man who prides himself on his professional decorum, and a bit of a perfectionist to boot. In fact his professionalism is what makes this singular lapse in judgment so amusing and fairly uncharacteristic of any decision he would make (or have let me make ;-) today.
Andrew has been involved in a fair share of the database performance work in ESE and as anyone decent at perf knows, one of most critical steps is having a way of accurately measuring performance, base-lining performance, at several levels in a software stack. Exchange 5.5 added ESE Database performance counters to facilitate this work. Most of these perf counters were intended for only internal ESE and Exchange developer usage (as you can probably deduce from one of the code snipits below, and looking at what these counters measure... things like B+ Tree Inserts/sec for example).
Enter cute girl ... So that autumn while Andrew was working on adding perf counters and working on Exchange 5.5's performance, a cute girl was managing computers in the Exchange Performance lab. Andrew was working on performance and adding performance counters while this girl worked with the powerful computers in the performance lab. It isn't a wonder how they ran into each other. She was cute, it is no wonder Andrew got a crush.
Some of the simplest ESE performance counters would also be helpful for advanced administrators in debugging Exchange server issues (Cache Size, File Operations/sec, etc). Andrew mentioned he thought if competing companies had access to some of the more detailed perf counters, they might be able to reverse engineer our implementation details and steal intellectual capital (now that's the professional Andrew I know, trying to protect our intellectual capital). After he says this, he rolls his eyes and says something to the effect of "I don't know what I was thinking!" (this is because the implementation details are more complex than what the counters show, but at the time this made sense). So there was a need to split the ESE Database perf counters into two kinds. ... internal (dev only) and external (admins). The internal ones would have to be hidden.
Now, fairly smitten with this girl, ANYTHING she did was fascinating (to Andrew). Oh come on, we've all been there. And one such thing she did was participate in a Giving Campaign, basically an event where we raise money for charities and non-profits. In order to encourage people to give money, various prizes would be donated and a raffle sort of mechanism to win the prizes would be done. The girl of interest donated a toy Squeaky Lobster to the charity raffle.
Andrew thought that was the funniest and most random thing to contribute as a prize for a giving campaign ever. Who would want a Squeaky Lobster? Most random prize in giving campaign I guess equals most uneasy to guess registry value ever. Anyway, a name was chosen that was about as random as you could get for 1997 (as there weren't very many words available in 1997, I think the Seattle was recovering from that awful grunge thing). Knowing the crowd Andrew hung out with back then, it would not surprise me if they decided it at or on the way back from lunch as they used to work 12 hour days and take extra long lunches.
An excerpt of Andrew's fateful check in from SLM logs (note, I've deleted the developer actual email aliases), relevant information highlighted:
#F eseperf.cxx v1
#T Fri Sep 26 11:15:00 1997
#C 47424, 41913 finalize perf ctrs
> #pragma const_seg( ".text" )
> const char szDisplayDevOnly = "Squeaky Lobster"; #pragma const_seg()
As an aside, note the ".text" segment notation, I had forgotten about that, now that's old school. Man, 8 years is a LONG time in the computer industry.
And so a little over 4 months later, the first Squeaky Lobster Enabled product shipped on Feb 3, 1998, with ESE97 in Exchange 5.5. In fact this is when the database engine under Exchange was renamed from JET Blue to ESE (to avoid confusion with JET Red, which has only vestigial relations to JET Blue / ESE). ESE97 shipped in 1998, just to confuse everyone. The above picture is of the original Exchange 5.5 packaging with the original Squeaky Lobster. But at this point in our story, the counters are secret so it's not a big deal.
But there is no way to keep any sort of more extensive analysis mechanism a secret for long ... eventually someone will need the information. And eventually an Exchange performance case came along sometime in 1998 (according to the earliest record I can find in the PSS DB) that required the extra analysis these performance counters offered. The PSS engineer told a customer how to enable the counters so they could analyze the customer's issue. I mean, it comes down to the customer, you do what is necessary and possible to support them. Then another case came in, and another, and eventually someone from PSS thought to publish a KB on 4/17/2000, KB 259895, what counters are enabled by squeaky lobster, _official_ Microsoft documentation admitting the existence of Squeaky Lobster!!!
Well, that didn't last long (about 6 months). In about 2000 someone public (Brian Sheaffer / Paul Thurrott) noticed and of course such a thing was far too silly for the serious professionals that control the web site of most big corporations. ;-) About as silly as Squeaky Lobster for a prize in a charity benefit! I mean we're a professional organization.
Now there are about 300 PSS related investigations, service tickets and KBs referencing this phrase (including the above one). There is a Squeaky Lobster in Andrew's office as well, but do not be fooled, it is not the original. That Squeaky Lobster was given to him by a PSS engineer who thought it was funny, or maybe to thank him for being repeatedly asked by customers, "What was that? Can you spell it?".
A mere 4 and a half years after the checkin, someone wised up ... and for Windows 2003 RTM, and Exchange 2003 SP1, code was checked in to try "Show Advanced Counters" first, and if that fails try "Squeaky Lobster". The comment in the code is:
> // deprecated name (yes, we are ending the insanity)
> err = DwPerfUtilRegQueryValueEx(hkeyPerf,(char*)"Squeaky Lobster",&Type,&lpbData);
Note: we moved away from that archaic .text seg stuff, the compilers are now smart enough.
And finally to come full circle, once this happened the name of the registry value appropriate for product documentation on our web site. Note that page specifies Exchange 2003 SP1. If you replace "Show Advanced Counters" with "Squeaky Lobster", the instructions should work for Exchange 5.5 through Exchange 2003 RTM as well as current products, though the registry key you use for step 2, varies. For Ex 5.5 it is "ESE97" instead of "ESE". In Ex2k you have to use "ESE98" before SP2, and finally for SP2 and later, just "ESE". The same process works for Windows, but use "ESENT" instead of "ESE", oh and you have to enable the performance counters for Windows first.
I promised an Exchange MVP, Michael B. Smith, about a year ago and I apologize for the delay, some would even mock me for taking so long with this single post, but at that time, the fact that there was a Ms. Squeaky Lobster unknowingly influencing Microsoft product development was not known. You got to wait until you get all the details, Eric. It wasn't till Andrew, accidentally spilled the information at a dinner. After that I blurted out, "I'm SOOO BLOGGING THAT!", Andrew had an immediately look of regret on his face, and now he won't really talk about it anymore. ;-) So that is about all the Squeaky Lobster trivia I could collect.
The next time you have to really dig into Exchange (or Active Directory) database performance, just remember how responsible professional ESE software engineers are adding easily discoverable (based upon the activities of their current crush at the time) mechanisms for diagnosing your top issues. So with that in mind, the next time you go to buy a software product, remember to check the box to see if this is one of the many Squeaky Lobster Enabled products. Oh just checked an Exchange 2003 box, not actually listed on the box, hmmm, I guess serious professionals are in control of the packaging too. ;-)
So Sept 26th of this year is Squeaky Lobster Day, the 9 year anniversary of the checkin of the Squeaky Lobster registry value, be sure to Squeaky Lobster a server at 11:15, and then go to an extra long lunch, and make a silly decision you'll regret and feel embarrassed about 9 years down the road ... sorry, Andrew, you can't stop the insanity ...