Eric Fleischman's WebLog

Superbowl commercials

efleis — Tue, 06 Feb 2007 18:05:00 GMT

I'm not one for the game really, but the commercials are usually a lot of fun to watch. This year was particularly entertaining as a friend took the time to put together a spreadsheet and track what we thought. You know you're in a good crowd when someone is writing VBA in Excel during the game to get that last stat that we just needed to know.

Yes, I'm nerd C. ;)

http://blogs.msdn.com/tyuhas/archive/2007/02/05/superbowl-geeks.aspx

The ribbon

efleis — Wed, 06 Dec 2006 06:01:00 GMT

I know different people feel differently about the ribbon (for those not familiar, check out Office 2007) but I must admit I’m a fan. I have liked it since the first minute I used it. I definitely feel like my productivity is up as a result.

I used to have funky issues in Outlook 2003 where it did not always remember where my toolbars were sitting. The ribbon has resolved all of this. I also find myself digging in the menus far less than I used to.

Little tips about the ribbon I’d share that I like….

- Make sure to double click an element in the ribbon or right click it and try “minimize the ribbon”. This is the mode I run in. I like that it is easy to bring back (a single click) but gets you all of that screen real estate again.

- I like in Outlook how it remembers my ribbon minimize setting on a per tool basis. That is to say, I keep it minimized in email viewing but visible in appointment viewing. It remembers this.

The DCR I have spotted that I'd like (so far ;)) is that I wish, when I am in minimize mode, simply hovering over the menu items would bring it back, and following my hovering over neighboring items (like from message to insert) also respected the hover. I have to click. Not the end of the world, but if someone in Office could make this an option you'd have at least one appreciative user.

Change visibility in the directory...or lack there of (aka "what's the point of aging?")

efleis — Sun, 29 Oct 2006 01:14:00 GMT

I’m often asked about aging in adamsync so I thought I’d present the more general problem here for people to ponder. Hopefully this gives some context around the problem which aging in adamsync is supposed to address.

Imagine you are writing a tool which sync’s changes out of AD. You (the person running this tool) have some set of permissions…whatever they may be. You are syncing along happily.

One day you get a phone call…”My user was moved from OU=bar in to OU=foo yet the sync target still shows me in OU=bar. What gives?” You begin to investigate only to find out that you don’t have permissions to OU=foo. As a result, you don’t have any of the objects in OU=foo in your target location. The reason is straight forward….you don’t have permissions to the target, so when the object moved from bar to foo you never saw this change. You couldn’t see this change! You didn’t have permissions to OU=foo.

This is one of many such cases. If you don’t have the ability to see some object in the target location, it is hard to say anything about your view of it from the source. You could still have the object in the source location and have no idea that it moved out of your view. The reason is of course straight forward….you can’t see the target so you didn’t see that mod and we don’t have any construct where the source can say “out of your purview but not here anymore.” So you simply don’t realize the object has changed.

Historically, this was not nearly as much of a problem. Most people use DirSync to sync changes out of AD. In Win2K, in order to use DirSync you needed to be a domain admin. So, you could see most things that happen (out of the box anyway). In Win2k3 we built a feature for DirSync that made this problem more common….DirSync object security mode. In this mode anyone can use DirSync to sync out of any partition they so choose, and DirSync only shows changes for objects you have access to see. This is a very useful feature.

So now let’s consider adamsync, a simple DirSync client, and the problem I've mentioned above. When we wrote adamsync we wanted to ensure that we could handle the scenario where you are not an admin and want to sync data out. So, we default the tool to object security mode. This is fairly convenient for non-admins that wish to use the tool.

However, consider a very mainstream case. You are using adamsync to sync objects out of some domain NC. Objects are deleted. You don’t have permissions to see the deletions (remember I said that you “lose changes” when you move an object to a place you don’t have perms to? Well normal users don’t have permissions to the deleted objects container out of the box. So it’s a very common mainstream case for this problem….). As a result, you never reflect the deletion in your target container in ADAM. You’re woefully out of date.

One fix for this problem could be that you just give more perms. In the deleted objects case, just give the user who is syncing permissions to read the deleted objects container. But some people might not find that acceptable, depending upon their scenario.

Enter aging. We wrote aging to be a periodic background thread that goes and checks to make sure objects which we haven’t seen change in a while are actually still there. So you can imagine that every now and then you go back and check to ensure that all objects you have in the target are still in the source. This is the aging approach. While the specifics are configurable, that’s the basic idea.

Aging is just one such mechanism. There are lots of approaches to this problem that one could consider. It’s just the one we chose for adamsync.

One minor point I’ll raise before ending this post. Aging in ADAMSync in R2 is unfortunately not working properly. There is a bug that basically breaks it in some cases. It’s hard to say when but you should assume you’ll hit it at some point….no idea if you will, but you never know. So if you need aging pre-LH (ie, you have a compelling scenario where you want to sync as a non-admin) please open a QFE request with PSS. Or just give perms to deleted objects for now (or whatever the container is which you can't see)…a much easier quick-fix.

(just updated some formatting)

Finding the lost&found container in S.DS.P...or anything that isn't ADSI really

efleis — Fri, 27 Oct 2006 01:46:00 GMT

I found myself writing a piece of C# which would go hunt for objects in lost&found today. This is a pretty straight forward task….find that container, pop in to it and search away. I usually do this by looking at the lost&found well known GUID (which is GUID_LOSTANDFOUND_CONTAINER_W in the platform SDK) then just crafting the search by hand.

Anyway, I was feeling particularly lazy today so I went to take a quick look at MSDN and just use their sample. Much to my surprise the only examples I could find did this via ADSI and talked about “binding to well-known objects using WKGUID.” (I’ve NEVER been a fan of the use of terms there…I don’t like how we say “binding to an object” in ADSI as that is a somewhat unnatural construct given what’s really going on under the hood. Further, it makes moving to wldap32/S.DS.P/etc. harder as the terms are different. If only I ruled the world.....) Given this, I figured I’d paste some sample code here.

First, the LDAP work itself. To do a search for L&F, you want to craft a search as follows:

Base DN: <WKGUID=GUID_LOSTANDFOUND_CONTAINER_W,dc=someNC,dc=com>

Search filter: (objectclass=*)

Search scope: Base

To dissect that baseDN some…..WKGUID means “well known GUID”, instead of the string I pasted in italics (GUID_LOSTANDFOUND_CONTAINER_W) you would of course want to actually use the GUID there, and then after the comma you put the DN of the naming context you wish to search (for example, dc=mydomain,dc=com).

So here’s a quick and dirty piece of C# written against System.DirectoryServices.Protocols APIs that would get this DN for you. Please note that this is sample code so you really want to robustify it some before actually putting this in an application, and clean up the little foreach I have there just because I’m being lazy. ;)

string lAndFBaseDN = String.Format("<WKGUID={0},{1}>", GlobalVars.GUID_LOSTANDFOUND_CONTAINER_W, myNCDN);

string[] attrList = {"dn"};

string searchFilter = “(objectclass=*)”

SearchRequest sr = new SearchRequest(lAndFBaseDN,searchFilter,SearchScope.Base,attrList);

SearchResponse searchResponse = (SearchResponse)myLdapConnection.SendRequest(sr);

if (searchResponse.Entries.Count == 0)

{

// Must be no l&f container in the target dn (like, target not an NC for example)

}

Debug.Assert(searchResponse.Entries.Count == 1, "More than one L&F container found. This is weird.");

// If there is more than one, we just return the first. But there should not be.

foreach (SearchResultEntry sre in searchResponse.Entries)

{

return sre.DistinguishedName;

}

I think it goes w/o saying but you can apply similar logic to other well known GUIDs. I just picked l&f as it was convenient. ;)

Housekeeping (again)

efleis — Thu, 19 Oct 2006 23:25:00 GMT

I've created a new adamsync tag and updated all of the adamsync posts to point there (in addition to normal tagging). A few people have asked me for this so I figured, what the heck. Apparently people use this blog as an adamsync bootstrapping system...who knew. Maybe I'll post some more on it with new things not yet mentioned....I'll try to do that this week.

Garbage collection & TSL warnings...why now?

efleis — Thu, 19 Oct 2006 23:17:00 GMT

I was recently pinged by a friend who is rolling out LH in their production environment. They were having an interesting issue where the LH DC showed these two events, in this order:

(event log entries snipped some for brevity)

Log Name: Directory Service

Source: NTDS General

Event ID: 1859

Task Category: Garbage Collection

Level: Warning

User: ANONYMOUS LOGON

Description:

Internal event: The current garbage collection interval is larger than the maximum value.

Current garbage collection interval (hours):

40000

Maximum value:

168

New value:

168

Log Name: Directory Service

Source: NTDS General

Event ID: 1088

Task Category: Internal Configuration

Level: Warning

User: ANONYMOUS LOGON

Description:

Internal event: The following tombstone lifetime registry value is too low or incompatible with the following garbage collection interval specified in the Active Directory Domain Services Configuration object. As a result, the following default registry values for the tombstone lifetime and garbage collection will be used.

This is an old domain which has been around for years. The win2k3 boxes are chugging along without a problem…only the LH boxes were throwing this event. So, the question is, what’s going on?

I took a peak at this for this friend and came up with the following. Thought I’d share it in case others see this too.

First event, the GC one. Well, the event told you the problem. J The GC interval is set to 40000 hours. That’s roughly “a long time.” GC interval is huge, we didn’t like it. We enforce in the product that GC is no greater than a week, in hours. So we went ahead and, in memory, set this to a week.

NOTE: We do not change the setting in the DS, we simply ignore it and use our own. So the setting stays wrong in AD.

Next event, the TSL event. This is the more interesting event. Well, we check a few things here. First, we make your TSL is greater than the min….check, this customer had a value which was ok (I forget, but it was somewhere in the teens). Next we check if the TSL is greater than 3 times the GC interval. You failed. Your GC interval is now 7 days (remember before I said it would set it to a week as your GC interval was nonsensically huge) but your TSL is not 3 times that. So we freak and show you the event. We further set TSL to the default in code, which is 60 days. Again, no config change, just in memory change.

Last item…why now? What changed? Why did this customer not get this before on 2k3? No values changed, after all.

This code is largely identical except for a small change. In 2k3 the code for this event firing code throws the event if GC logging is set to 2 or above. In LH, it fires the event if logging is set to 0 or above. Since all of the customer’s machines were set to 0, this explains why 2k3 didn’t throw an event whereas LH did.

(Personally, I’m happy about this change…I think we should have always flagged this condition. It’s weird, we should have always drawn your attention to it.)

Mr. T likes a double-shot, who knew

efleis — Fri, 07 Jul 2006 22:02:00 GMT

I remember watching The A-Team well as a kid. It was my sat evening ritual.
Anyway, I read this post about someone bumping in to him in Starbucks and nearly lost it.

The story of squeaky lobster

efleis — Fri, 16 Jun 2006 00:42:00 GMT

I am rarely as excited about a blog post as I am about this one. Brett has finally told the story of squeaky lobster. Thankfully, it is documented for all of time!
I was at the fateful dinner where it was revealed that a crush was involved. I remember it like it was yesterday...

Large AD database? Probably not this large...

efleis — Fri, 09 Jun 2006 01:32:00 GMT

Over the last few months there have been a series of threads in regard to max <fill in the item here...there have been many> in a database. These items have ranged from database size to # of objects and other such things. I figured, after the latest thread over on activedir.og, I'd do a little testing and put some numbers behind it so we could say "we have done this" and not "the system should do this."

What should this testing accomplish?

First, raw DB size. Gotta create a big DB or it probably doesn't matter.

Next, # of objects. For my testing, this was the real metric I was interested in. As mentioned over on ActiveDir (I would provide a link to the thread but I can’t seem to get the mail archives to work right now…I’ll try and provide one later), there is a theoretical max # of objects in the lifetime of a database which is, all said and done, 2^31 objects. I wanted to shoot for this. After all, Dean asked what error you would get, and I didn’t know. :)

I wrote a tool which started banging against an ADAM SP1 x64 instance. It was creating pretty small objects as I wanted reduce the amt of time this test took. My objects looked like this:

dn: cn=leafcontX,cn=parentcontY,cn=objectsZ,ou=objdata

changetype: add

objectclass: container

(Of course, sub in values for X, Y and Z as appropriate)

I had it use anywhere from 16 to 40 threads for this work depending upon the phase of import, and I simply wrapped around ldifde for it….I figured, there is a well tested tool for this, why not let it do most of the hard work?

Next, I got my hands on a test box (thx EEC!), put it on a SAN, installed ADAM, and away I went.

Along the way, we did a few other perf tests (looking at increased checkpoint depths and the like) so it added a bit of time to the import. However, after about a month, I had nearly filled my 2TB partition:

06/08/2006 10:41 AM 2,196,927,299,584 adamntds.dit

I created just shy of 2^31 objects. When I went to create that next object (done here by hand in LDP to illustrate the error)…

***Calling Add...

ldap_add_s(ld, "cn=sample1,OU=ObjData", [1] attrs)

Error: Add: Operations Error. <1>

Server error: 000020EF: SvcErr: DSID-0208044C, problem 5012 (DIR_ERROR), data -1076

If you look up -1076, you’ll find it is JET_errOutOfAutoincrementValues (from esent98.h). Woo hoo! I ran out of DNTs.

With this DB in hand, it was time to find out what else works and what else does not…

- Promotion of a replica fails. This makes perfect sense….it tries to create a couple of objects in the config NC, and that fails.

- Create of an NC fails. Again, to be expected, this task consumes DNTs.

- I ran esentutl /ms. It chugged for nearly 30 seconds, but worked perfectly.

- I also ran esentutl /k to make sure the DB did not have any physical corruption, but also to just see how long that took. :)

- Other standard tasks (kicking off garbage collection, online defrag, restarting the service, etc.) all worked perfectly.

- Search works like a champ. Sure it takes a good bit of I/O for most interesting searches, but that’s to be expected, of course.

It is worth noting that anything which failed did so gracefully. There were no nastygrams in my event logs either.

So for those of you who are worrying….you can sleep well at night now. We have tried rolling over DNT, and it works just fine.

A fun stat…..from the esentutl /ms output:

Name Type ObjidFDP PgnoFDP PriExt Owned Available

==============================================================

<EFleis – snip to save some space>

nc_guid_Index Idx 25 43 1-m 10870892 5

That owned number is in pages. That’s right, my NC_GUID index is 82.9GB…bigger than most databases. :)

While there were no major issues, we (Brett was looking at this too) did hit a few bumps along the way, and Brett was kind enough to write a few ESE tools for me to help monitor how we were doing. I’ll outline all of these things over the next few days as I have time to write them up. I’ll also provide more clarity around specific of what we did and saw as we went along.

ArrayList and Synchronized()...the misleading method

efleis — Thu, 01 Jun 2006 16:45:00 GMT

I recently found myself wondering about synchronization when using an ArrayList in managed code. I was building up an ArrayList and iterating through it. Both insertion into and iteration over the list are being done by many threads simultaneously. Naturally, I wanted to ensure my code was thread safe.

I started out by looking at the ArrayList class definition and immediately noticed the Synchronized() method. Great! This must be what I’m looking for.

So, I carried on with my programming…..

ArrayList foo = new ArrayList();

ArrayList fooSync = ArrayList.Synchronized(foo);

Then in other threads I am doing:

fooSync.Add(someObject);

And finally, there is another set of threads taking something off of the front of the list as illustrated below:

if (fooSync.Count>0)

{

Object bar = fooSync[0];

fooSync.Remove(0);

// Do something with bar

}

(This is of course a trivial version of the algorithm, but it illustrates the point)

And viola! We have misused Synchronized().

ArrayList does not guarantee that your code is thread safe, even when you have a Synchronized() version of your ArrayList. Rather, it only guarantees that the internal data structures are. This of course is logical, but that’s really what this code is assuming. Said differently, a Synchronized ArrayList guarantees that doing things like Add() and Remove() do not corrupt the list. However, anything you do above and beyond that is not protected.

In this example, we have a block of code where I am checking if there is anyone in the list, and if so doing something with the first element, then removing it. This is not thread safe. Imagine there is one element in the list and two threads. Thread1 goes in and sees that Count>0, so it starts going through the loop. Before it gets to Remove(0), it gets preempted, and Thread2 starts. Thread2 also sees that something is in the list…namely the element that Thread1 is already operating over! It too starts doing something with object bar. And well, who knows what happens from here. Maybe this is bad enough in your app (doing the action on bar twice).

(It might sound bad that this could AV, but just think of an even worse scenario. Imagine this instead….another thread has, while they are doing their thing, done an insertion in to the list, and that one is now getting removed before it is even being operated on! Now the app will continue but silently lose an object that was not operated on at all.)

Of course, this makes perfect sense. The ArrayList developers could write a version of their class which protects their data structures, but how could they possibly make this scenario work? Short of some sort of language-level contract around where atomicity might be preserved (I’m not a language guy, but I can’t see how that would work, though I don’t deny it might be possible), they can’t possibly solve this problem. How do they know where atomicity is assumed in my code?

Moral of the story: Always make sure you understand your class & method contracts.