Cluster and Stale Computer Accounts

Cluster and Stale Computer Accounts

  • Comments 15
  • Likes

Hi, Mike here again. Today, I want to write about a common administrative task that can lead to disaster: removing stale computer accounts from Active Directory.

Removing stale computer accounts is simply good hygiene-- it’s the brushing and flossing of Active Directory. Like tartar, computer accounts have the tendency to build up until they become a problem (difficult to identify and remove, and can lead to lengthy backup times).

Oops… my bad

Many environments separate administrative roles. The Active Directory administrator is not the Cluster Administrator. Each role holder performs their duties in a somewhat isolated manner-- the Cluster admins do their thing and the AD admins do theirs. The AD admin cares about removing stale computer accounts. The cluster admin does not… until the AD admin accidentally deletes a computer account associated with a functioning Failover Cluster because it looks like a stale account.

Unexpected deletion of Cluster Name Object (CNO) or Virtual computer Object (VCO) is one of the top issues worked by our engineers that support Clustering and High-Availability. Everyone does their job and boom-- Clustered Servers stop working because CNOs or the VCOs are missing. What to do?

What's wrong here

I'll paraphrase an article posted on the Clustering and High-Availability TechNet blog that solves this scenario. Typically, domain admins key on two different attributes to determine if a computer account is stale: pwdlastSet and LastLogonTimeStamp. Domains that are not configured to a Window Server 2003 Domain Functional Level use the pwdLastAttribute. However, domains configured to a Windows Server 2003 Domain Functional Level or later should use the lastLogonTimeStamp attribute. What you may not know is that a Failover Cluster (CNO and VCO) does not update the lastLogonTimeStamp the same way as a real computer.

Cluster updates the lastLogonTimeStamp when it brings a clustered network name resource online. Once online, it caches the authentication token. Therefore, a clustered network named resource working in production for months will never update the lastLogonTimeStamp. This appears as a stale computer account to the AD administrator. Being a good citizen, the AD administrator deletes the stale computer account that has not logged on in months. Oops.

The Solution

There are few things that you can do to avoid this situation.

  • Use the servicePrincipalName attribute in addition to the lastLogonTimeStamp attribute when determining stale computer accounts. If any variation of MSClusterVirtualServer appears in this attribute, then leave the computer account alone and consult with the cluster administrator.
  • Encourage the Cluster administrator to use -CleanupAD to delete the computer accounts they are not using after they destroy a cluster.
  • If you are using Windows Server 2008 R2, then consider implementing the Active Directory Recycle Bin. The concept is identical to the recycle bin for the file system, but for AD objects. The following ASKDS blogs can help you evaluate if AD Recycle Bin is a good option for your environment.

Mike "Four out of Five AD admins recommend ASKDS" Stephens

  • Great post as usual guys!  Just wanted to offer up a few automated methods for the actual cleanup.  We are fortunate to have everyone agreed upon containing our "special" objects in specific OUs.  This allows for some simple exceptions when getting the computers.  You can make use of the still great OldCmp from Joeware (www.joeware.net/.../oldcmp)  However, I use some simple PowerShell scripts to get the job done these days.  Below is a stripped down version of the script, which the output is ultimately passed to Remove-ADComputer (The actual script bundles up and emails the output too...)

    Param

    (

       [int]$MaxDays = 45,

       [array]$OUExceptions = @("Cluster Objects","SAN Objects")

    )

    $Now = Get-Date

    $OUExceptions += "Domain Controllers"

    [regex]$OUExceptions_regex = ‘(‘ + (($OUExceptions |foreach-object  {[regex]::escape($_)}) –join “|”) + ‘)’

    $Computers = Get-ADComputer -filter * -Properties Name, PwdLastSet, CanonicalName

    $Computers = $Computers | Select-object Name, @{Name="Age"; Expression={($now - ([datetime]::FromFileTime($_.pwdlastset))).days}},@{Name="pwdlastset"; Expression= {[datetime]::FromFileTime($_.pwdlastset)}},CanonicalName

    $Computers = $Computers | where-object {$_.Age -gt $MaxDays}

    $Computers = $Computers | Where-Object {$_.CanonicalName -notmatch $OUExceptions_regex}

    Write-Output $Computers

  • Great article Mike.  But this scenario will only occur if you are using Windows 2000/2003 cluster.  Correct?  

    In Windows 2008/2008 R2, cluster will rotate the CNO and VCO passwords.  So it will be like a regular computer object inside AD.

  • Yes, but that means you have to rely on pwdlastset being set - and that is not guaranteed to ever change. A computer or cluster does not have to do it if it has local security policy enabled to not change password (the domain password policy is irrelevant). So using the SPN as a failsafe is always the safe approach.

  • Thanks Ned.  When the password for CNO or VCO is changed based on the domain MaximumPasswordAge policy, the credentials will get updated and logon will happen.  That process should update the LastLogonTimeStamp?

  • Domain policy does not affect computers - only the local setting on every computer. It appears the author of that cluster post is partially misunderstanding how computer account password operates. Computers change their passwword if and when *they* feel like it. The domain has nothing to do with it at all.

    This is controlled by:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters

    DisablePasswordChange

    MaximumPasswordAge

    RefusePasswordChange

    If those aren't set, the default is for a computer to *tell* the domain it's new password every 30 days. If it cannot tell it, or if it doesn;t feel like it, nothing happens in the domain - the DCs do not care about the computers and they will not be locked out or restricted.

    Since the cluster is piggybacking on the computer's own password settings according to that post, they will behave the same way. It's not clear from the article what will happen if you had differnet nodes configured with differnet password settings, you should ask them about that.

    And yes, if they do change their password, that will require they generate a new Kerberos AS_REQ to logon - and that means they get a new lastlogontimestamp.

    See below

  • Ah - and to be clear - there are policies for those registry settings. They are not the "domain password policy" settings though, as you will see. They are specific to netlogon and computers, and do nothing to users.

  • Santhosh,

    Let's remember that password change does not mean logon. True, the cluster service manages CNO and VCO passwords. Additionally, it honors computer password security policy settings.  However, the LastlogonTimeStamp is only updated when a network name resource comes online (logon).  If that resource continues to work for months, then the LastLogonTimeStamp does not update, and gives the perception of a stale computer account.

    Changing a password and computer logon are two distinct actions.

    Mike Stephens [MSFT]

  • Ned,

    Sorry.  I was talking about the 30 day computer password change.  Not the domain password policy.   That was my little “typo” :)  Thanks for clarifying that.  

  • Mike,

    I was under impression that the computer password change process updates the LastLogonTimeStamp.   Is that still a valid assumption?  

  • I guess it is a valid statement based on the following Ned’s comment:

    “And yes, if they do change their password, that will require they generate a new Kerberos AS_REQ to logon - and that means they get a new lastlogontimestamp”

  • And I am changing my previous answer regarding lastlogontimestamp to "it depends", as Mike was pointing out.

    For instance, my act of changing the machine account password (with nltest /sc_change_pwd) is not causing a new logon to occur, even if I flush the computer's kerberos tickets with klist -li 0x3e7 and ran a GPUPDATE /force to cause me to get a new tickets using my new computer password.

    You should not rely on lastlogontimestamp to ever change unless that computer is rebooted, which will guarantee it.

  • <sarcasm> So then reboot all systems daily is what I'm getting from that. </sarcasm> :)  Wow, this got busy fast!  All good conversation though of course... well maybe other than this post.

  • Well I can pretty much guarantee you will at least monthly...

    en.wikipedia.org/.../Patch_Tuesday

    ;-)

  • Oh - and for those interested about how lastlogontimestamp works: blogs.technet.com/.../the-lastlogontimestamp-attribute-what-it-was-designed-for-and-how-it-works.aspx

    Any questions and you can bug Warren. :-D

  • Thanks Mike and Ned for clarifying these items.  Great conversation..