How many times have you came across reports of users complaining that they cannot logon to the domain?

Logon fails with "The trust relationship between this workstation and the primary domain failed."

Additionally the NETLOGON service also logs:

Event ID 5723

"The session setup from the computer DOMAINMEMBER failed to authenticate.
The name of the account referenced in the security database is DOMAINMEMBER$.

The following error occurred: Access is denied."

What do you have to do in those cases?

Exactly... I heard you saying "Disjoin and rejoin the machine to the domain"

Some of you may wonder why we can’t just reset the computer account password in AD and fix the problem.

Well by resetting the computer account password, AD sets the password to the default value and the probability of the member having the same password is practically null.

Every domain member has a secure channel (SC) with a domain controller, this SC is created and mantained by the NETLOGON service on both member and DC.
(Like users, computers also have passwords which are managed automatically).

If the SC gets broken the member won't be able to authenticate in the domain.

Note:

A secure channel is an authenticated remote procedure call (RPC) connection between two machines in a domain with an established security context used for signing and encrypting RPC packets.

For the SC to be successfully established, the computer account password (stored in Active Directory) and the member's password (known as LSA secret and stored in the member) have to be in sync.

The member has to change its password every 30 days by default.

This setting is controlled by "Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options\Domain member: Maximum machine account password age" using Local computer policies or domain GPOs.

Note:
If you increase this interval significantly, or set it to 0 so that the computers no longer change their passwords, an attacker will have more time to undertake a brute force attack to guess the password of one or more computer accounts.

For more information on recommended security settings see Microsoft Security Compliance Manager (SCM) which you can download from: http://www.microsoft.com/en-us/download/details.aspx?id=16776

So you might be asking what can go wrong if the member's password is automatically changed and managed.

One thing that is very important to keep in mind is that the requirements for a machine to change its password has changed in versions later than Windows XP/Server 2003.

Later OSes only change their password if they can contact a DC whereas Windows XP/Server 2003 change their password even without being able to contact a DC.

Which breaks the secure channel immediately.

Let me give you 3 examples for clarity sake:

Example 1:

Joe laptop (running XP) changes its password on Nov 15 (and syncs with AD)

Joe goes on leave and takes his laptop with him on Dec 1

On Dec 15 while Joe was preparing his agenda for his 1st day of work, his laptop decides to change its password without being able to notify the DC

Joe returns on Dec 16

when he tries to logon he gets the message mentioned above - and you get a phone call :)

Example 2:

Same situation but now Joe closes his laptop and leaves it on his desk.

When he returns notices his laptop has been stolen...

please ignore previous line :)

starts his laptop and logs on without problems.

The computer password gets reset later on the same day and life goes on.

Example 3:

Wisely the IT department has upgraded all laptops to Windows 8.

So when Joe returns from leave experiences no problems logging on to the domain because his computer only changes its password if it can contact a DC.

By the way if you read my previous blog, mind the Windows XP support lifecycle note.

Recently I came across an issue that has been puzzling System Admins minds for some time.

Since deployment of Windows 7 that machines in training rooms where "re-imaged" as needed via SCCM.

However broken secure channels where happening daily on many of them.

You may ask how that can happen if Windows 7 doesn't change the password unless it can communicate with a DC.

Well cutting a long story short, after enabling netlogon logging on the DCs (http://support.microsoft.com/kb/109626) and analysing a number of netlogon logs, I ended up finding a number of Windows XP machines that no one was aware of and that didn't even appear in SCCM since they were not managed via SCCM.

Basically found out that a particular branch was still using the old re-imaging process with Windows XP images using the same names as the Windows 7 machines, thus using and changing the password of Windows 7 members.

So once a Windows 7 member tries to authenticate it would fail and that was happening on an endless loop, because the Windows 7 admins would disjoin/re-jointhe Windows 7 box (thus fixing the SC problem) and then the XP admin would do the same on the XP machine.

This post is a bit long now so I will add another one for the ones interested in knowing that SC issues may also occur on DCs and Domain trusts.
Additionally, tools we can use for troubleshooting.

As always you are very welcome to add your comments!

Thanks

Paulo

While there can be several reasons for AD replication to fail due to an "access denied" error (you may find more information in KB article 2002013 - Troubleshooting AD Replication error 5: Access is denied http://support.microsoft.com/kb/2002013/EN-US),
in here we will be focusing on broken secure channel issues on Domain Controllers and how to reset them.
Then as promised in my previous post we will also have a look at tools/commands you may use to verify and reset secure channels and trusts.

A replication issue may actually be the cause for a broken secure channel between DCs in the first place.
Another common cause may be restoring from backup a DC to a previous point in time (or to a system restore point in case of client machines) or time jumps which don't cause the broken secure channel directly but may seen be as a colateral damage.
(by the way take a look at http://blogs.technet.com/b/askpfeplat/archive/2012/11/19/did-your-active-directory-domain-time-just-jump-to-the-year-2000.aspx and http://blogs.technet.com/b/askpfeplat/archive/2012/11/26/fixing-when-your-domain-traveled-back-in-time-the-great-system-time-rollback-to-the-year-2000.aspx
which have been updated recently).

To reset the Secure Channel of a Domain Controller you have 2 methods, one requires reboot the other doesn't however requires more tasks to be completed.

If restarting the DC is an issue for you due to business, availability or change control requirements do the following:

1. Set the Kerberos Key Distribution Center (KDC) service startup type to Disabled, and restart the domain controller
(particularly important if you have more than one DC on the same domain, so this way you will force the affected DC to contact another DC for kerberos authentication, instead of using itself)

2. Once the DC restarts, reset the secure channel:

netdom resetpwd /server: /userd:\ /passwordd:*
(you should get the foolowing message: "The machine account password for the local machine has been successfully reset")

3. Then, set the startup type for the KDC service back to Automatic and restart the DC once again.

Note: Do not start the service manually as it will be started automatically during reboot.

The 2nd way is:

1. Delete any mapped drives that might exist on the affected DC, for that run:

net use * /d

2. Stop the KDC service.

3. Purge all kerberos tickets from the computer credentials cache (this is the memory space were the computer stores its tickets, there's nothing in common with cached credentials).

Depending on the Operating System run the following:

On Windows Server 2003:

at /interactive cmd.exe
(check current system time and add a couple of extra minutes, for example if current time is 14:14 type 14:16)

Note:
Another command prompt will launch under the System account context (SVCHost.exe )on the Console session of the problem DC at the time you entered (14:16 in the example above).
If running the command via RDP ensure that you are connected to session 0 (mstsc /console or /admin depending on mstsc version) or you won't be able to see the 2nd command prompt.

klist purge

On Windows Server 2008 or above:

klist -li 0x3e7 purge

Note:

The -li 0x3e7 gets klist into the computer credentials cache so there's no need for the "at" command.

4. Reset the Secure Channel:

netdom resetpwd /server: /userd:\ /passwordd:*
(You should get the following message "The machine account password for the local machine has been successfully reset"

 
net use \\\IPC$
(The DC will request new kerberos tickets)

5. Force the DC to replicate from/to the PDCemulator (so both ways first inbound then outbound):

repadmin /replicate <destination> <source> <NC> /force

example:
repadmin /replicate DC1 DC2 DC=Contoso,DC=com /force
(you should get the following message "sync from DC2 (the PDCe) to DC1 (the affected DC) completed successfully")
repadmin /replicate DC2 DC1 DC=Contoso,DC=com /force
(you should get the following message "sync from DC1 (the affected DC) to DC2 (the PDCe) completed successfully")

Note:
If the affected DC successfully replicates from the PDCe then start the KDC service. If not successful, you will have to use the method the requires restart.
If replication to the PDCe fails, delete the old outbound connection object and trigger KCC (repadmin /kcc) in order to rebuild them, and force outbound replication again. (this might be required for COs with other replication partners)

Done.

In order to verify the health status of secure channels you may use Nltest.exe or Netdom.exe (which is also used for trusts).

If using Nltest run:

Nltest /sc_verify:<DomainName>
You should get "Trust Verification Status = 0 0x0 NERR_Success" if not you must reset the secure channel.

Note:
This command doesn't work on a PDCe, so please don't reset the PDCe secure channel once you run it thus getting an error message.
Don't use /sc_query to verify secure channels because it doesn't verify the SC, it just tells you the information about the last established SC. (so you will get a valid answer even when you cannot reach a DC)

If using Netdom:

netdom verify /domain:<DomainName>

In order to verify Trusts:
(Trusts work in a similar way as Secure Channels, there is a TDO (Trust Domain Object) maintained in each trusting and trusted domain partition, which password has to be in sync, of not the trust gets broken).

netdom trust /Domain: /verify

Note:
Please take into account that depending on security settings between domains this command may fail, for more troubleshooting information see the article below:
Client, service, and program issues can occur if you change security settings and user rights assignments
http://support.microsoft.com/kb/823659

Hope it has been useful for you. Feel free to share your thoughts.

In a number of ocasions have been asked to configure services running under the svchost process to pass through the Firewall on Windows XP.
Particularly for WMI access as administrators and applications might need it for performing data collection, monitoring and other administrative tasks.

The problem is that Windows Firewall on XP doesn’t allow to add SVCHOST.exe as a Program exception (the reason behind it is that other services run under SVCHOST and there are several SVCHOST processes running at the same time).
An example is wmimgmt which runs along with AudioSrv, Browser, CryptSvc, etc, under the same SVCHOST process.
Additionally the RPC ports used by the services running under the SVCHOST processes are assigned dynamically by default.

This is very easy to configure if you are using Windows Firewall with Advanced Security on clients running Windows Vista or above, on XP is a diferent story though.
By the way the Mainstream Support phase ended on 2009/04/14 and the Extended Support phase will end on 2014/04/08.
Which means you have had 5 years to plan and upgrade to a newer client OS.
If haven't done so yet I strongly recommend you to start right away (You may get more info on the support lifecycle for Windows XP in http://support.microsoft.com/lifecycle/?LN=en-gb&C2=1173).

Note:
When using GPOs to configure firewall settings XP settings should be configured under Computer Configuration/Administrative Templates/Network/Network Connections/Windows Firewall.
Computer Configurations/Windows Settings/Security Settings/Windows Firewall with Advanced Security only applies to Windows Vista or above.
    
So, first we need to configure WMI to run on a separate SVCHOST process:

To configure Windows Management Instrumentation Service (wmimgmt) to run under a separate SVCHOST process.
1.Install Hotfix that is mentioned in http://support.microsoft.com/kb/897571
2.Click Start, click Run, type Cmd, and then click OK.
3.At a command prompt, type net stop winmgmt, and then press ENTER.
4.Click Start, click Run, type Notepad, and then click OK.
5.Copy the following code into Notepad. 
 
Windows Registry Editor Version 5.00
 
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\SvcHost]
"Winmgmt"=hex(7):77,00,69,00,6e,00,6d,00,67,00,6d,00,74,00,00,00,00,00
 
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\SvcHost\winmgmt]
"CoInitializeSecurityParam"=dword:00000001
"AuthenticationCapabilities"=dword:00003020
"CoInitializeSecurityAppID"="{D16904E8-7F7D-4821-ACF5-FDE160CBE65E}"
 
[HKEY_CLASSES_ROOT\AppID\{D16904E8-7F7D-4821-ACF5-FDE160CBE65E}]
@="Svchost_winmgmt"
"EndPoints"=hex(7):6e,00,63,00,61,00,63,00,6e,00,5f,00,69,00,70,00,5f,00,74,00,\
63,00,70,00,2c,00,30,00,2c,00,34,00,33,00,32,00,31,00,00,00,00,00

 
Note:
In this code we configure the port number 4321 as an example.
You may use Dcomcnfg.exe tool to set the port number later.
 
6.Save the file that you created in step 5. Name the file Winmgmtsvc.reg
7.Double-click Winmgmtsvc.reg, click Yes to add the information to the registry, and then click OK.
8.Click Start, click Run, type sc config winmgmt binPath= "%systemroot%\system32\svchost.exe -k winmgmt", and then click OK.
 
Note:
This command configures the WMI service to run in a separate process.
In this command, the quotation marks are required.
In this command, the space after binPath= is required.
 
 "Path to executable"

Now to configure wmimgmt to use a static endpoint. (in order to change the port used in the example above ie. 4321)

9.Click Start, click Run, type dcomcnfg , and then click OK.
10.Under Console Root, expand Component Services, expand Computers, expand My Computer, and then expand DCOM Config.
11.Right-click Svchost_winmgmt, and then click Properties.
12.On the Endpoints tab, click Add.
13.Click Static endpoint, configure the static endpoint that you want, and then click OK two times.
14.Click Start, click Run, type Cmd, and then click OK.
15.At a command prompt, type net start winmgmt, and then press ENTER.
 
Note:

This specifies a static port instead of the WMI default dynamic port behavior.
The static endpoint will not be visible by using Netstat.exe until a remote WMI request is sent to the server.
 

Finally configure “Define Inbound Port Exception” under Computer Configuration/Administrative Templates/Network/Network Connections/Windows Firewall/Domain profile for the same port configured on the action above.
The settings should look as follows:
Windows Firewall: Protect all network connections  Enabled
Windows Firewall: Define port exceptions
135:TCP:*:enabled:EPM-DCOM135
4321:TCP:*:enabled:winmgmt4321 (NOTE: This is according to the example above, Should be changed to the port you selected)
 
Note:
This procedure is described on KB897571 and although the article doesn’t refer to Windows XP, the same hotfix is also available for XP.
hotfix - fix230723 is REQUIRED, without the hotfix these settings are ignored.
You might be able to use the same concept for other services running under SVCHOST.

Important: I strongly recommend the above to be tested for functionality and security before deploying to production.

You're more than welcome to share your views.

Thanks

Paulo
 

During my interactions with Premier Microsoft customers I have found out that the main reason for not using AGPM (Advanced Group Policy Management) in order to enforce change control procedures on Group Policy management is the lack of information on how to prevent GPOs from being created or edited outside of AGPM.
Basically their experience tells them that any Domain/GPO admin will be able to use the normal GPMC (Group Policy Management Console) to create/edit GPOs thus bypassing the desired change control enforcement supposedly provided by AGPM. I am writing these lines to address that problem.

This post explains how to install and configure AGPM in order to prevent the creation of GPOs from outside AGPM.

AGPM is a plug-in for GPMC that provides the following features:

Offline editing
Change control
Role-based delegation
Search and filter capabilities
Cross-forest management
 
AGPM can be found on the MDOP (Microsoft Desktop Optimization Pack) for Software Assurance.

To install the AGPM follow the steps below:

1. Create an AGPM Service Account
2. Add the AGPM Service Account to the following groups:
2.1. Group Policy Creator Owners
2.2. Backup Operators
2.3. Local Administrators (on the Client(s) and Server selected to install the AGPM Client and Server components respectively)
3. Install the AGPM Server component on the selected server.
4.1. Run the AGPM server installer
4.2. Select the Archive Path (can be a local folder or network share)
4.3. Select the AGPM Service Account under which the AGPM service will run.
4.4. Assign the Archive Owner/AGPM Administrator (Full Control) role to the Group (or individual User) that will have Full Control over AGPM thus will be able to assign AGPM roles and permissions to other Group Policy Administrators.
4.5. Select the port listener for AGPM service (default:4600)

NOTE: The AGPM Service Account requires (at least) the following permissions:
- Full Control to the AGPM Archive
- Full Control to %systemroot%\temp folder
- Full Control to existing GPOs

5. Install the AGPM Client component on the selected workstation.
5.1. Run AGPM client installer
5.2. Insert the AGPM Server and Port for connecting to the AGPM server service.

To prevent the creation of GPOs from outside AGPM do the following:

1. Remove All members (except the AGPM Service Account and the Archive Owner/AGPM Administrator) of the "Group Policy Creator Owners" group

To prevent changes of existing GPOs outside AGPM

1. Remove Domain Admins and Enterprise Admins permissions from every GPO in the domain.

NOTE: Domain Admins may re-add themselves permissions to all GPOs. Depending on your environment additional Groups may have to be removed from all GPOs. Ensure that Authenticated Users have Allow Read permissions and that target Groups have Allow Apply Group Policy. Also that you don't remove the AGPM Administrator account assigned in step 4.4.

In order to easy this task consider the use of GrantPermissionOnAllGPOs.wsf included in GPMCSampleScripts.msi which can be downloaded from http://www.microsoft.com/en-us/download/details.aspx?id=14536
You may also find many other useful scripts there.

From now on the AGPM Administrator may configure e-mail notifications, control policies and manage roles and delegated permissions and approve requests via AGPM client without worrying about GPO admins circumventing change control with GPMC.

Hope it helps!

Hello my name is Paulo Viralhadas and I'm a Premier Field Engineer at Microsoft.

On one of my previous posts I wrote about vDC cloning which is my preferred feature in Windows Server 2012 "http://blogs.technet.com/b/reference_point/archive/2012/12/11/so-you-wanted-to-deploy-domain-controllers-faster-now-you-can.aspx".

VDC cloning gives you the ability to scale up your production forest and to recover from disasters faster, or simply to build a lab in a blink of the eye.
 
In this post you may watch a number of videos that show how to clone a vDC.

Note: I tried to keep the videos as simple as possible for quick reference.

HOW TO CLONE A vDC

The Requirements are:
Hypervisor must have support for VMGID (VM-Generation ID).
ADDS schema version 56
Windows Server 2003 Forest Functional Level.
Source DC must be running Windows Server 2012.
PDCe must be running on a Windows Server 2012 DC.
PDCe and RID master online and available.

The video contents are:

Video 1 - Pre-requisit check.

Step 1 - Verify that the source VDC is running on a supported hypervisor.
Step 2 - Verify Schema version.
Step 3 - Verify Forest Functional Level.
Step 4 - Check if the VDC source Operating System.
Step 5 - Verify that the PDCe FSMO role is running on a Windows Server 2012 DC
Step 6 - Ensure that PDC and RID master are available during cloning process.

Video 2 - Getting the Clone ready.

Step 7 - Create DCCloneConfig.xml file.
Step 8 - Add the source VDC to the "Cloneable Domain Controllers" security group.
Step 9 - Shutdown the source VDC.

Video 3 - Cloning...

Step 10 - Export the source VM (Virtual Machine).
Step 11 - Import the VM with the option "Copy the virtual machine (create a new unique ID)".
Step 12 - Start the new VM.

Detailed steps:

Step 1 -  Verify that the source VDC is running on a supported hypervisor.

On the source vDC:
open [Device Manager]
expand [System Devices]
open properties of [Microsoft Hyper-V Generation Counter]
select the "Driver" tab
click "Driver details"
verify that the driver is "vmgencounter.sys"
This is the driver that makes vDC cloning and snapshot restore possible in Windows Server 2012.

Step 2 - Verify Schema version.

On any DC in the forest:
run [regedit]
browse to HKLM\System\CCS\Services\NTDS\Parameters
verify that "Schema Version" REG_DWORD value is 56.
This is the Windows Server 2012 version of the schema.

Step 3 - Verify Forest Functional Level.

On any DC in the forest:
open [Powershell]
run [Get-ADForest]
verify that "ForestMode" value is "Windows2003Forest" or higher.

Step 4 - Check the vDC source Operating System.

On the source vDC:
run [winver]
verify that source vDC is a Windows Server 2012.

Step 5 - Verify that the PDCe FSMO role is running on a Windows Server 2012 DC

On any DC in the domain:
open [cmd]
run [netdom query fsmo]
copy the PDC FQDN
open [Powershell]
run [Get-ADDomainController -server <paste the PDC FQDN here>
verify that OperatingSystemVersion value is 6.2 (9200) or higher

Step 6 - Ensure that PDC and RID master are available during cloning process.

Step 7 - Create DCCloneConfig.xml file.

open [Powershell]
run [New-ADDCCloneconfigFile]
(this will create an empty configuration file, you might want to have a look on the table below before you add
 configuration information to this file)

Step 8 - Add the source VDC to the "Cloneable Domain Controllers" security group.

open [ADAC]
browse your domain to the "Users" container
double-click "Cloneable Domain Controllers" security group
Select "Members" tab and click "Add" button to add the source domain controller account

Step 9 - Shutdown the source VDC.

Step 10 - Export the source VM (Virtual Machine).

open [Hyper-V Manager]
Right-click the source vDC VM
Select Export
Specify where you want to save the files

Step 11 - Import the VM with the option "Copy the virtual machine (create a new unique ID)".

open [Hyper-V Manager]
click on "Import Virtual Machine"
Locate Folder
Select Virtual Machine
Choose import type:  "Copy the virtual machine (create a new unique ID)"

Step 12 - Start the new VM.

(Refer to the diagram below in order to understand the cloning/snapshot restore decision process)

The Cloning/Snapshot safeguards are:

 The following table puts together the outcomes of the diagram above:

By the way you may find a playlist of all 3 videos above at: 

http://www.youtube.com/playlist?list=PLRiiq9ROPBOtJhPx2SciZcMfhJ4PN4K7y 

Hope it helps!

 Best regards

Paulo

Hello my name is Paulo Viralhadas and I'm a Premier Field Engineer at Microsoft.

The AD recycle bin feature has been released on Windows Server 2008 R2 without a graphical user interface, which made it's deployment and usability (I mean recovering deleted objects from AD) somewhat difficult for system admins.

In this post I will write about how to enable the ADRB feature on both WS2008 and WS2012.

This will provide you with the skills necessary to perform object recovery regardless of the operating system you are using currently.

Be amazed on how easy it is to recover deleted objects in WS2012.

This feature can be enabled if your forest is running at WIN2008R2 functional level.

If you already have all DCs in the forest running on Windows Server 2008R2 or higher you may use the following powershell command to raise the FFL:

Set-ADForestMode 4 -Identity <forestname>

which requires that all domains in the forest run at WIN2008R2 domain functional level, so if needed run:

Set-ADDomainMode 4 -Identity <domainname>

Before running the powershell commands above and if using WS2008R2 you have to import the Active Directory module for powershell (WS2012 does it automatically).

Important: Enabling Active Directory Recycle Bin is an irreversible procedure.

To enable the AD Recycle Bin feature using powershell run:

Enable-ADOptionalFeature –Identity ‘CN=Recycle Bin Feature,CN=Optional Features,CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration,DC=contoso,DC=com’ –Scope ForestOrConfigurationSet –Target ‘<forestname>’

To check if recycle bin is enabled:

Get-ADOptionalFeature -filter *

To restore a deleted object do the following:

Get-ADObject -Filter {displayName -eq "name"} -IncludeDeletedObjects | Restore-ADObject

Watch the video - Enable AD Recycle Bin and restore a single object using powershell on WS2008R2

Although restoring a single object is fairly simple, the restoration procedures get more complex when restoring multiple nested objects.

For example when you need to restore an OU with hundreds or thousands of objects in it (like users, groups, computers or even sub OUs which in its turn may contain more objects).

You will have to investigate how deep OU nesting is by querying deleted users lastknownparent attribute in order to understand the deleted OU structure.

Then you must start restoring from top to bottom so one hierarchy level at a time.

The reason behind is that when an object is deleted and moves to the deleted objects container, the object distinguished name (DN) gets mangled and the deleted objects container doesn't contain/keep an hierarchy.

The only way to return the objects hierarchy is by searching for it's lastknownparent attribute.

For example consider the following scenario:

[picture 1]

In this scenario we need to start by searching for known objects that have been deleted, for example in this case by finding the lastknownparent of a user named Peter:

Get-ADObject -SearchBase "CN=Deleted Objects,DC=contoso,DC=com" -ldapFilter:"(msDs-lastKnownRDN=Peter)" – IncludeDeletedObjects –Properties lastKnownParent

Then by finding all objects that have HumanResources as lastknowparent (note: add another \ before 0ADEL:):

Get-ADObject –SearchBase "CN=Deleted Objects,DC=contoso,DC=com" -Filter {lastKnownParent -eq 'OU=HumanResources\ \0ADEL:c876daac-da9b-57ce-bded-978aed9c0e2b,CN=Deleted Objects,DC=contoso,DC=com'} -IncludeDeletedObjects - Properties lastKnownParent | ft

At this stage we must go through the output and look for other OUs contained within HumanResources OU, then we need to search for other deleted objects inside it in case we find extra OUs (in this case we explore the Users OU within HumanResources):

Get-ADObject –SearchBase "CN=Deleted Objects,DC=contoso,DC=com" -Filter {lastKnownParent -eq 'OU=Users\ \0ADEL:6b507c43-172b-8145-93bf-61e00302bb4a,CN=Deleted Objects,DC=contoso,DC=com'} -IncludeDeletedObjects - Properties lastKnownParent | ft

So far we found that HumanResources OU has been deleted (by noticing the mangled DN) and with the following command we may validate if there is any other OU above it and if it was also deleted (if so we have to perform the same steps as above to find other objects within it):

Get-ADObject -SearchBase "CN=Deleted Objects,DC=contoso,DC=com" -ldapFilter:"(msDs-lastKnownRDN=HumanResources)" –IncludeDeletedObjects –Properties lastKnownParent

After investigation is complete, then we can start restoring the objects from top to bottom:

to restore the HumanResources OU:

Get-ADObject -ldapFilter:"(msDS-LastKnownRDN=HumanResources)" –IncludeDeletedObjects | Restore-ADObject

to restore all deleted objects under it (including the Users OU):

Get-ADObject -SearchBase "CN=Deleted Objects,DC=contoso,DC=com" -Filter {lastKnownParent -eq "OU=HumanResources,DC=contoso,DC=com"} -IncludeDeletedObjects | Restore-ADObject

To restore all deleted objects under the Users OU:

Get-ADObject -SearchBase "CN=Deleted Objects,DC=contoso,DC=com" -Filter {lastKnownParent -eq "OU=Users,OU=HumanResources,DC=contoso,DC=com"} -IncludeDeletedObjects | Restore-ADObject

Note: Depending on your infrastructure you may have to go deeper into OU hierarchy, but then you just have to repeat the steps above accordingly.

Watch the video - Restore multiple objects using powershell on WS2008R2

Now that we covered how to restore deleted objects using the recycle bin attribute in WS2008R2 lets see how easy it is to do the same in WS2012 with the new recycle bin GUI:

You have to open the Deleted Objects container using ADAC and perform the same searches on it just by using the UI and by working out the hierarchy by looking at the Last Known Parent attribute.

If looking for specific objects we may click add criteria and in case we don't know exactly what to search for -this is one of the reasons why having AD proper documented is so important - an idea would be to add "and Last modified between these dates:")

Watch the video - Restore multiple objects using the NEW Recycle Bin GUI in WS2012

Hope it helps!

Enjoy!

PS: I will add the videos throughout the week.

In my next post I will share more information on how actually the AD recycle bin works.

A Domain Controller must have a unique name, invocation ID, and security identifier (SID) in the entire forest.
Up to Windows Server 2008 R2 promoting "syspreped" standalone images multiple times, was the fastest you could go in order to deploy a large number of Domain Controllers.
Sysprep was needed for ensuring that the deployed images were unique. Yes, scripts and answer files could be used for unnattended deployment in order to minimize administrative ovehead and deployment times.

As you could guess from my previous post, Microsoft has introduced new features in Hyper-V and Windows Server 2012 that allow to do things with Virtualized DCs that before were impossible.
This has became possible due to the introduction of an identifier called VM-Generation ID on the Hypervisor driver and the corresponding attribute in AD (msDS-GenerationID).

The feature I want to tell you about is VDC Cloning.

Now a Cloned Windows Server 2012 Domain Controller automatically executes Sysprep and uses the existing NTDS and SYSVOL data for promotion (similar to IFM - Install from Media). Additionally, you may create a configuration file (DCCloneConfig.xml).
Then just need to copy or export the source VDC and it's done.
You get faster deployment, scalability and easier disaster recovery.

Security wise, Domain Admins select DCs that can be cloned.
The Hyper-V administrators can only deploy the approved cloned DCs thus ensuring that unauthorized users cannot create rogue DCs.

The following table exemplifies how VDC Cloning is achieved:

So in a nutshell cloning succeeds the following will happen:

If both IDs don't match and a DCCloneConfig.xml file exists. (If xml file doesn't exist then VDC performs Snapshot safe restore - please refer to my previous blog for more information).

DSRM boot flag is set (to prevent boot into production in case cloning fails)

Then VDC checks for VDCisCloning DWORD under NTDS parameters reg key, and if it doesn't exist NTDS invalidates RID pool and resets Invocation ID; if exists then cloning has failed before (and went to DSRM as a safety mechanism) and this a re-attempt of cloning.

The Cloned VM gets the configuration settings (from DCCloneConfig.xml) and all ADDS related services are stopped.

Time synchronization is enforced (using DOMHIER) and Kerberos tickets are purged.

All FRS/DFSR database files are deleted (Not the SYSVOL content which will be used later for pre-seeding in order to minimize SYSVOL replication traffic) and services set to start automatically.

The cloned VM is renamed and DC promotion starts using the existing NTDS.DIT file as source (which will minimize replication traffic later).

Gets a new RID pool from the RID Master.

New Invocation ID and NTDS settings are created.

Starts inbound replication.

FRS/DFSR services start and non-authoritative restore of the SYSVOL is triggered.

Enables DNS client registration.

Runs Sysprep (DefaultDCCloneAllowList.xml determines which Sysprep modules are run)

Once promotion completes the DSRM boot flag is removed, the DCCloneConfig is renamed (will prevent it from being read during reboot),

the VDCisCloning DWORD removed and sets "VDC Cloning Done" to 1.

The value of VM-GenerationID (from HV) is copied to the VDC computer account VM-Generation ID (msDS-GenerationID attribute).

Finally once the VDC reboots it starts advertising itself as a DC.

Note: Currently, this feature only works on Windows Server 2012 Hypervisor or HV2012.

Hope this helps on your future DC deployments and feel free to comment.

Thanks

Paulo

The USN rollback issue has been causing hundreds of support calls and AD replication halts throughout the world since the introduction of AD in Windows 2000 Server and up to Windows Server 2008 R2.

Every DC maintains a table - ReplUpToDateVector - (or Up-to-Dateness Vector) per Naming Context (NC or AD partition).
These tables record data from the local DC and its replication partners, this data includes the uuidDSA (or DSA GUID); usnHighPropUpdate (or High Watermark) and timeLastSyncSuccess (or the time stamp of last successfull replication from that replication partner) for a particular partition.

When a change is made (ie. an object is created or deleted, an attribute of an object is modified) one (or more) attribute(s) will have their Originating and Local USN incremented.

Example:

repadmin /showobjmeta * OU=USNROLLBACK,DC=contoso,DC=com

Additionally, the ReplUptoDateVector table (UTDVEC table from now on) on the local DC for itself will be updated.

screenshot of repadmin /showutdvec * dc=contoso,dc=com

Normal operation, In this case there is no outstanding replication from ContosoDC1 to ContosoDC2 (values of ContosoDC1 match on both DCs; on the other hand ContosoDC1 will have to replicate from ContosoDC2 66 changes (220356-220290)

Then a replication partner will compare its own version of the table and requests the changes that are higher than the High Watermark from the source.

If the USN for the DC on the replication partners is higher than the one the DC has for itself, you are dealing with a USN Rollback.

Example:

screenshot of repadmin /showutdvec ( relevant USNs highlighted)

USN Rollback. Note that ContosoDC1 "thinks" that ContosoDC2 has a higher "High Watermark" than in reality. So without the USN rollback protection mechanism the next 182 (220380-220198) changes originated on ContosoDC2 will be discarded by ContosoDC1

In that case the originating DC will log Replication Event ID 2095 in Directory Service log and will disable inbound and outbound replication as a protection mechanism in order to avoid further damage.

screenshot of event 2095

Without this safety valve further changes held on the originating DC will never be replicated, and eventually only when the originating DC catches-up with the USN known by its replication partners will start replicating again, however any changes in between are lost forever.

NOTE: Ensure that "Allow replication with corrupt and divergent partners" is not in use, or this protection will be ignored.
 
In order to avoid this issue you should backup your DCs using a supported method, that is an AD aware backup application (NTBackup and Windows Backup are AD aware as so other 3rd party backup applications).
What you should NOT do as a replacement for the applications above is to restore AD from unsupported backup methods like:

Disk Mirroring
Cloning
VHD copies
VM snapshots
or any other cloning method that doesn't reset the DSA Invocation ID when an AD restore is executed.

Note:

The DSA invocation ID is reset once you restore AD using a supported method. Thus replication partners will update their UTDVEC tables with the new value for the restored DC. This doesn't happen when using unsupported methods.

To fix the problem there are two supported methods:

1. Reinstall Active Directory on the affected Domain Controller.

Transfer any FSMO roles if needed.
Demote the DC.
Perform a metadata clean-up of all references to the DC.
Re-promote the DC.

2. Restore the System State.

If a valid system state backup was made before the DC was restored from one unsupported method. Restore the system state from the most recent backup.

For more information on how AD replication works and USN rollback please refer to the following articles:

How the Active Directory Replication Model Works
http://technet.microsoft.com/en-us/library/cc772726(WS.10).aspx

Running Domain Controllers in Hyper-V
http://technet.microsoft.com/en-us/library/d2cae85b-41ac-497f-8cd1-5fbaa6740ffe(v=WS.10)#usn_and_usn_rollback

In Windows Server 2012 virtualized Domain Controllers, you can now restore snapshots without permanently damage domain controllers.
While this does not prevent other issues for other technologies and applications, it does make domain controller virtualization safer.

Now virtualized domain controller snapshot restore resets the DC's unique Invocation ID.
Additionally discards the local RID pool and non-authoritatively restores the SYSVOL folder. 
This means that accidentally restoring a snapshot is no longer an unsafe operation on domain controllers.

The following process describes how Virtualized DC (VDC) Safe Restore is achieved:

1. Restore of an existing virtual machine (VM) domain controller from a snapshot in a hypervisor that supports VM-Generation ID (Windows Server 2012 Hyper-V for instance).

Assuming that this VM already has an existing VM Generation-ID on its DC computer object when the snapshot was taken as part of the msDS-GenerationID attribute (Schema version 56).

2. The VM then reads the VM-Generation ID provided by Hyper-V VMGenerationCounter driver and compares the VM-Generation IDs values.

If they do not match, it continues with snapshot restoration operations.
Once restored, the Generation-ID set on the DC computer object (in AD) is updated to match the new ID provide by the hypervisor host.

If the hypervisor does not provide a VM-Generation ID for comparison, the guest will operate like a Windows Server 2008 R2 or earlier virtualized domain controller.

3. The Virtualized DC then:

Invalidates the local RID pool.
Sets a new DSA invocation ID.

4. Non-authoritative inbound replication is triggered  from a replication partner. The DC requests changes starting at a USN that precedes the USN at which the local directory service was restored. The UTDVEC table of the destination DC is updated appropriately.

5. The virtualized DC synchronizes the SYSVOL:

If using FRS, it stops the NTFRS service and sets the BURFLAGS registry value (D2).
It then starts the NTFRS service, thus performing a non-authoritative restore of the SYSVOL.

If using DFSR, it stops the DFSR service and deletes the DFSR database files. It then starts the DFSR service, thus performing a non-authoritative restore of the SYSVOL.

6. The VM updates the msDS-GenerationID attribute on its own DC object to match the current Hypervisor VM-Generation ID

If you haven't got started with Windows Server 2012, download Windows Server 2012 from:
http://technet.microsoft.com/en-us/evalcenter/hh670538.aspx

To finish this post as a personal note, and from all improvements in Windows Server 2012 this my preferred feature. And I think that whomever imagined it deserves a big round of applause from all of us.

Feel free to leave your comments.

Cheers.

Hello my name is Paulo Viralhadas and I'm a Premier Field Engineer at Microsoft.

Have you ever used Fine Grained Password Policies?

This feature introduced in Windows Server 2008 allows you to override password policy set at the domain level.

It applies password settings to subsets of users that you may like to differentiate from the domain policy.

In Windows Server 2012 we added a GUI (Graphical User Interface) so now you don't have to use ADSIedit, LDP or Powershell to create PSOs (Password Settings Objects).

Note that PSOs are not like GPOs:

1. They're not managed via GPMC.

2. They're not linked to OUs, Sites or Domains.

PSOs apply to User and Group objects (ie. ultimately apply to User Accounts)

As an example, with FGPP you can have a Domain password policy that defines a minimum password length of 8 characters which will be applied to all users in the domain.

Then have a PSO that sets 24 characters for all user accounts that are members of the "All Service Accounts".

I added the following video that walks you through the steps needed to implement this (once again I've kept it short and simple and no sound).

Anyway here's the high level steps you have to follow:

1. Using ADAC (Windows 8 or Server 2012) open the Password Settings container (under System container).

2. Add a New PSO (Password Settings Object).

3. Configure the desired PSO properties (Max password Age, Min Password Length, etc).

4. Assign the PSO to a user or group

Hope it helps!

Enjoy!

Recently I have been working with a premier customer in South Africa to cleanup their forest from lingering objects.

It is a complex environment with 15 domains,30+ sites and 130+ DCs where power failures and network related issues frequently disrupt AD operations.

So I wanted to share with you the method I used to remove lingering objects and hopefully you find it useful somehow.

First download repldiag from http://activedirectoryutils.codeplex.com/releases/view/13664 as it will save you a lot of typing.

Then create a .bat file similar to the following:

/enable Strict Replication Consistency across the forest

repadmin /regkey * +strict

/dump all repldiag commands to a .txt file (repldiag enumerates all domains, finds all DCs and creates all necessary repadmin commands for removing lingering objects)

repldiag /removelingeringobjects /outputrepadmincommandlinesyntax >output.txt

/dump domain specific repldiag commands to a .bat file (this way you may cleanup one domain at a time)

findstr "domain_a" output.txt >domain_a.bat
(...)
findstr "domain_n" output.txt >domain_n.bat

/call all .bat files in order to remove lingering objects

call domain_a.bat
(...)
call domain_n.bat

/Then run repadmin /showrepl against all DCs in each domain and pipe it to a .txt file

repadmin /showrepl *.domain_a /errorsonly > domain_a.txt
(...)
repadmin /showrepl *.domain_n /errorsonly> domain_n.txt

/Look for "failed, result 8606 (0x219e): Insufficient attributes were given to create an object. This object may not exist because it may have been deleted and already garbage collected."

/Identify the destination DC (the DC logging the event), Source DC (the DC that contains lingering objects) and affected Naming Context (the NC that contains lingering objects)

/With the information above force replication to resume on the affected DCs/NCs

repadmin /replicate destination sourceGUID NC /force

as an example:

**output from showrepl**

Repadmin: running command /SHOWREPL against full DC DC1.domain_a.fqdn

site1\DC1

DSA Options: IS_GC

Site Options: (none)

DSA object GUID: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa

DSA invocationID: zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz

==== INBOUND NEIGHBORS ======================================

DC=domain_n,DC=fqdn

    Site30\DC via RPC

        DSA object GUID: nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn

        Last attempt @ 2014-10-16 10:55:42 failed, result 8606 (0x219e):

            Insufficient attributes were given to create an object. This object may not exist because it may have been deleted and already garbage collected.

        529 consecutive failure(s).

        Last success @ 2014-09-30 07:00:33.

In this example the command to force replication to resume would be

repadmin /replicate DC1.domain_a.fqdn  nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn DC=domain_n,DC=fqdn /force

Hope it helps!

Paulo