Blog - Title

February, 2012

  • Purging Old NT Security Protocols

    Hi folks, Ned here again (with some friends). Everyone knows that Kerberos is Microsoft’s preeminent security protocol and that NTLM is both inefficient and, in some iterations, not strong enough to avoid concerted attack. NTLM V2 using complex passwords stands up well to common hash cracking tools like Cain and Abel, Ophcrack, or John the Ripper. On the other hand, NTLM V1 is defeated far faster and LM is effectively no protection at all.

    I discussed NTLM auditing years ago, when Windows 7 and Windows Server 2008 R2 introduced the concept of NTLM blocking. That article was for well-controlled environments where you thought that there was some chance of disabling NTLM – only modern clients and servers, the latest applications, and Active Directory. In a few other articles, I gave some further details on the limitations of the Windows auditing system logging. It turns out that while we’re ok at telling when NTLM was used, we’re not great at describing which flavor. For instance, Windows Server 2008+security auditing can tell you about the NTLM version through the 4624 event that states a Package Name (NTLM only): NTLM V1 or Package Name (NTLM only): NTLM V2, but all prior operating systems cannot. None of the older auditing can tell you if LM is used either. Windows Server 2008 R2 NTLM auditing only shows you NTLM usage in general.

    Today the troika of Dave, Jonathan, and Ned are here to help you discover which computers and applications are using NTLM V1 and LM security, regardless of your operating system. It’s safe to say that some people aren’t going to like our answers or how much work this entails, but that’s life; when LM security was created as part of LAN Manager and OS/2 by Microsoft and IBM, Dave and I were in grade school and Jonathan was only 48.

    If you need to keep using NTLM V2 and simply want to hunt down the less secure precursors, this should help.

    Finding NTLM V1 and LM Usage via network captures

    The only universal, OS-agnostic way you can tell which clients are sending NTLMv1 and LM challenges is by examining a network trace taken from destination computers. Using Netmon 3.4 or another network capture tool, look for packets with a negotiated NTLM security mechanism.

    This first example is with LMCompatibilityLevel set to 0 on clients. This example is an SMB session request packet, specifying NTLM authentication.

    Here is the SMB SESSION SETUP request, which specifies the security token mechanism:

      Frame: Number = 15, Captured Frame Length = 220, MediaType = ETHERNET

    + Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-15-5D-05-B4-44],SourceAddress:[00-15-5D-05-B4-49]

    + Ipv4: Src = 10.10.10.20, Dest = 10.10.10.27, Next Protocol = TCP, Packet ID = 747, Total IP Length = 206

    + Tcp: Flags=...AP..., SrcPort=49235, DstPort=Microsoft-DS(445), PayloadLen=166, Seq=2204022974 - 2204023140, Ack=820542383, Win=32724 (scale factor 0x2) = 130896

    + SMBOverTCP: Length = 162

    - SMB2: C   SESSION SETUP (0x1)

        SMBIdentifier: SMB

      + SMB2Header: C SESSION SETUP (0x1),TID=0x0000, MID=0x0002, PID=0xFEFF, SID=0x0000

      - CSessionSetup:

         StructureSize: 25 (0x19)

         VcNumber: 0 (0x0)

       + SecurityMode: 1 (0x1)

       + Capabilities: 0x1

         Channel: 0 (0x0)

         SecurityBufferOffset: 88 (0x58)

         SecurityBufferLength: 74 (0x4A)

         PreviousSessionId: 0 (0x0)

       - securityBlob:

        - GSSAPI:

         - InitialContextToken:

          + ApplicationHeader:

          + ThisMech: SpnegoToken (1.3.6.1.5.5.2)

          - InnerContextToken: 0x1

           - SpnegoToken: 0x1

            + ChoiceTag:

            - NegTokenInit:

             + SequenceHeader:

             + Tag0:

             + MechTypes: Prefer NLMP (1.3.6.1.4.1.311.2.2.10)

             + Tag2:

             + OctetStringHeader:

             -MechToken: NTLM NEGOTIATE MESSAGE

              - NLMP: NTLM NEGOTIATE MESSAGE

                 Signature: NTLMSSP

                 MessageType: Negotiate Message (0x00000001)

               + NegotiateFlags: 0xE2088297 (NTLM v2128-bit encryption, Always Sign)

               + DomainNameFields: Length: 0, Offset: 0

               + WorkstationFields: Length: 0, Offset: 0

               + Version: Windows 6.1 Build 7601 NLMPv15

    Next, the server sends its NTLM challenge back to the client:

      Frame: Number = 16, Captured Frame Length = 447, MediaType = ETHERNET

    + Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-15-5D-05-B4-49],SourceAddress:[00-15-5D-05-B4-44]

    + Ipv4: Src = 10.10.10.27, Dest = 10.10.10.20, Next Protocol = TCP, Packet ID = 24310, Total IP Length = 433

    + Tcp: Flags=...AP..., SrcPort=Microsoft-DS(445), DstPort=49235, PayloadLen=393, Seq=820542383 - 820542776, Ack=2204023140, Win=512 (scale factor 0x8) = 131072

    + SMBOverTCP: Length = 389

    - SMB2: R  - NT Status: System - Error, Code = (22) STATUS_MORE_PROCESSING_REQUIRED  SESSION SETUP (0x1), SessionFlags=0x0

        SMBIdentifier: SMB

      + SMB2Header: R SESSION SETUP (0x1),TID=0x0000, MID=0x0002, PID=0xFEFF, SID=0x0019

      - RSessionSetup:

         StructureSize: 9 (0x9)

       + SessionFlags: 0x0

         SecurityBufferOffset: 72 (0x48)

         SecurityBufferLength: 317 (0x13D)

       - securityBlob:

        - GSSAPI:

         - NegotiationToken:

          + ChoiceTag:

          - NegTokenResp:

           + SequenceHeader:

           + Tag0:

           + NegState: accept-incomplete (1)

           + Tag1:

           + SupportedMech: NLMP (1.3.6.1.4.1.311.2.2.10)

           + Tag2:

           + OctetStringHeader:

           - ResponseToken: NTLM CHALLENGE MESSAGE

            - NLMP: NTLM CHALLENGE MESSAGE

              Signature: NTLMSSP

               MessageType: Challenge Message (0x00000002)

            + TargetNameFields: Length: 12, Offset: 56

             + NegotiateFlags: 0xE2898215 (NTLM v2128-bit encryption, Always Sign)

             + ServerChallenge: 67F9C5F851F2CD73

               Reserved: Binary Large Object (8 Bytes)

             + TargetInfoFields: Length: 214, Offset: 68

             + Version: Windows 6.1 Build 7601 NLMPv15

               TargetNameString: CORP01

             + AvPairs: 7 pairs

    The client calculates the response to the challenge, using the various available hashes of the password. Note how this response includes both LM and NTLMv1 challenge responses.

      Frame: Number = 17, Captured Frame Length = 401, MediaType = ETHERNET

    + Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-15-5D-05-B4-44],SourceAddress:[00-15-5D-05-B4-49]

    + Ipv4: Src = 10.10.10.20, Dest = 10.10.10.27, Next Protocol = TCP, Packet ID = 748, Total IP Length = 387

    + Tcp: Flags=...AP..., SrcPort=49235, DstPort=Microsoft-DS(445), PayloadLen=347, Seq=2204023140 - 2204023487, Ack=820542776, Win=32625 (scale factor 0x2) = 130500

    + SMBOverTCP: Length = 343

    - SMB2: C   SESSION SETUP (0x1)

        SMBIdentifier: SMB

      + SMB2Header: C SESSION SETUP (0x1),TID=0x0000, MID=0x0003, PID=0xFEFF, SID=0x0019

      - CSessionSetup:

         StructureSize: 25 (0x19)

         VcNumber: 0 (0x0)

       + SecurityMode: 1 (0x1)

       + Capabilities: 0x1

         Channel: 0 (0x0)

       SecurityBufferOffset: 88 (0x58)

         SecurityBufferLength: 255 (0xFF)

         PreviousSessionId: 0 (0x0)

       - securityBlob:

        - GSSAPI:

         - NegotiationToken:

          + ChoiceTag:

          - NegTokenResp:

           + SequenceHeader:

           + Tag0:

           + NegState: accept-incomplete (1)

           + Tag2:

           + OctetStringHeader:

           - ResponseToken: NTLM AUTHENTICATE MESSAGEVersion:v1, Domain: CORP01, User: Administrator, Workstation: CONTOSO-CLI-01

            - NLMP: NTLM AUTHENTICATE MESSAGEVersion:v1, Domain: CORP01, User: Administrator, Workstation: CONTOSO-CLI-01

               Signature: NTLMSSP

               MessageType: Authenticate Message (0x00000003)

             + LmChallengeResponseFields: Length: 24, Offset: 154

             + NtChallengeResponseFields: Length: 24, Offset: 178

             + DomainNameFields: Length: 12, Offset: 88

             + UserNameFields: Length: 26, Offset: 100

             + WorkstationFields: Length: 28, Offset: 126

             + EncryptedRandomSessionKeyFields: Length: 16, Offset: 202

             + NegotiateFlags: 0xE2888215 (NTLM v2128-bit encryption, Always Sign)

             + Version: Windows 6.1 Build 7601 NLMPv15

             + MessageIntegrityCheckNotPresent: 6243C42AF68F9DFE30BD31BFC722B4C0

               DomainNameString: CORP01

               UserNameString: Administrator

               WorkstationString: CONTOSO-CLI-01

             + LmChallengeResponseStruct: 3995E087245B6F7100000000000000000000000000000000

             + NTLMV1ChallengeResponse: B0751BDCB116BA5737A51962328D5CCD19EEBEBB15A69B1E

             + SessionKeyString: 397DACB158C9F10EF4903F10D4CBE032

           + Tag3:

           + OctetStringHeader:

           + MechListMic: Version: 1

    The server then responds with successful negotiation state:

      Frame: Number = 18, Captured Frame Length = 159, MediaType = ETHERNET

    + Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-15-5D-05-B4-49],SourceAddress:[00-15-5D-05-B4-44]

    + Ipv4: Src = 10.10.10.27, Dest = 10.10.10.20, Next Protocol = TCP, Packet ID = 24312, Total IP Length = 145

    + Tcp: Flags=...AP..., SrcPort=Microsoft-DS(445), DstPort=49235, PayloadLen=105, Seq=820542776 - 820542881, Ack=2204023487, Win=510 (scale factor 0x8) = 130560

    + SMBOverTCP: Length = 101

    - SMB2: R   SESSION SETUP (0x1), SessionFlags=0x0

        SMBIdentifier: SMB

      + SMB2Header: R SESSION SETUP (0x1),TID=0x0000, MID=0x0003, PID=0xFEFF, SID=0x0019

      - RSessionSetup:

         StructureSize: 9 (0x9)

       + SessionFlags: 0x0

         SecurityBufferOffset: 72 (0x48)

         SecurityBufferLength: 29 (0x1D)

       - securityBlob:

        - GSSAPI:

         - NegotiationToken:

          + ChoiceTag:

          - NegTokenResp:

           + SequenceHeader:

           + Tag0:

           +NegState: accept-completed (0)

           + Tag3:

           + OctetStringHeader:

           + MechListMic: Version: 1

    To contrast this, consider the challenge response packet when LMCompatibility is set to 4 or 5 on the client (meaning it is not allowed to send anything but NTLM V2). The LM response is null, while the NTLMv1 response isn't included at all.

      Frame: Number = 17, Captured Frame Length = 763, MediaType = ETHERNET

    + Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-15-5D-05-B4-44],SourceAddress:[00-15-5D-05-B4-49]

    + Ipv4: Src = 10.10.10.20, Dest = 10.10.10.27, Next Protocol = TCP, Packet ID = 844, Total IP Length = 749

    + Tcp: Flags=...AP..., SrcPort=49231, DstPort=Microsoft-DS(445), PayloadLen=709, Seq=4045369997 - 4045370706, Ack=881301203, Win=32625 (scale factor 0x2) = 130500

    + SMBOverTCP: Length = 705

    - SMB2: C   SESSION SETUP (0x1)

        SMBIdentifier: SMB

      + SMB2Header: C SESSION SETUP (0x1),TID=0x0000, MID=0x0003, PID=0xFEFF, SID=0x0021

      - CSessionSetup:

         StructureSize: 25 (0x19)

         VcNumber: 0 (0x0)

      + SecurityMode: 1 (0x1)

       + Capabilities: 0x1

         Channel: 0 (0x0)

         SecurityBufferOffset: 88 (0x58)

         SecurityBufferLength: 617 (0x269)

         PreviousSessionId: 0 (0x0)

       - securityBlob:

        - GSSAPI:

         - NegotiationToken:

          + ChoiceTag:

          - NegTokenResp:

           + SequenceHeader:

           + Tag0:

           + NegState: accept-incomplete (1)

           + Tag2:

           + OctetStringHeader:

           - ResponseToken: NTLM AUTHENTICATE MESSAGEVersion:v2, Domain: CORP01, User: Administrator, Workstation: CONTOSO-CLI-01

            - NLMP: NTLM AUTHENTICATE MESSAGEVersion:v2, Domain: CORP01, User: Administrator, Workstation: CONTOSO-CLI-01

               Signature: NTLMSSP

               MessageType: Authenticate Message (0x00000003)

             + LmChallengeResponseFields: Length: 24, Offset: 154

             + NtChallengeResponseFields: Length: 382, Offset: 178

             + DomainNameFields: Length: 12, Offset: 88

             + UserNameFields: Length: 26, Offset: 100

             + WorkstationFields: Length: 28, Offset: 126

             + EncryptedRandomSessionKeyFields: Length: 16, Offset: 560

             + NegotiateFlags: 0xE2888215 (NTLM v2128-bit encryption, Always Sign)

             + Version: Windows 6.1 Build 7601 NLMPv15

             + MessageIntegrityCheck: 2B69C069DD922D4A841D0EC43939DF0F

               DomainNameString: CORP01

               UserNameString: Administrator

               WorkstationString: CONTOSO-CLI-01

             + LmChallengeResponseStruct: 000000000000000000000000000000000000000000000000

             + NTLMV2ChallengeResponse: CD22D7CC09140E02C3D8A5AB623899A8

             + SessionKeyString: AF31EDFAAF8F38D1900D7FBBDCB43760

           + Tag3:

           + OctetStringHeader:

           + MechListMic: Version: 1

    By taking traces and filtering on the NTLMV1ChallengeResponse field, you find those hosts that are sending NTLMv1 responses and determine if you need to upgrade them or if they simply have the wrong LMcompatibility values set through security policy.

    Finding LM usage via Netlogon debug logs

    If you just want to detect LM authentication and not looking to spend time in network captures, you can instead enable Netlogon logging on all DCs and servers in the environment.

    Nltest /dbflag:2080ffff
    net stop NetLogon
    net start NetLogon

    This creates the netlogon.log in the C:\Windows\Debug folder and it can grow to a maximum of 20 Mb by default. At that point, the server renames the file to netlogon.bak and a new netlogon.log file started. At 20Mb, the server deletes netlogon.bak, renames the netlogon.log to netlogon.bak, and a new netlogon.log file started. To make these log files larger, you can use a registry entry or group policy:

    Registry

    Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters
    Value Name: MaximumLogFileSize
    Value Type: REG_DWORD
    Value Data: <maximum log file size in bytes>

    Group Policy

    \Computer Configuration\Administrative Templates\System\Net Logon\Maximum Log File Size

    You aren't trying to capture all data here - just useful samples - but if they wrap so much that you're unsure if they are accurate at all, increasing size is a good idea. As an alternative, you can create a scheduled task that runs ONSTART or a computer startup script. Either of them can use this batch file to make backups of the netlogon log by date/time and the computer name:

    REM Sample script to copy the netlogon.bak to a netlogon_DATETIME_COMPUTERNAME.log backup form every 5 minutes

    :start
    if exist %windir%\debug\netlogon.bak goto copylog

    :
    copylog_return
    sleep 300
    goto start

    :copylog
    for /f "tokens=1-7 delims=/:., " %%a in ("%DATE% %TIME%") do (set DATETIME=%%a-%%b-%%c_%%d-%%e-%%f)
    copy /v %windir%\debug\netlogon.bak %windir%\debug\netlogon_%DATETIME%_%COMPUTERNAME%.log
    if %ERRORLEVEL% EQU 0 del %windir%\debug\netlogon.bak
    goto copylog_return

    Periodically, gather all of the NetLogon logs from the DCs and servers and place them in a single folder. Once you have assembled the NetLogon logs into a single spot, you may then use the following LogParser command from that folder to parse them all for a count of unique UAS logons to the domain controller by workstation:

    Logparser.exe "SELECT TO_UPPERCASE(EXTRACT_SUFFIX(TEXT,0,'returns ')) AS ERR, TO_UPPERCASE (extract_prefix(extract_suffix(TEXT, 0, 'NetrLogonUasLogon of '), 0, 'from ')) as USER, TO_UPPERCASE (extract_prefix(extract_suffix(TEXT, 0, 'from '), 0, 'returns ')) as WORKSTATION, COUNT(*) FROM '*netlogon.*' WHERE INDEX_OF(TO_UPPERCASE (TEXT),'LOGON') >0 AND INDEX_OF(TO_UPPERCASE(TEXT),'RETURNS') >0 AND INDEX_OF(TO_UPPERCASE(TEXT),'NETRLOGONUASLOGON') >0 GROUP BY ERR, USER, WORKSTATION ORDER BY COUNT(*) DESC" -i:TEXTLINE -rtp:-1 >UASLOGON_USER_BY_WORKSTATION.txt

    UASLOGON_USER_BY_WORKSTATION.txt contains the unique computers and counts. LogParser is available for download from here.

    FIND and PowerShell are options here as well. The simplest approach is just to return the lines, perhaps into a text file for later sorting in say, Excel (which is very fast at sorting and allows you to organize your data).

    image

    image

    image

    I'll wager someone in the comments will take on the rather boring challenge of exactly duplicating what LogParser does. I didn't have the energy this time around. :)

    Final thoughts

    Microsoft stopped using LM after Windows 95/98/ME. If you do find specific LM-only usage and you don't have any (unsupported) Win9X computers, this is a third party application. A really heinous one.

    All supported versions of Windows obey the LMCompatibility registry setting, and can use NTLMv2 just as easily as NTLMv1. At that point, analyzing network traces just becomes useful for tracking down those hosts that have applied the policy, but have not yet been rebooted. Considering how unsafe LM and NTLMv1 are, enabling NoLMHash and LMCompatibility 4 or 5 on all computers may be a faster alternative to auditing. It could cause some temporary outages, but would definitely catch anyone requiring unsafe protocols. There's no better auditing that a complaining application administrator.

    Finally, do not limit your NTLM inventory to domain controllers and file or application servers. A comprehensive project requires you examine all computers in the environment, as even a Windows XP workstation can be a "server" for some application. Use a multi-pronged approach, where you also inventory operating systems through network probing - if you have Windows 95 or old SAMBA lying around somewhere on a shop floor, they are almost guaranteed to use insecure protocols.

    Until next time,

    - Ned “and Dave and Jonathan and Jonathan's in-home elderly care nurse” Pyle

  • Friday Mail Sack: Get Off My Lawn Edition

    Hi folks, Ned here again. I know this is supposed to be the Friday Mail Sack but things got a little hectic and... ah heck, it doesn't need explaining, you're in IT. This week - with help from the ever-crotchety Jonathan Stephens - we talk about:

    Now that Jonathan's Rascal Scooter has finished charging, on to the Q & A.

    Question

    We want to create a group policy for an OU that contains various computers needs to run for just Windows 7 notebooks only. All of our notebooks are named starting with an "N". Does group policy WMI filtering allows stacking conditions on the same group policy? 

    Answer

    Yes, you can chain together multiple query criteria, and they can even be from different classes or namespaces. For example, here I use both the Win32_OperatingSystem and Win32_ComputerSystem classes:

    image

    And here I use only the Win32_OperatingSystem class, with multiple filter criteria:

    image

    As long as they all evaluate TRUE, you get the policy. If you had a hundred of these criteria (please don’t) and 99 evaluate true but just one is false, the policy is skipped.

    Note that my examples above would catch Win2008 R2 servers also; if you’ve read my previous posts, you know that you can also limit queries to client operating systems using the Win32_OperatingSystem property OperatingSystemSKU. Moreover, if you hadn’t used a predictable naming convention, you can also filter on with Win32_SystemEnclosure and query the ChassisTypes property for 8, 9, or 10 (respectively: “Portable”, “Laptop”, and “Notebook”). And no, I do not know the difference between these, it is OEM-specific. Just like “pizza box” is for servers. You stay classy, WMI.

    Question

    Is changing LDAP MaxPoolThreads a good or bad idea?

    Answer

    MaxPoolThreads controls the maximum number of simultaneous threads per-processor that a DC uses to work on LDAP requests. By default, it’s four per processor core. Increasing this value would allow a DC/GC to handle more LDAP requests. So if you have too many LDAP clients talking to too few DCs at once, raising this can reduce LDAP application timeouts and periodic “hangs”. As you might have guessed, the biggest complainer here is often MS Exchange and Outlook. If the performance counters “ATQ Threads LDAP" & "ATQ Threads Total" are constantly at the maximum number based on the number of processor and MaxPoolThreads value, then you are bottlenecking LDAP.

    However!

    DCs are already optimized to quickly return data from LDAP requests. If your hardware is even vaguely new and if you are not seeing actual issues, you should not increase this default value. MaxPoolThreads depends on non-paged pool memory, which on a Win2003 32-bit Windows OS is limited to 256MB (more on Win2008 32-bit). Meaning that if you still have not moved to at least x64 Windows Server 2003, don’t touch this value at all – you can easily hang your DCs. It also means you need to get with the times; we stopped making a 32-bit server OS nearly three years ago and OEMS stopped selling the hardware even before that. A 64-bit system's non-paged pool limit is 128GB.

    In addition, changing the LDAP settings is often a Band-Aid that doesn’t address the real issue of DC capacity for your client/server base.  Use SPA or AD Data Collector sets to determine "Clients with the Most CPU Usage" under section "Ldap Requests”. Especially if the LDAP queries are not just frequent but also gross - there are also built-in diagnostics logs to find poorly-written requests:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics\
    15 Field Engineering

    To categorize search operations as expensive or inefficient, two DWORD registry keys are used:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\
    Expensive Search Results Threshold

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\
    Inefficient Search Results Threshold

    These DWORD registry keys have the following default values:

    • Expensive Search Results Threshold: 10000
    • Inefficient Search Results Threshold: 1000

    For example, here’s an inefficient result written in the DS event log; yuck, ick, argh!:

    Event Type: Information
    Event Source: NTDS General
    Event Category: Field Engineering
    Event ID: 1644
    Description:
    The Search operation based at RootDSE
    using the filter:
    & ( | ( & ( (objectCategory = <val>) (objectSid = *) ! ( (sAMAccountType | <bit_val>) ) ) & ( (objectCategory = <val>) ! ( (objectSid = *) ) ) & ( (objectCategory = <val>) (groupType | <bit_val>) ) ) (aNR = <substr>) <startSubstr>*) )

    visited 40 entries and returned 0 entries.

    Finally, this article should be required reading to any application developers in your company:

    Creating More Efficient Microsoft Active Directory-Enabled Applications -
    http://msdn.microsoft.com/en-us/library/windows/desktop/ms808539.aspx#efficientadapps_topic04

    (The title should be altered to “Creating even slightly efficient…” in my experience).

    Question

    I want to implement many-to-one certificate mappings by using Issuer and Subject DN match. In altSecurityIdentities I put the following string:

    X509:<I>DC=com,DC=contoso,CN=Contoso CA<S>DC=com,DC=contoso,CN=users,CN=user name

    In a given example, a certificate with “cn=user name, cn=users, dc=contoso, dc=com” in the Subject field will be mapped to a user account, where I define the mappings. But in that example I get one-to-one mapping. Can I use wildcards here, say:

    X509:<I>DC=com,DC=contoso,CN=Contoso CA<S>DC=com,DC=contoso,CN=users,CN=*

    So that any certificate that contains “cn=<any value>, cn=users, dc=contoso, dc=com” will be mapped to the same user account?

    Answer

    [Sent from Jonathan while standing in the 4PM dinner line at Bob Evans]

    Unfortunately, no. All that would do is map a certificate with a wildcard subject to that account. The only type of one-to-many mapping supported by the Active Directory mapper is configuring it to ignore the subject completely. Using this method, you can configure the AD mappings so that any certificate issued by a particular CA can be mapped to a single user account. See the following: http://technet.microsoft.com/en-us/library/bb742438.aspx#ECAA

    Question

    I've recently been working on extending my AD schema with a new back-linked attribute pair, and I used the instructions on this blog and MSDN to auto-generate the linkIDs for my new attributes. Confusingly, the resulting linkIDs are negative values (-912314983 and -912314984). The attributes and backlinks seem to work as expected, but when looking at the MSDN definition of the linkID attribute, it specifically states that the linkID should be a positive value. Do you know why I'm getting a negative value, and if I should be concerned?

    Answer

    [Sent from Jonathan’s favorite park bench where he feeds the pigeons]

    The negative numbers are correct and expected, and are the result of a feature called AutoLinkID. Automatically generated linkIDs are in the range of 0xC0000000-0xFFFFFFFC (-1,073,741,824 to -4). This means that it is a good idea to use positive numbers if you are going to set the linkID manually. That way you are guaranteed not to conflict with automatically generated linkIDs.

    The bottom line is, this is expected under the circumstances and you're all good.

    Question

    Is there any performance advantage to turning off the DFSR debug logging, lowering the number of logs, or moving the logs to another drive? You explained how to do this here in the DFSR debug series, but never mentioned it in your DFSR performance tuning article.

    Answer

    Yes, you will see some performance improvements turning off the logging or lowering the log count; naturally, all this logging isn’t free, it takes CPU and disk time. But before you run off to make changes, remember that if there are any problems, these logs are the only thing standing between you and the unemployment line. Your server will be much faster without any anti-virus software too, and your company’s profits higher without fire insurance; there are trade-offs in life. That’s why – after some brief agonizing, followed by heavy drinking – I decided not to include it in the performance article.

    Moving the logs to another physical disk than Windows is safe and may take some pressure of the OS drive.

    Question

    When I try to join this Win2008 R2 computer to the domain, it gives an error I’ve never seen before:

    "The following error occurred attempting to join the domain "contoso.com":
    The request is not supported."

    Answer

    This server was once a domain controller. During demotion, something prevented the removal of the following registry value name:

    HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NTDS\Parameters\
    DSA Database file

    Delete that "Dsa Database File" value name and attempt to join the domain again. It should work this time. If you take a gander at the %systemroot%\debug\netsetup.log, you’ll see another clue that this is your issue:

    NetpIsTargetImageADC: Determined this is a DC image as RegQueryValueExW loaded Services\NTDS\Parameters\DSA Database file: 0x0
    NetpInitiateOfflineJoin: The image at C:\Windows\system32\config\SYSTEM is a DC: 0x32

    We started performing this check in Windows Server 2008 R2, as part of the offline domain join code changes. Hurray for unintended consequences!

    Question

    We have a largish AD LDS (ADAM) instance we update daily through by importing CSV files that deletes all of yesterday’s user objects and import today’s. Since we don’t care about deleted objects, we reduced the tombstoneLifetime to 3 days. The NTDS.DIT usage, as shown by the 1646 Garbage Collection Event ID, shows 1336mb free with a total allocation of 1550mb – this would suggest that there is a total of 214MB of data in the database.

    The problem is that Task Manager shows a total of 1,341,208K of Memory (Private Working Set) in use. The memory usage is reduced to around the 214MB size when LDS is restarted; however, when Garbage Collection runs the memory usage starts to climb. I have read many KB articles regarding GC but nothing explains what I am seeing here.

    Answer

    Generally speaking, LSASS (and DSAMAIN, it’s red-headed ADLDS cousin) is designed to allocate and retain more memory – especially ESE (aka “Jet”) cache memory – than ordinary processes, because LSASS/DSAMAIN are the core processes of a DC or AD/LDS server. I would expect memory usage to grow heavily during the import, the deletions, and then garbage collection; unless something else put pressure on the machine for memory, I’d expect the memory usage to remain. That’s how well-written Jet database applications work – they don’t give back the memory unless someone asks, because LSASS and Jet can reuse it much faster when needed if it’s already loaded; why return memory if no one wants it? That would be a performance bug unto itself.

    The way to show this in practical terms is to start some other high-memory process and validate that DSAMAIN starts to return the demanded memory. There are test applications like this on the internet, or you can install some app that likes to gobble a lot of RAM. Sometimes I’ll just install Wireshark and load a really big saved network capture – that will do it in a pinch. :-D You can also use the ESE performance counters under the “Database” and “Database ==> Instances” to see more about how much of the memory usage is Jet database cache size.

    Regular DCs have this behavior too, as does DFSR and do other applications. You paid for all that memory; you might as well use it.

    (Follow up from the customer where he provided a useful PowerShell “memory gobbler” example)

    I ran the following Windows PowerShell script a few times to consume all available memory and the DSAMAIN process started releasing memory immediately as expected:

    $chunk = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
    for ($i = 0; $i -lt 5000; $i++)

           $chunk += $chunk
    }

    Question

    When I migrate users from Windows 7 to Windows 7 using USMT 4.0, their pinned and automatic taskbar jump lists are lost. Is this expected?

    Answer

    Yes. For those poor $#%^&#s readers still using XP, Windows 7 introduced application taskbar pinning and a special menu called a jump list:

    image

    Pinned and Recent jump lists are not migrated by USMT, because the built-in OS Shell32 manifest called by USMT (c:\windows\winsxs\manifests\*_microsoft-windows-shell32_31bf3856ad364e35_6.1.7601.17514_non_ca4f304d289b7800.manifest) contains this specific criterion:

    <pattern type="File">%CSIDL_APPDATA%\Microsoft\Windows\Recent [*]</pattern>

    Note how it is not Recent\* [*], which would grab the subfolder contents of Recent. It only copies the direct file contents of Recent. The pinned/automatic jump lists are stored in special files under the CustomDestinations and AutomaticDestinations folders inside the Recent folder. All the other contents of Recent are shortcut files to recently opened documents anywhere on the system:

    image

    If you examine these special files, you'll see that they are binary, unreadable, and totally proprietary:

    image

    Since these files are binary and embed all their data in a big blob of goo, they cannot simply be copied safely between operating systems using USMT. The paths they reference could easily change in the meantime, or the data they reference could have been intentionally skipped. The only way this would work is if the Shell team extended their shell migration plugin code to handle it. Which would be a fair amount of work, and at the time these manifests were being written, customers were not going to be migrating from Win7 to Win7. So no joy. You could always try copying them with custom XML, but I have no idea if it would work at all and you’re on your own anyway – it’s not supported.

    Question

    We have a third party application that requires DES encryption for Kerberos. It wasn’t working from our Windows 7 clients though, so we enabled the security group policy “Network security: Configure encryption types allowed for Kerberos” to allow DES. After that though, these Windows 7 clients stopped working in many other operations, with event log errors like:

    Event ID: 4
    Source: Kerberos
    Type: Error
    "The kerberos client received a KRB_AP_ERR_MODIFIED error from the server host/myserver.contoso.com. This indicates that the password used to encrypt the kerberos service ticket is different than that on the target server. Commonly, this is due to identically named machine accounts in the target realm (domain.com), and the client realm. Please contact your system administrator."

    And “The target principal name is incorrect” or “The target account name is incorrect” errors connecting to network resources.

    Answer

    When you enable DES on Windows 7, you need to ensure you are not accidentally disabling the other cipher suites. So don’t do this:

    image

    That means only DES is supported and you just disabled RC4, AES, etc.

    Instead, do this:

    image

    If it exists at all and you want DES, this registry DWORD value to be 0x7fffffff on Windows 7 or Win2008 R2:

    MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies\System\Kerberos\Parameters\
    SupportedEncryptionTypes

    If it’s set to 0x3, all heck will break loose. This security policy interface is admittedly tiresome in that it has no “enabled/disabled” toggle. Use GPRESULT /H or /Z to see how it’s applying if you’re not sure about the actual settings.

    Other Stuff

    Windows 8 Consumer Preview releases February 29th, as if you didn’t already know it. Don’t ask me if this also means Windows Server 8 Beta the same exact day, I can’t say. But it definitely means the last 16 months of my life finally start showing some results. As will this blog…

    Apparently we’ve been wrong about Han and Greedo since day one. I want to be wrong though. Thanks for passing this along Tony. And speaking of which, thanks to Ted O and the rest of the gang at LucasArts for the awesome tee!

    This is a … creepily good music video? Definitely a nice find, Mark!


    This is basically my home video collection

    My new favorite site of the week? The Awesomer. Do not visit if you have to be somewhere in an hour.

    Wait, no… my new favorite site is That’s Nerdaliscious. Do not read if hungry or dorky. 

    Sick of everyone going on about Angry Birds? Love Chuck Norris? Go here now. There are a lot of these; don't miss Mortal Combat versus Donkey Kong.

    Ah, there’s Waldo.

    Likely the coolest advertisement for something that doesn’t yet exist that you will see this year.


    I need to buy stock in SC Johnson. Can you imagine the Windex sales?!

    Until next time.

    - Ned “Generation X” Pyle with Jonathan “The Greatest Generation” Stephens

  • Friday Mail Sack: VROOM VROOM Edition

    Hi folks, Jonathan here again. Ned’s a little busy right now trying to get items off the top shelf of the cabinet, I thought I’d grab some responses he was working on off this desk and put this week’s Mail Sack together. Today we talk about:

    Let me go get Ned a step stool, and then we’ll get started on the Q & A.

    Question

    If I use Auditing and remove a user’s group membership, I see Security Group Management events (4729, 4759, etc.). If I delete that user though, I only see “a user account was deleted (4726) events.  There’s no group membership event – is that normal?

    Answer

    [Carefully crafted by Ned in his little Treebicle.]

    User deletion means that the System performs the group membership removal.  You will see the same behavior when you create a user – there is no audit event when they are added to the local Users group, for example. This lack of System update auditing is intentional; otherwise, the log would explode from useless information.

    Question

    I was reading documentation about the Account Operators group’s default behavior. I have found that despite what it says here, members of the account operators group can delete administrators. Is the documentation wrong or is this expected?

    Answer

    [Straight from the (tiny) desk of Ned.]

    Let’s analyze what the article says versus what the author meant:

    Members of this group can create, modify, and delete accounts for users, groups, and computers located in the Users or Computers containers and organizational units in the domain, except the Domain Controllers organizational unit.

    Mostly true. If you look at the default permissions on the Users container, for example, you see they definitely have create and delete rights:

    clip_image002

    It will be similar for your custom OUs, because those OU objects inherit those default permissions from the AD schema upon creation.

    clip_image004

    If your administrative accounts live in the Users container or a custom OU where you have not modified the default permissions, members of the account operators group can delete those users with impunity. If you want to stop this behavior, place your administrative users in a custom OU where you remove the Account Operators group from the permissions.

    Members of this group do not have permission to modify the Administrators or the Domain Admins groups, nor do they have permission to modify the accounts for members of those groups.

    True, but sometimes it takes a bit. At first, every user created allows the Account Operators group full control – this comes from the default schema security. They cannot modify administrative users, change their passwords, remove their group memberships, or otherwise manipulate them once AdminSDHolder and SDProp have their way with the account. Moreover, the author did not mean, “modification equals deletion”, even though you and I know as IT pros that it “is the ultimate modification”, of a sort. Modifying its existence. J At no point can Account Operators modify the members of the high security groups like Domain Admins, regardless of SDProp timing. Otherwise an Account Operator could elevate a normal user to the highest privilege levels in the domain.

    Members of this group can log on locally to domain controllers in the domain and shut them down.

    True (and subsequent <Rodney Dangerfield collar pull>). If you are dead set on using the Account Operators, removing this right (stored in the Default Domain Controllers policy) is probably a good idea. These users can deny service to your entire network, by shutting down every DC at once.

    Because this group has significant power in the domain, add users with caution.

    True! The Account Operators group is one of those NT 4.0 legacies that date back to an operating system that didn’t have a hierarchical management structure like X.500/LDAP, and instead a big blob of goo called a SAM database. Using this group is not ideal; it has far too many privileges and based on SDProp timing, can have too much power over brand new admin users for brief periods of time. We spent countless millions of dollars creating AD and Delegation of Control so that our customers could abandon the legacy management systems. If the Account Operators group is awesome, why would we have bothered otherwise?

    Question

    There are a lot of articles that discuss CIFS interoperability, and they refer to LAN Manager Authentication Level, but there are very few that mention the registry parameter AllowLegacySrvCall. What does this setting actually do?

    Key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0
    Value: AllowLegacySrvCall
    Type: REG_DWORD
    Default: 0x0

    Answer

    Maybe you should sit down.

    When a client attempts to connect to an SMB server the server generates a challenge and sends it to the client. The idea is that the client manipulates the challenge using its secret knowledge (the password) and sends the result back to the server as the response. Local System Authority Subsystem (LSASS) on the server evaluates that response and determines if the user has been properly authenticated. This is standard NTLM challenge/response authentication mechanics. With extended security support, LSASS performs some other checks to evaluate if the response has been tampered with, and if it has, the user is denied access. Unfortunately, this introduced a bug that was discovered in Windows Vista and Windows Server 2008. Creating the registry value AllowLegacySrvCall was our way of resolving this bug.

    If the client supports extended security, LanManServer goes back to LSASS to generate the challenge to send to the client. If the client does not support extended security for NTLMv2, then LanManServer optimizes by generating its own challenge to the authentication request. Unfortunately, this challenge wasn't created by LSASS so it is missing some information when LSASS later evaluates the response to the challenge. This causes the response to be considered invalid and so authentication fails.

    AllowLegacySrvCall enables logic in LSASS such that it can detect that a particular response was created from a challenge generated by LanManServer (as opposed to LSASS). In this case, LSASS will omit the extended security checks on the response. The effect of this setting is that if you have older SMB clients that do not support extended security then your NTLMv2 security is slightly compromised because there is no way to detect tampering of the authentication response on the wire.

    So when do you need to enable AllowLegacySrvCall?

    1. You are enforcing NTLMv2 for authentication.
    2. Your SMB client does not support extended security. This usually means older Mac OS X, jcifs, and Samba. Note that NT 4.0 would also be affected, here.

    Question

    I realize this is an old article but couldn't you get around the USN rollback issue by doing an authoritative restore on the DC you bring back from a snapshot? Don't actually restore a backup but just run NTDSUTIL to make the DC authoritative for all objects. That would push all that DC's USNs up by 100,000 and the objects would replicate out -- hence no USN rollback issue.

    Answer

    Not quite. Consider the scenario where an object is created or an attribute is set on DC1 after the snapshot is taken. This change propagates out to all replication partners with the originating change designated as being on DC1. Now you restore your snapshot and use NTDSUTIL to mark authoritative all the objects and attributes in the Active Directory on DC1. Those objects and attributes will indeed replicate out, but what about the objects (or attributes) on DC1's partners that actually originated on DC1? Those changes will not propagate back to DC1 because the partner must assume that DC1 is already aware of them because the invocation ID of the partner has not changed.

    This is why the invocation ID changes when AD is restored using a supported method. A new invocation ID indicates to all partners that this is essentially a new database with some pre-synchronized data in it and it needs to be updated from all partners. It is not just the USN value itself that impacts the rollback status of a DC, but it is also the invocation ID that distinguishes DC1's restored database from its original database. With the new invocation ID, changes that originated on DC1 after the backup was taken will propagate back to DC1 because partners won't think the changes originated on the now restored DC. Restoring a snapshot does not change the invocation ID, and thus basically breaks AD's ability to properly recover from a restore operation.

    Long story short…don't do it.

    If you have further questions, I recommend Rich Peckham's blog post on the topic of authoritative restores.

    Question

    I have read the W32Time documentation and blogs but I do not understand one thing. What is the difference in flags 0x1 and 0x8 in the registry parameter below:

    Key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Parameters
    Value: NtpServer
    Type: REG_SZ
    Example: time.windows.com,0x8

    Answer

    The flags value for NtpServer are briefly documented in the following KB article: Time synchronization may not succeed when you try to synchronize with a non-Windows NTP server in Windows Server 2003.

    0x01 - use special poll interval SpecialInterval
    0x02 – UseAsFallbackOnly
    0x04 - send request as SymmetricActive mode
    0x08 - send request as Client mode

    You can find more detail about how these flags interact in the Microsoft Communications Protocol Program (MCPP) library on MSDN: [MS-SNTP]: Network Time Protocol (NTP) Authentication Extensions.

    If you need to understand the difference between Symmetric Active mode and Client mode, then you should consult RFC 5905. It’ll put hair on your chest.

    Other Stuff

    Is this the greatest metal album cover of all time? I think so.


    Plow tank with gatling guns on a skull road with hardcore driver demanding answers from an uncaring corporate world? Check.

    Do you have a set of wheels appropriate for the Zombie Apocalypse? Why not skip the Marauder and buy American?

    And here is some stuff that makes me wish I still had a dorm room to decorate.

    Until next time, folks.

    - Jonathan “Average Height and Build” Stephens with Ned “Not” Pyle.

  • Congrats Sean and Mark, the Newest Masters!

    Hey all, Ned here again. You probably know our pals Sean Ivey and Mark Renoden from their AskDS blog contributions. Both of them were once Directory Services Support Engineers and are now Premier Field Engineers, traveling the globe to help solve your problems. Much like the A-Team. Or not.

    Anyway, what you probably don't know is that yesterday they joined the elite fraternity of Microsoft Certified Masters along with nine of their new best friends. Having taught that certification since day 0, I can tell you it is a royal gentleman's fruit buster and to get it takes serious dedication and serious smarts; heck, after five years and fourteen rotations, MCM DS only finally crossed the 100 graduate mark! If you haven't explored the certification that will set you apart from everyone in the IT industry, I suggest you start. Make sure you bank some sleep first though, and don't forget to ask Ryan about the Banana Crown.

    We're awful proud of our former DS support brothers. Congratulations fellas.

    - Ned "and all your old buddies" Pyle