'Cos the world needs one more...

I wrote a KB!

maweeras — Sun, 01 Nov 2009 13:26:48 GMT

Well….half a KB to be precise :) But I still feel good that I contributed to an official support article in some minute way. Incidentally I wrote the ldp.exe based details in KB 2001769.

Lessons learned on CritSit or The importance of updating drivers

maweeras — Sat, 17 Oct 2009 19:59:07 GMT

As a premier field engineer I have a responsibility to do several on call shifts a year. The week just gone was one such on call shift learned the importance of updating drivers during it. Allow me to elaborate.

I got a call around 2AM Wednesday about a customer that needed help recovering the business after accidentally deleting the OU that had all the user accounts in the organisation. I assumed it would be a simple case of performing an authoritative restore and accepted the case. After turning up onsite I learned that authoritative restores had already been performed but when they ran the LDIF file to recover group membership, the destination servers hung. Without running the LDIF file, they had managed to get all the users replicate out successfully. But the DCs geographically spread across the country had inconsistent group membership information.

As the LDIF file could not be replicated out and as the customer was desperate for resolution, we resorted to rebuilding all the DCs using the source DC backup and performing IFM based promotions. This turned out to be the resolution mechanism while we attempted root cause analysis. A colleague of mine joined me onsite and we recreated the problem in an isolated lab network. Analysis of the memory dump revealed issues with the storage related drivers. Specifically HpCISSs2.sys.

We then tested the DC (All HP DL360 G5) after updating the following.

Controller Firmware (1.82 as per KB969550)
Disk Firmware (version HPDA for DH036ABAA5 although customer had DH036ABAA6 disks)
Controller Drivers (as per KB969550 we installed 6.14.0.32 from a smartstart CD)
Storport.sys (KB957910 )

We could no longer reproduce the issue and a fix was now available. Yay! Customer decided to keep the DCs that were recovered using IFM online and to turn off the remainder and perform metadata and DNS cleanup. They are now planning to rebuild the remainder later after completely rebuilding them using latest HP SmartStart CD and Windows Server 2003 R2 SP2 CDs. Online DCs are to be also gradually updated with updates shown above.

Lessons learnt were as follows.

Importance of preventing accidental deletions of key OUs.
Importance of applying all relevant Windows Server service packs/hotfixes (disable SNP, hotfixes for LVR, ntdsutil etc.)
Importance of updating hardware firmware/drivers

I have tried to keep this post short by skipping information about the long hours and sleep lost, the time it took for trial and error parts of resolution, challenges with 3rd party DNS that had to be circumvented using Windows DNS temporarily during recovery and the pain to customers while they were down for 3 days. But I assure you that failure to learn the lessons highlighted above, will be very painful if this happens to you.

Regards

Broadcom IPV4 Large Send Offload

maweeras — Wed, 15 Jul 2009 00:01:41 GMT

I have run into a couple of scenarios where this setting has caused issues and hence decided to blog about it.

Windows 2008 X64 based domain controllers didn’t replicate with each other. However they were pulling updates inbound from other DCs (that happened to be Windows 2000 Server based) fine. Running commands such as repadmin /bind <W2k8DCname> and /replsum switches indicated replication failed with access denied. Additionally terminal server sessions –(remote admin mode) were dropped repeatedly within milliseconds of establishing a session. It turned out that the newly built Windows 2008 based DCs had Broadcom cards in them and the driver installer enabled the above setting. Once this was turned off, everything started working perfectly.
Was at a customer recently that claimed when they made their Windows 2003 X64 DC in a single domain forest a GC, after a while the server was unresponsive to certain RPC traffic. once the server was removed from been a GC and rebooted, it would not cause issues. I didn’t believe them at first but then they demonstrated by configuring the DC to a GC role. Sure enough a few hours later the server was having issues. repadmin /bind <dcname> traffic once captured over netmon showed resets immediately. dir \\dcname\c$ and other SMB share access access such as SYSVOL and same DC worked. LDAP and GC traffic was perfect. However repadmin based commands like /bind, /replsum did not work. portqry commands to port 135 failed (unfortunately I don’t have error details at the moment). Once the above setting was turned off on the NIC and server was rebooted, no further recurrences were noted.

So in case you run into issues where “RPC traffic is not responded to properly”, check and disable the above setting if configured on your server NIC. Please post a comment if you run into similar issues resolved by this setting.

Thanks

w2k3_bridges_required

maweeras — Tue, 23 Jun 2009 20:30:03 GMT

repadmin’s w2k3_bridges_required setting is often a misunderstood setting. Even I was confused to its usage. So when elite PFE engineer Glenn explained in an internal DL, I felt it would be good to share with others.

The +W2K3_BRIDGES_REQUIRED has nothing to do with DFS. Repeat after me….”+W2K3_BRIDGES_REQUIRED has nothing to do with DFS”

This setting, when configured on a site, tells the KCC on the ISTG in said site to ignore the BASL setting (on or off) when determining site link transitiveness for the purpose of creating connection objects. Nothing more, nothing less.

If you want DFS to provide intelligence that can take advantage of site link costing, then you turn on sitecostedreferrals ( see DFS Tools and Settings for details). If you want that intelligence to extend beyond the adjacent site, then you must have a site link bridge to the transitive sites containing DFS namespace and link servers.

One way to accomplish this transitiveness is the catchall BASL. Another completely accurate way is manual site link bridge for each referral path for which you would like the costed referral to be something less than infinity. That does not necessarily necessitate a full mesh of site link bridges.

DFS is just a consumer of ISM and its site cost matrix services.

If there is an adjacent or transitive path from this site to some other site, then that other site will have a cost from this site. If there is no adjacent or transitive path from this site to some other site, then the cost of the other site is infinity.

DFS will order referrals for DFS namespace servers and link servers based on that servers site location and its cost away from the callers site.

DCDiag reports “not advertising as time server”

maweeras — Fri, 08 May 2009 18:46:08 GMT

Just a quick post.

Was onsite recently and had a DC that was not advertising is a time server. DCDiag confirmed it.

Checked announceflags and it was set to 10 in hex (0x00000010) instead of 10 in decimal (0x0000000A).

Changed registry value and use “w32tm /config /update” and all was well. Interestingly it appears to only read last value. So even he (0x0000001A) also works and advertises as time server. Presumably because it reads the last character “A” and thinks

A = 10 = (0x8) + (0x2)

and as per http://technet.microsoft.com/en-us/library/cc773263(WS.10).aspx#w2k3tr_times_tools_uhlp

AnnounceFlags

Registry path

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config

Version

Windows XP, Windows Vista, Windows Server 2003, and Windows Server 2008

This entry controls whether this computer is marked as a reliable time server. A computer is not marked as reliable unless it is also marked as a time server.

0x00 Not a time server
0x01 Always time server
0x02 Automatic time server
0x04 Always reliable time server
0x08 Automatic reliable time server

The default value for domain members is 10. The default value for stand-alone clients and servers is 10.

Definition of bad weekend

maweeras — Mon, 30 Mar 2009 13:55:06 GMT

A non tech post today.

1. Get thumb crushed on car door. End up with an agonisingly painful thumb which causes you to visit the Accident and Emergency dept to drain blood under thumbnail by drilling a hole into the thumbnail.

2. Hours after the thumb was crushed, decide to fill up the car and put 15L of petrol instead of Diesel and then await recovery company to take car away for repair

FRS and host files

maweeras — Tue, 24 Feb 2009 15:08:37 GMT

Just a quick blog post about something I saw recently. I was at a customer site performing an ADRAP and I ran into one of the customer’s domains where FRS was not converging. Among the troubleshooting steps I tried to verify that DNS had records for the DCs which it did and I verified using nslookup from the ADRAP machine I was using for gathering data from the DCs.

To cut a long story short the customer had a host file on the DCs of that domain in the following format.

10.1.1.1 DC1 DC1.Domain.Com

10.2.3.1 DC2 DC1.Domain.Com

As both IP addresses were bound to the same FQDN, FRS on each DC was having issues pulling updates from its partner. However AD replication was working fine. The reason for this is the DCs were using the <GUID>._msdcs.forestFQDN and therefore DNS was resolving the record.

Note to self, when troubleshooting name resolution always do the test from the host machine directly.

HTH

DCLocator and Closest Site info

maweeras — Sat, 14 Feb 2009 23:32:55 GMT

I’ve been doing some research to prepare for my upcoming first delivery of AD Troubleshooting workshop. The agenda of content includes DCLocator and Netlogon content. Hence this post.

For the real detail on dsgetdcname and details on the nltest /dsgetdc flags, please see http://msdn.microsoft.com/en-us/library/ms675983(VS.85).aspx.

I made a lab environment which consisted of a single domain forest and 1 x Vista and 1 x XP client. Here are my observations.

If the Vista client’s IP address is from a subnet known by AD, a Windows Server 2008 DC will provide the closest site info details as well. It does not do this for XP. Presumably other down-level OS editions are also not given this detail but I didn’t check.

Here is the LDAP filter one of the UDP based CLDAP ping performed by the Vista client.

Filter: (&(DnsDomain=adatum.com)(Host=ADTMCLIENT1)(DomainGuid={AEF3F962-37E0-D147-8611-D8CA1AAED85E})(NtVer=16:00:00:00))

And the response back from the Windows Server 2008 DC is as follows.

Frame: Number = 86, Captured Frame Length = 200, MediaType = ETHERNET

+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-03-FF-0D-FA-00],SourceAddress:[00-03-FF-A5-9B-45]

+ Ipv4: Src = 10.1.1.2, Dest = 10.1.1.6, Next Protocol = UDP, Packet ID = 3904, Total IP Length = 186

+ Udp: SrcPort = LDAP(389), DstPort = 58266, Length = 166

+ Cldap: (CLDAP)Search Result Entry, MessageID: 1, Status: Success

- NetlogonAttribute: LogonSAMLogonResponseEX (SAM Response to SAM logon request): 23 (0x17)

- SamLogonResponseEx: ADTMDC1.adatum.com

Opcode: LogonSAMLogonResponseEX

Sbz: 0 (0x0)

+ Flags: 0x000013FD

DomainGuid: {62F9F3AE-E037-47D1-8611-D8CA1AAED85E}

DnsForestName: adatum.com

DnsDomainName: adatum.com

DnsHostName: ADTMDC1.adatum.com

NetbiosDomainName: ADATUM

NetbiosComputerName: ADTMDC1

UserName:

DcSiteName: HQ

ClientSiteName: HQ

Unknown: Binary Large Object (7 Bytes)

+ Version: 0x00000015 NT Version 5 Client

+ LmNtToken: Windows NT Networking: 0xFFFF

+ Lm20Token: OS/2 LAN Manager 2.0 (or later) Networking: 0xFFFF

The italic Unknown: Binary Large Object (7 Bytes) has hex details corresponding to the following.

05 53 69 74 65 32 00

.Site2.

Here is the site topology created using the ADTD Visio tool.

Currently Netmon 3.2 with latest parsers decodes as above. Wireshark as of v1.0.5 did not decode this.

Please note that the above “Unknown: Binary Large Object (7 Bytes)” field is NOT available if the DC cannot see a subnet that the Vista machine belongs to. Netlogon.log on the client shows that the client did not use any try_next_closest_site flags. Yet the DC still presents the information. It can do this as the filter has a NtVer=16:00:00:00 string identifying the Client OS. XP for example would use NtVer=06:00:00:00. Hence a Windows Server 2008 DC will not respond with closest site info to them.

Sysinternals to the rescue

maweeras — Mon, 12 Jan 2009 09:48:10 GMT

This is a quick post to get me back into the spirit of blogging. Some time back I was onsite performing an ADRAP to assist the customer with some issues they were having with their AD. Among the many issues we found was one DC reporting the following in its system/DS and FRS event logs.

“Registration of the DNS record '_kpasswd._tcp.domain.com. 600 IN SRV 0 100 464 DC1.adatum.com.' failed with the following error: %An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.”
“No Windows NT or Windows 2000 Domain Controller is available for domain adatum. The following error occurred: %Not enough storage is available to process this command.”
“The attempt to establish a replication link with parameters.

Partition: CN=Configuration,DC=adatum,DC=com

Source DSA DN: CN=NTDS Settings,CN=DC3,CN=Servers,CN=Branch1,CN=Sites,CN=Configuration,DC=adatum,DC=com

Source DSA Address: 5bbb4c2b-47bf-4593-b0dc-460ea4916d49._msdcs.adatum.com

Inter-site Transport (if any): CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=adatum,DC=com

failed with the following status:

Not enough storage is available to complete this operation.

The record data is the status code. This operation will be retried.”

4. Following is the summary of warnings and errors encountered by File Replication Service while polling the Domain Controller dc1.adatum.com for FRS replica set configuration information.

Could not bind to a Domain Controller. Will try again at next polling cycle.

Realising we had a network issue I used “netstat –an” on the DC1 to see the network connections it had established. Output similar to below was

Proto Local Address          Foreign Address        State
TCP    0.0.0.0:135            0.0.0.0:0              LISTENING
TCP    0.0.0.0:445            0.0.0.0:0              LISTENING
TCP    0.0.0.0:990            0.0.0.0:0              LISTENING
<snip>
TCP    0.0.0.0:1025          0.0.0.0:0              LISTENING
TCP    0.0.0.0:1026          0.0.0.0:0              LISTENING
TCP    0.0.0.0:1027          0.0.0.0:0              LISTENING
<snip>
TCP    0.0.0.0:4998          0.0.0.0:0              LISTENING
TCP    0.0.0.0:4999          0.0.0.0:0              LISTENING
TCP    0.0.0.0:5000          0.0.0.0:0              LISTENING

As this was a Windows 2000 Server I did not have the “-o”option of neststat to print out the process ID that was listening on the ports. But you will note that in the above output all ports between 1024-5000 were in use. This is the ephemeral port range. I was pretty certain that the server was infected by now. I wanted to know what the process was so I used TCPView from Sysinternals. TCPview revaled the process name and a quick search on the Malware Protection Center revealed it to be an IRC Bot. Further investigation revealed the DC did not have any anti-virus software and was missing many critical and important security updates. Unfortunately I don’t recall the exact name of the worm.

This was the first time I’d come across a real production DC that was infected. This particular server had replication issues because it did not have any free ports available for use for replication. AD and FRS replication was affected as a result.

HTH

The case of the DNS records that didn't resolve

maweeras — Sat, 01 Nov 2008 01:13:59 GMT

I was at a customer today who was in the process of moving their AD from Windows Server 2003 to Windows Server 2008 x64 platform. In the process at some point he changed the DHCP servers to use the new Windows Server 2008 based DC/DNS servers for name resolution. He then said he was having issues with name resolution and said the problem disappeared when he used the Windows Server 2003 based DC/DNS.

I wasn't sure whether I could believe him :) I did some tests using nslookup and dcdiag and then assumed it was a non existent issue. However, he reproduced the issue. I was absolutely amazed at the time as the same query done on W2K3 worked but it didn't on W2K8. This was for a record that was in an AD integrated zone and so I didn't want to assume a corrupt zone. Netmon traces clearly showed the record as non existent despite visible in dnsmgmt.msc and dnscmd.

Then I did what I should have done first. I checked the event log :-). In there was an event about "global query block list". The record that they could not resolve was an important record. I.e. it was the wpad record for proxy settings. In this case the customer had literally typed wpad into the proxy server settings (instead of the proxy server name) along with the relevant port. wpad resolution is typically only required if "Auto detect settings" checkbox is selected to detect the proxy settings. As they were literally using the name "wpad" in the proxy settings dialogs box, they couldn't access the Internet.

With the mystery solved I proceeded to advise them of the behaviour and point them at Technet resources for more details. The recommendation was to change the proxy settings to use a more meaningful name such as the real proxy server name as they were not reliant on auto proxy settings discovery. Please see Managing the Global Query Block List for details.

HTH

The case of the blocked FRS replication

maweeras — Mon, 15 Sep 2008 01:03:44 GMT

So I haven't written anything in quite a while and felt compelled to write something I recently ran into. A customer of mine is in the process of implementing Windows Server 2008 RODC in their branch offices. In order to do that, they are planning to in-place upgrade the W2K3 DCs to W2K8 at the hub sites.

The customer did an in-place upgrade of their W2K3 DC in their lab and found that while the upgrade worked, the DC didn't appear to be completely functional. Specifically FRS replication appeared to be broken. Event 13508 was logged by the W2K8 DC reporting that it couldn't replicate. Similarly W2K3 DCs reported the W2K8 to be unavailable. The customer even turned the W2K8 firewall off to no avail. I asked for FRSDiag content from the customer and the results I got back were partial. There was no connstat.txt or any other content from ntfrsutl. Instead I had RPC errors scattered through the FRSDiag data.

The customer in question has a very large global AD implementation and site to site communication is secured with firewalls in place. To ensure AD and FRS replication works through firewalls, the customer has configured static ports for AD and FRS replication. Firewall rules are also configured to ensure these ports are open and the replication traffic between DCs are allowed. http://support.microsoft.com/kb/224196/ and http://support.microsoft.com/kb/319553/ have details of how to lock these ports down.

Analysis of the FRSDiag data showed that FRS was trying to listen on port TCP 49153. This had been enforced through Group Policies. However, it couldn't as the port was in use. In this case, we used "portqry -n <server> -e 135" to query the RPC endpoint mapper. This showed that the Event Log TCPIP was using the port. In W2K8 and Vista we made change to the ephemeral port range to start from 49152 (instead of 1025) and end at 65535 (instead of 5000). So the event log in W2K8 starts early on and grabs this port (49153). In W2K3 it would have grabbed something early in the 1025-5000 range. So the customer didn't see an issue until the upgrade to W2K8.

Changing FRS to use a dynamic port (by deleting the reg key that enforced a static port) and restarting FRS fixed the issue temporarily. So we knew we had the cause figured out. The problem was the customer did not want to have to go and update firewall rules to allow the in-place upgrade of a W2K3 DC to W2K8 work. So he wanted to know whether we could have the Event Log use another port. From the research I've done, it does not appear to be possible to do this. However, in W2K8 you can move the ephemeral port range to start from another value instead of the default 49152. So I pointed the customer to the fine article at Ned's Askds blog (http://blogs.technet.com/askds/archive/2007/08/24/dynamic-client-ports-in-windows-server-2008-and-windows-vista-or-how-i-learned-to-stop-worrying-and-love-the-iana.aspx) and advised him to move the range to start from 49160 ( which freed 49152-49159 for customer requirements) and test!

FRSDiag is such a great tool. For more good details on its usage, head on to Ned's blog for some quality articles such as How to get the most from your FRSDiag…and tips.

Can't Open XPS Documents on Vista

maweeras — Sun, 25 May 2008 14:49:48 GMT

Just thought I'd post a quick observation I made on my laptop. Some time back I ran into an interesting issue on my laptop where I could create XPS documents using the "XPS Document Writer" printer or even using applications like Office 2007. I could even see a preview of it in Windows Explorer. But I could not open them.

I accidentally then discovered that if I change my colour scheme to Windows Basic, I can open any XPS files. So I thought I'd post the workaround for others benefit. Incidentally, I am following this up with our engineers here and if I find a fix I will update you all.

Go to Control Panel - Personalization

Choose the first Windows Colour and Appearance

Click the "Open Classic Appearance Properties for More Colour Options" at the bottom

Choose Windows Basic as colour scheme.

This seems to be a rare issue that does not affect all Vista users. But I hope it helps those that are affected. Incidentally I have a 32-bit Vista Enterprise SP1 laptop (Toshiba M400) with all Windows Updates applied to date.

HTH

NTDSUTIL - Group Membership Evaluation

maweeras — Fri, 02 May 2008 15:25:50 GMT

I just had an issue at a customer where we were troubleshooting an issue where users could not use the SQL Management studio to connect to a SQL server remotely. Local login to SQL server interactively and then launching the client tools worked. The cause was the number of groups the user belonged to.

To troubleshoot this, we busted out ntdsutil and tried to enumerate the group membership. But to our surprise it only showed 3 groups. Then after changing servers we got 600 plus groups and then again 3 groups. So basically the resultant number of groups were changing all the time.

It turns out this is a bug in ntdsutil for which a hotfix is now available. If you are using the group membership evaluation feature in ntdsutil, try this version. http://support.microsoft.com/kb/934185

HTH

User initiated crash dump in Vista Laptop

maweeras — Fri, 21 Mar 2008 20:34:00 GMT

I have a vista SP1 based Toshiba M400. As we tend to regularly dogfood software builds internally I also assist with self hosting any software builds I am interested in. These builds are not perfect and sometimes they cause laptops to hang, deplete resources etc...

I've not done much to generate crash dumps and have always had one "helpfully" created for me. I have had the "pleasure" of having to use the instructions in http://support.microsoft.com/kb/244139 article for servers in the past but never used it on a laptop. When I recently had the need to use crashonctrlscroll I was annoyed to discover my laptop had no right Ctrl key. I figured I'd use a USB keyboard but as per http://support.microsoft.com/kb/944564 the feature is not available using USB keyboards. At least not on Windows Server 2008. Not sure if it applies to Vista too but it wouldn't surprise me if it did.

I unlocked my session from last night only to see the spinning donut (although it wasnt spinning) and the desktop as it last was. No response to keyboard and mouse input. I just tried to use the normal Ctrl +Function+ F12 and was "pleasantly" surprised when it did do the crash dump. Note my M400 needs Fn + F12 for scroll lock. I didn't have a USB keyboard attached although I do have the steps from the 1st KB applied.

So there you go. You can generate crash dumps on laptops running Vista even without the right Ctrl.

HTH

Wevtutil Error

maweeras — Sat, 19 Jan 2008 03:29:06 GMT

I just found out today that the command listed to view all audit events for Vista and Windows Server 2008 does not work on installations if using a language other than English USA. The following command will result in an error as follows.

C:\Windows\system32>wevtutil gp Microsoft-Windows-Security-Auditing /ge /gm:true

name: Microsoft-Windows-Security-Auditing
guid: 54849625-5478-4994-a5ba-3e3b0328c30d
helpLink: http://go.microsoft.com/fwlink/events.asp?CoName=Microsoft%20Corporati
on&ProdName=Microsoft%c2%ae%20Windows%c2%ae%20Operating%20System&ProdVer=6.0.600
0.16386&FileName=adtschema.dll&FileVer=6.0.6000.16386
resourceFileName: %SystemRoot%\system32\adtschema.dll
parameterFileName: %SystemRoot%\system32\msobjs.dll
messageFileName: %SystemRoot%\system32\adtschema.dll
message:
channels:
channel:
    name: Security
    id: 10
    flags: 1
    message:
levels:
level:
    name: win:Informational
    value: 4
    message:
opcodes:
opcode:
    name: win:Info
    value: 0
      task: 0
      opcode: 0
    message:
tasks:
task:
    name: SE_ADT_SYSTEM_SECURITYSTATECHANGE
    value: 12288
    eventGUID: 00000000-0000-0000-0000-000000000000
Failed to get message property. the message resource is present but the message
is not found in the string/message table

The Windows Events Development team is aware and will address this in the near future.

HTH