• Broadcom IPV4 Large Send Offload

    I have run into a couple of scenarios where this setting has caused issues and hence decided to blog about it.

    1. Windows 2008 X64 based domain controllers didn’t replicate with each other. However they were pulling updates inbound from other DCs (that happened to be Windows 2000 Server based) fine. Running commands such as repadmin /bind <W2k8DCname>  and /replsum switches indicated replication failed with access denied. Additionally terminal server sessions –(remote admin mode) were dropped repeatedly within milliseconds of establishing a session. It turned out that the newly built Windows 2008 based DCs had Broadcom cards in them and the driver installer enabled the above setting. Once this was turned off, everything started working perfectly.
    2. Was at a customer recently that claimed when they made their Windows 2003 X64 DC in a single domain forest a GC, after a while the server was unresponsive to certain RPC traffic. once the server was removed from been a GC and rebooted, it would not cause issues. I didn’t believe them at first but then they demonstrated by configuring the DC to a GC role. Sure enough a few hours later the server was having issues. repadmin /bind <dcname> traffic once captured over netmon showed resets immediately. dir \\dcname\c$ and other SMB share access access such as SYSVOL and same DC worked. LDAP and GC traffic was perfect. However repadmin based commands like /bind, /replsum did not work. portqry commands to port 135 failed (unfortunately I don’t have error details at the moment). Once the above setting was turned off on the NIC and server was rebooted, no further recurrences were noted.

    So in case you run into issues where “RPC traffic is not responded to properly”, check and disable the above setting if configured on your server NIC. Please post a comment if you run into similar issues resolved by this setting.

    Thanks

    M

  • Can't Open XPS Documents on Vista

    Just thought I'd post a quick observation I made on my laptop. Some time back I ran into an interesting issue on my laptop where I could create XPS documents using the "XPS Document Writer" printer or even using applications like Office 2007. I could even see a preview of it in Windows Explorer. But I could not open them.

    I accidentally then discovered that if I change my colour scheme to Windows Basic, I can open any XPS files. So I thought I'd post the workaround for others benefit. Incidentally, I am following this up with our engineers here and if I find a fix I will update you all.

  • Go to Control Panel - Personalization

  • Choose the first Windows Colour and Appearance

  • Click the "Open Classic Appearance Properties for More Colour Options" at the bottom

  • Choose Windows Basic as colour scheme.

  • This seems to be a rare issue that does not affect all Vista users. But I hope it helps those that are affected. Incidentally I have a 32-bit Vista Enterprise SP1 laptop (Toshiba M400) with all Windows Updates applied to date.

    HTH

    M

  • DCDiag reports “not advertising as time server”

    Just a quick post.

    Was onsite recently and had a DC that was not advertising is a time server. DCDiag confirmed it.

    Checked announceflags and it was set to 10 in hex (0x00000010) instead of 10 in decimal (0x0000000A).

    Changed registry value and use “w32tm /config /update” and all was well. Interestingly it appears to only read last value. So even he (0x0000001A) also works and advertises as time server. Presumably because it reads the last character “A” and thinks

    A = 10 = (0x8) + (0x2)

    and as per http://technet.microsoft.com/en-us/library/cc773263(WS.10).aspx#w2k3tr_times_tools_uhlp 

    AnnounceFlags
    Registry path

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config

    Version

    Windows XP, Windows Vista, Windows Server 2003, and Windows Server 2008

    This entry controls whether this computer is marked as a reliable time server. A computer is not marked as reliable unless it is also marked as a time server.

    • 0x00 Not a time server
    • 0x01 Always time server
    • 0x02 Automatic time server
    • 0x04 Always reliable time server
    • 0x08 Automatic reliable time server

    The default value for domain members is 10. The default value for stand-alone clients and servers is 10.

  • w2k3_bridges_required

    repadmin’s w2k3_bridges_required setting is often a misunderstood setting. Even I was confused to its usage. So when elite PFE engineer Glenn explained in an internal DL, I felt it would be good to share with others.

     

    The +W2K3_BRIDGES_REQUIRED has nothing to do with DFS.  Repeat after me….”+W2K3_BRIDGES_REQUIRED has nothing to do with DFS”

    This setting, when configured on a site, tells the KCC on the ISTG in said site to ignore the BASL setting (on or off) when determining site link transitiveness for the purpose of creating connection objects. Nothing more, nothing less.

    If you want DFS to provide intelligence that can take advantage of site link costing, then you turn on sitecostedreferrals ( see DFS Tools and Settings for details).  If you want that intelligence to extend beyond the adjacent site, then you must have a site link bridge to the transitive sites containing DFS namespace and link servers.

    One way to accomplish this transitiveness is the catchall BASL.  Another completely accurate way is manual site link bridge for each referral path for which you would like the costed referral to be something less than infinity.  That does not necessarily necessitate a full mesh of site link bridges.

    DFS is just a consumer of ISM and its site cost matrix services.

    If there is an adjacent or transitive path from this site to some other site, then that other site will have a cost from this site.  If there is no adjacent or transitive path from this site to some other site, then the cost of the other site is infinity.

    DFS will order referrals for DFS namespace servers and link servers based on that servers site location and its cost away from the callers site.

  • Lessons learned on CritSit or The importance of updating drivers

    As a premier field engineer I have a responsibility to do several on call shifts a year. The week just gone was one such on call shift learned the importance of updating drivers  during it. Allow me to elaborate.

    I got a call around 2AM Wednesday about a customer that needed help recovering the business after accidentally deleting the OU that had all the user accounts in the organisation. I assumed it would be a simple case of performing an authoritative restore and accepted the case. After turning up onsite I learned that authoritative restores had already been performed but when they ran the LDIF file to recover group membership, the destination servers hung. Without running the LDIF file, they had managed to get all the users replicate out successfully. But the DCs geographically spread across the country had inconsistent group membership information.

    As the LDIF file could not be replicated out and as the customer was desperate for resolution, we resorted to rebuilding all the DCs using the source DC backup and performing IFM based promotions. This turned out to be the resolution mechanism while we attempted root cause analysis. A colleague of mine joined me onsite and we recreated the problem in an isolated lab network. Analysis of the memory dump revealed issues with the storage related drivers. Specifically HpCISSs2.sys.

    We then tested the DC (All HP DL360 G5) after updating the following.

    • Controller Firmware (1.82 as per KB969550)
    • Disk Firmware (version HPDA for DH036ABAA5 although customer had DH036ABAA6 disks)
    • Controller Drivers (as per KB969550 we installed 6.14.0.32 from a smartstart CD)
    • Storport.sys (KB957910 )

    We could no longer reproduce the issue and a fix was now available. Yay! Customer decided to keep the DCs that were recovered using IFM online and to turn off the remainder and perform metadata and DNS cleanup. They are now planning to rebuild the remainder later after completely rebuilding them using latest HP SmartStart CD and Windows Server 2003 R2 SP2 CDs. Online DCs are to be also gradually updated with updates shown above.

    Lessons learnt were as follows.

    1. Importance of preventing accidental deletions of key OUs.
    2. Importance of applying all relevant Windows Server service packs/hotfixes (disable SNP, hotfixes for LVR, ntdsutil etc.)
    3. Importance of updating hardware firmware/drivers

    I have tried to keep this post short by skipping information about the long hours and sleep lost, the time it took for trial and error parts of resolution, challenges with 3rd party DNS that had to be circumvented using Windows DNS temporarily during recovery and the pain to customers while they were down for 3 days. But I assure you that failure to learn the lessons highlighted above, will be very painful if this happens to you.

    Regards

    M