Blog - Title

September, 2011

  • Friday Mail Sack: Robert Wagner Edition

    Hello folks, Ned here again. This week, we discuss:

    Things have been a bit quiet this month blog-wise, but we have a dozen new posts in the pipeline and coming your way next week. Some of them from infamous foreigners! It’s all very exciting, keep your RSS reader tuned to this station.

    On to the sackage.


    As far as I know, each computer name must be unique within single domain. How do domain controllers check this uniqueness? Most applications (ADUC, ADSIEDIT, etc.) displays entry common name that matches computer account name, which may not be unique.


    The samaccountname attribute – often referred to as the “pre-Windows 2000 name” - is what needs to be unique, as it’s the real “user ID.” That uniqueness isn’t  enforced by DCs when you create principals. You can create multiple computers, users, or groups with the same samaccountname. Well-written apps like DSA.MSC or DSAC.EXE will block you, but not because they are abiding by a DC’s rules:


    If you use a less polite or more powerful app, a DC will let you create a duplicate samaccountname. At the first logon using that principal though, the DC will notice the duplicate and rename its samaccountname to “$DUPLICATE-<something>”.

    If you want to see this for yourself:

    1. Configure AD Recycle Bin in your lab.

    2. Create a user and then delete it.

    3. Recreate the user manually in another OU (same name, samaccountname, UPN – just in a different location).

    4. Restore the deleted user to its previous location using the recycle bin.

    5. Note how the identical users exist and have an identical samaccountname.

    6. Logon as that user and the restored user will have its samaccountname mangled with $DUPLICATE.

    The “name” of the object is unique because it has to form a distinguished name, so you get that free thanks to LDAP. Only samaccountname and UPN will allow duplicates. And obviously, while I can create two computers with the same name in different OUs of the same domain, DNS is not going to be pleased and name resolution isn’t going to work – so this is all rather moot.


    When you were testing DFSR performance, what size file did you use for this statement?

    • Turn off RDC on fast connections with mostly smaller files - later testing (not addressed in the chart below) showed 3-4 times faster replication when using LAN-speed networks (i.e. 1GBb or faster) on Win2008 R2. This is because it was faster to send files in their totality than send deltas, when the files were smaller and more dynamic and the network was incredibly fast. The performance improvements were roughly twice as fast on Win2008 non-R2. This should absolutely not be done on WAN networks under 100 Mbit though as it will likely have a very negative affect.


    97,892 files in 32,772,081,549 total bytes for an average file size of 334,777 bytes. That test used a variety of real-world files, so there was no specific size, nor were they automatically generated with random contents like some of the tests.


    When using AD Users and Computers, what is the difference for unlocking between this:


    And this:



    The first one is sort of a "placeholder" (it would have been better as a button that grayed out when not needed, in my opinion) to let you know where unlocking happens. An actual account lockout raises the extra text and clicking that checkbox now does something.

    I prefer the way AD Administrative Center handles this:



    Even better, I can just find the locked accounts and unlock them right there.


    Or even betterer…er:


    Woot, let’s unlock everyone and hit the bar!

    Reminder: account lockouts are yuck. It’s just a way to create denial of service attacks. Use intrusion detection with auditing to find villains trying to brute force passwords. Even better, use two-factor auth with smart cards, which chemically neuters external brute force. If your security department thinks account lockout is better than this, get a new security department; yours is broken.


    Are RDC Recursion depth, Comparator buffer size, horizon size, hash window size RDC parameters configurable for DFSR?


    No, no, no, and no. :) All you can choose is the minimum size to use RDC, or if you don’t want RDC at all.



    That’s a great doc on how to write your own RDC application, by the way. It’s shocking how few there have been; we have an internal RDC copy utility that is the bomb. I wish we’d sell it to you, I like money.

    Not as much as this guy, obviously


    How can I use USMT offline migration with vendor-provided full disk encryption, like McAfee Endpoint Encryption, Symantec PGP Whole Disk Encryption, Check Point Full Disk Encryption, etc. I already know that with Microsoft BitLocker I just need to suspend it temporarily.


    Any official documentation on making WIN PE mount a vendor-encrypted volume would come from the vendor, as they have to provide a driver and installation steps for WIN PE to mount the volume, or steps on how to “suspend” outside of PE like BitLocker. For example, McAfee’s tool of choice seems to be EETech (here is its user guide). I’d highly recommend opening a case with the vendor before proceeding, even if they provide online documentation. Easy to lose your data forever when you start screwing around with encrypted drives.

    USMT does not have any concept of an encrypted volume (any more than Notepad.exe would); he’s a user-mode app.


    We use DFS Namespace interlinks, where a domain-based namespace links to standalone namespaces which then link to file shares. When we restart a standalone namespace root server though, clients start trying to get referrals as soon as it is available through SMB paths and not when its DFS service is ready to accept referrals. Is this expected?


    This is expected behavior and demonstrates why deploying standalone DFS root servers on non-clustered servers goes against our best practices. The client bases server availability on SMB, which is ready at that point on the standalone server – it doesn’t know that this is yet another DFSN referral, and it’s not going to work yet. Interlinks are gross, for this reason. If you must use this, cluster the standalone servers so that they can survive a node reboot for Patch Tuesday without hurting your users’ feelings.

    This is also why Win2008 (V2) namespaces were invented: so that customers could stop creating complex and fragile interlinked domain-standalone DFS namespaces in order to get around scalability limits. V2 scales nearly infinitely and if you deploy it, you can cut out the middle layer of servers and hopefully, save a bunch of dough.


    Have you ever seen the DFSR ConflictAndDeleted folder grow larger than the quota set, even when the XML file is not corrupt? E.g. 5GB when quota is set to the default of 660MB.


    Yes, starting in Windows Server 2008. Previously, a damaged conflictanddeletedmanifest.xml required manual deletion. In Win2008 and later, the DFSR service detects errors parsing that XML file. It writes “Deleting ConflictManifest file” in the DFSR debug log and automatically deletes the manifest file, then creates a brand new empty one. Any files that were previously noted in the deleted manifest are no longer tracked, so they become orphaned in the folder. Not an ideal solution, but now you’re less likely to run out of disk space due to a corrupt manifest. That’s the downside to using a non-transactional file like XML– if there’s a disk hiccup, voltage gremlin, or trans-dimensional rift, you get incomplete tags.

    I bet a bunch of DFSR admins are now checking their ConflictAndDeleted folders…

    Aha, there’s that spreadsheet I was looking for… eww, it’s got eggshell goop on it.

    Other Stuff

    Black Hat put up their 2011 USA presentations, make sure you browse around. The ones I found most interesting (and include a whitepaper, slide deck, or video):

    • How a Hacker Has Helped Influence the Government - and Vice Versa (the writer of L0phtcrack talks about being a PM at DARPA)
    • Corporate Espionage for Dummies: The Hidden Threat of Embedded Web Servers (endless web servers you didn’t even know you had running)
    • Killing the Myth of Cisco IOS Diversity: Towards Reliable, Large-Scale Exploitation of Cisco IOS (he who controls the spice, controls the universe!)
    • Easy and quick vulnerability hunting in Windows (he points out how to examine your vendor apps carefully, as your vendor often isn’t)
    • Faces Of Facebook-Or, How The Largest Real ID Database In The World Came To Be (or: the reason Ned does not use social media)
    • OAuth – Securing the Insecure (or: the other reason Ned does not use social media)
    • Battery Firmware Hacking (Good lord, start FIRES?!)
    • Hacking Medical Devices for Fun and Insulin: Breaking the Human SCADA System (Never mind fires, hacking people into diabetic comas!)

    There were several Apple and iOS pwnage talks too. I don’t care about those but you might, especially if you’re the new “owner” of those unmanaged boxes in your environment, thanks to the Sales Borgs wanting iPads for no legitimate reason… Another hidden cost of “IT consumerization”.

    A few people asked for a DOCX copy of the Accelerating Your IT Career post. Grab it here.

    Free Artificial Intelligence class online from a Research Professor of Computer Science at Stanford University. Looks neato for the price.

    Did you watch the Futurama season finale last night? The badly dubbed manga was a gigantic trough of awesome. I was right next to Katey Sagal at my hotel at Comic-Con. She is teeny. has a terrific infographic of the 45 years of Star Trek.

    The entire history of Star Trek is in this timeline infographic.
    Source: All about our solar system, outer space and exploration

    At Microsoft, you name your own computers and dogfooding means you can join as many to the domain as you like. My user domain alone has 58,420 computers and it’s a “small” one in our forest, so trying to control machine names is counterproductive even for bureaucratabots. I have a test server called Stink, and yesterday I needed to remote its registry. When I typed in the name, I found I wasn't the first to think of smelly nomenclature:

    The last one should be a band name

    For MarkMoro and all those like him, the 10 best ’80s cop show opening credits (warning: a couple sweary words in the text, but all movies totally SFW; this is old American network TV, after all).

    Finally, the best email conversation thread of last month:

    Jonathan: Darn you, Ned. I lost track of an hour on this ConceptRobots site.
    Ned: I should get a commission from the sites I push in Friday mail sacks.
    Jonathan: Yes… you wield your influence so adroitly.
    Ned: Or was it… androidly?



    Jonathan: I'm going to destroy your cubicle when you go to lunch.
    Ned: That's fair.


    Have a great weekend, folks.

    Ned "still has a thing for Stephanie Powers" Pyle

  • Accelerating Your IT Career

    Your career isn’t win or lose anymore, it is win or die. The days of guaranteed work, pensions, and sticking with one company for fifty years are gone. Success has returned to something Cro-Magnon man would recognize: if you’re good at what you do, you get to eat.

    I recently spoke to university graduates about their future as new Microsoft engineers. For the first time, that meant organizing my beliefs. It distills four simple pillars: discipline, technical powerhouse, communication, and legacy. In the tradition of Eric Brechner and in honor of Labor Day, I’d like to share my philosophy.


    Learn constantly, not just when life is forcing you. Read every trustworthy article you can get your hands on, before you need to know it; the time to learn AD replication is not when its failure blocks your schema upgrade. Understanding architecture is the key to deploying and troubleshooting any complex system. If you get nothing else from this post, remember that statement - it can alter your life. For Directory Services, start here.

    Don’t be good at one thing - be amazing at a few things, and good at the rest. We all know someone who's the expert on X. He guards X jealously, making sure he is "indispensable.” Notice how he’s always in a lousy mood: he's not allowing anyone to relieve his boredom and he lives in fear that if anyone does, he'll be replaced. Learn several components inside and out. When you get jaded, move on to a few more and give someone else a turn. You'll still be the expert while they learn and if things get gnarly, you can still save the day. Over time, you become remarkable in many areas. Keep your skills up on the rest so that you can pinch hit when needed. Surround yourself with smart people and absorb their knowledge.

    Admit your mistakes. The only thing worse than making a mistake is trying to cover it up. Eventually, everyone is caught or falls under a career-limiting cloud of suspicion. Now colleagues will remember times they trusted you, and won’t make that "mistake" again. Plead guilty and start serving community service, where you help the team fix the glitch.

    Get a grip. It's never as bad as you think. Losing your composure costs you concentration and brainpower. Remaining emotional and depressed makes you a poor engineer, and a lousy person to be around to boot. Learn how to relax so you can get back to business.

    Never surrender. Your career path is a 45-degree angle leading up to infinity, not an arc - arcs come back down! Keep learning, keep practicing, keep refreshing, keep growing. Keep a journal of "I don't know" topics, and then revisit it weekly to see what you've learned. IT makes this easy: it's the most dynamic industry ever created. In my experience, the Peter Principle is usually a self-induced condition and not the true limit of the individual.

    Technical Powerhouse

    Figure out what makes you remember long term. There is a heck-of-a-lot to know when dealing with complex distributed systems - you can't always stop to look things up. Find a recall technique that works for you and practice it religiously. You’re not cramming for a test; you’re building a library in your brain to serve you for fifty years. No amount of learning will help if you can’t put it to good use.

    Be able to repro anything. When I first came to Microsoft, people had fifteen computers at their desk. Thanks to free virtualization, that nonsense is over and you can run as many test environments as you need, all on one PC. "Oh, but Ned, those virtual machines will cost a fortune!" Gimme a break, it’s walking-around money. A lab pays for itself a thousand times every year, thanks to the rewards of your knowledge and time. It's the best investment you can make. Study and memory are powered by experience.

    Know your dependencies. What does the File Replication Service need to work? DNS, LDAP, Kerberos, RPC. What about AD replication? DNS, LDAP, Kerberos, RPC. Interactive user logon? DNS, LDAP, Kerberos, RPC. Windows developers tend to stick with trusted protocols. If you learn the common building blocks of one component, you become good at many other components. That means you can troubleshoot, design, teach, and recognize risks to them all.

    Understand network captures. It's hard to find an IT system talking only to itself. Notepad, maybe (until you save a file to a network share). There are many free network capture tools out there, and they all have their place. Network analysis is often the only way to know how something works between computers, especially when logging and error messages stink - and they usually do. I'd estimate that network analysis solves a quarter of cases worked in my group. Learn by exploring controlled, working scenarios; the differences become simple to spot in failure captures. Your lab is the key.

    Learn at least one scripting language. PowerShell, CMD, VBS, KiXtart, Perl, Python, WinBatch, etc. – any is fine. Show me an IT pro who cannot script and I'll show you one that grinds too many hours and doesn't get the bonus. Besides making your life easier, scripting may save your business someday and therefore, your career. An introductory programming course often helps, as they teach fundamental computer science and logic that applies to all languages. This also makes dependencies easier to grasp.

    Learn how to search and more importantly, how to judge the results. You can't know everything, and that means looking for help. Most people on the Internet are spewing uninformed nonsense, and you must figure out how to filter them. A vendor is probably trustworthy, but only when talking about their own product. TechNet and KB trump random blogs. Stay skeptical with un-moderated message boards and "enthusiast" websites. Naturally, search results from AskDS are to be trusted implicitly. ;-P


    Learn how to converse. I don’t mean talk, I mean converse. This is the trickiest of all my advice: how to be both interesting and interested. The hermit geek in the boiler room - that guy does not get promotions, bonuses, or interesting projects. He doesn't gel with a team. He can't explain his plans or convince anyone to proceed with them. He can't even fill the dead air of waiting… and IT troubleshooting is a lot of waiting. Introverts don’t get the opportunities of extroverts. If I could learn to suppress my fear of heights, you can learn to chat.

    Get comfortable teaching. IT is education. You’re instructing business units in the benefits and behavior of software. You're schooling upper management why they should buy new systems or what you did to fix a broken one. You're coaching your colleagues on network configuration, especially if you don’t want to be stuck maintaining them forever. If you can learn to teach effortlessly and likably, a new aspect to your career opens up. Moreover, there's a tremendous side effect: teaching forces you to learn.

    Learn to like an audience. As you rise in IT, the more often you find yourself speaking to larger groups. Over time they become upper management or experienced peers; an intimidating mix. If you let anxiety or poor skills get in the way, your career will stall. Arm yourself with technique and get out in front of people often. It's easier with practice. Do you think Mark Russinovich gets that fat paycheck for his immaculate hair?

    Project positive. Confidence is highly contagious. When the bullets are flying, people want to follow the guy with the plan and the grin. Even if deep down he's quivering with fear, it doesn’t show and he charges forward, knowing that everyone is behind him. People want to be alongside him when the general hands out medals. Self-assurance spreads throughout an organization and you'll be rewarded for it your whole career. Often by managers who "just can't put their finger" on why they like you.

    Be dominant without domineering. One of the hardest things to teach new employees in Microsoft Support is how to control a conference call. You’re on the phone with a half dozen scared customers, bad ideas are flying everywhere, and managers are interrupting for “status updates”. You can’t be rude; you have to herd the cats gently but decisively. Concentration and firmness are paramount. Not backing down comes with confidence. Steering the useless off to harmless tasks lets you focus (making them think the task is important is the sign of an artist). There's no reason to yell or demand; if you sound decisive and have a plan, everyone will get out of the way. They crave your leadership.


    Share everything. Remember "the expert?" He's on a desert island but doesn’t signal passing ships. Share what you learn with your colleagues. Start your own internal company knowledgebase then fill it. Have gab sessions, where you go over interesting topics you learned that week. Talk shop at lunch. Find a reason to hang out with other teams. Set up triages where everyone takes turn teaching the IT department. Not only do you grow relationships, you're leading and following; everyone is improving, and the team is stronger. A tight team won't crumble under pressure later, and that's good for you.

    Did you ever exist? Invent something. Create documentation, construct training, write scripts, and design new distributed systems. Don’t just consume and maintain - build. When the fifty years have passed, leave some proof that you were on this earth. If a project comes down the pipe, volunteer - then go beyond its vision. If no projects are coming, conceive them yourself and push them through. The world is waiting for you to make your mark.

    I used many synonyms in this post, but not once did I say “job.” Jobs end at quitting time. A career is something that wakes you up at midnight with a solution. I can’t guarantee success with these approaches, but they've kept me happy with my IT career for 15 years. I hope they help with yours.

    Ned "good luck, we're all counting on you" Pyle