Kevin Remde's IT Pro Weblog

  • Doh! (What I learned about Windows Failover Clustering. The hard way.)

    Okay.. I feel like sharing this because it’s pretty stupid, but in a geeky-sort-of-way the solution was interesting enough to share.  Think Chicken & Egg. (or “Catch-22”).

    As the title of this post suggests, the subject is Windows Failover Clustering.  For those of you who are not familiar with it, Windows Failover Clustering is a built-in feature available in Windows Server 2008 R2 Enterprise and Datacenter editions.  Along with shared storage (for which we used the free iSCSI Software Target from Microsoft to implement), it provides a very easy-to-configure and use cluster for serving up highly available services.  In our case, this would be virtual machines running on two clustered virtualization hosts.

    The Background

    Our 5 Dells, and Harold's Back

    As a training platform, but primarily for use as a demonstration platform for our presentations (and certainly more real-world than one laptop alone can demonstrate), our team received budget to acquire several Dell servers.  We found a partner (Thank you Groupware Technology!) who was willing to house the servers for us.  The idea was that we, the 12 IT Pro Evangelists (ITEs) in the US would travel to San Jose in groups of 3-4 and do the installation of a solid private cloud platform, using Microsoft’s current set of products (Windows Server 2008 R2 and System Center).  This past week I was fortunate enough to be a member of the first wave, along with my good buddies Harold Wong, Chris Henley, and John Weston.  The goal was to build it, document it, and then hand if off to the next groups to use our documentation and start from scratch, eventually leaving us with great documentation, and a platform to do demonstrations of Microsoft’s current and future management suites.

    The Process

    We all arrived in San Jose Monday morning, and installed all 5 server operating systems in the afternoon.  We installed them again Tuesday morning.

    “Huh?  Why?”

    It’s a long story involving how Dell had configured the storage we ordered.  We needed to swap some drives between machines and set up RAID and partitioning in a way that was more workable to our goals.  I’ll leave that discussion for one of my teammates to blog about.

    Anyway, once we had the servers up, I installed and configured the Microsoft iSCSI software target on our “VMSTORAGE” server, and configured two other servers as Hyper-V Hosts in a host cluster, with Windows Failover Clustering and CSV storage.  By the end of the week we had overcome hardware, networking, missed-BIOS-checkmarks (did you know that Hyper-V will install, but you can’t actually use it if you somehow miss enabling Virtualization support on the CPU on one of the host cluster machines?  Who’da thunk it?!) , we had 5 physical and a half-dozen virtual servers installed and running, with Live Migration enabled for the VMs in the cluster.  Our domain had two domain controllers; one as a clustered, highly-available VM, and the other as a VM that was not-clustered, but still living in the CSV volume; C:\ClusterStorage\Volume1 in our case.  (That’s a hint, by the way.  Do you see the problem yet?)

    Brain ScanThe “Doh!”

    Homer Jay SimpsonOne of the many hurdles we had to overcome early on was an inadequate network switch for our storage network.  100Mbps wasn’t going to cut it, so until our Gig-Ethernet switch arrived on Friday, Harold used his personal switch that he carries with him.  On Friday before we left for the airport, we shut down the servers and let the folks there install the new switch.  Harold need his switch back at home.

    But in restarting the servers, here’s the catch: Windows Failover Clustering requires Active Directory.  The storage mount-point (C:\ClusterStorage\Volume1) on our cluster nodes requires the Failover Clustering.  And remember where I said our domain controllers were?

    “Um.. So… Your DCs couldn’t start, because their location wasn’t available.  And their location wasn’t available, because the DC’s hadn’t started.  And your DC’s couldn’t start, because their storage location wasn’t available, and… !!”

    Bingo.  Exactly.  Chicken, meet Egg.  It was our, “Oh shoot!” moment.  (Not exactly what I said, but you get the idea.)

    “So how did you fix it?”

    I’ll tell you…

    The Resolution

    Ah ha!Our KVM was a Belkin unit that supports IP connections and access to the machines through a browser.  We configured it to be externally accessible.  So I was able to use that to get in to the physical servers and try to solve this “missing DCs” puzzle; though to make matters much more difficult, the web interface for that KVM is really, REALLY horrible.  The mouse didn’t track to my mouse directly, no ALT+ key support, TAB key didn’t work.. I ended up doing a lot of the work from a command-line simply because it was easier than trying to line up and click on things!  Perhaps in a future blog post I will give Belkin a piece of my mind regarding this piece-of-“shoot” device…

    So, my solution was based on two important facts:

    1. the Microsoft iSCSI Target creates storage “devices” that are really just fixed-size .VHD files, and
    2. Windows Server 2008 R2 natively supports the mounting of VHD files into the file system.

    “Ah ha!  So on the storage machine, you mounted the .VHD file that was your cluster storage disk, and you copied out the .VHD file from one of the domain controller VMs!”

    Yeah.. that’s basically it.  Though I did have one problem.  The .VHD file was in-use; probably by the iSCSI Software Target service.  So when I tried to attach it, the OS wouldn’t let me.

    Fortunately I found that by stopping that “Microsoft iSCSI Software Target” service  on the storage server (I also stopped the “Cluster Service” on the two Hyper-V cluster nodes), I was able to attach to the .VHD, navigate into it, and copy out the .VHD disk for the needed Domain Controller.  (Actually, I also removed the .VHD from its original location.  I didn’t want the DC to come alive again when the storage came back online, if the identical DC was already awake and functioning.)

    So after that, it was as simple (?) as this:

    • Re-create the DC virtual machine on one of our standalone Hyper-V Servers (using local storage this time),
    • Attach the DC’s retrieved .VHD file to the new machine,
    • Fire it up,
    • Reconfigure networking within the DC (the running machine saw it's NIC as a different adapter, but that was easily fixed) and verify that it was alive on the network,
    • And then restart the iSCSI Target service on the VMSTORAGE server, and then the Cluster Service on the Hyper-V nodes. 

    Everything came back to life almost immediately; including the Remote Desktop Gateway that we had configured so that we could remotely connect to the machines in a more meaningful, functional way.

    So the moral of the story is:
    When you’re building your own test lab, or even considering where to put your DCs in your production environment, make sure you have at least one DC that comes online without depending upon other services (such as high-availability solutions) that, in turn, require a DC to be functioning. 

    All-in-all, it was a great week. Smile


    Do you have any similar stories?  Share them with us in the comments.  We’d love to hear ‘em!

  • Where is that evaluation download?

    DownloadWe IT Pro-focused Technical Evangelists are often asked, “Hey.. Where can I find the download to try out such-and-such software?”

    And after I find out what they actually meant by “such-and-such”, I usually just tell them essentially to “google it on Bing”.

    But that’s not sufficient.

    So what I am doing instead, on this page, is to give you a list of the most important, most commonly requested downloads and evaluations.  And here are the latest and greatest:

    We also have evaluations of our recent-yet-older products:

    If you're interested in evaluating the newly released System Center 2012 R2  management product, and all of the components therein, you can get them here:

    Do you want to try Windows Azure for FREE?

    "Yes!  Yes I do!"

    Here you go:

    How about some free online training?  We've got literally hundreds of hours of up-to-date training at the Microsoft Virtual Academy:

    Try them all!

    And if this is not exactly what you need, you can also go to the TechNet Evaluation Download Center and find everything an IT Pro could ever want to try out.

  • Breaking News: More New System Center 2012 RCs and Betas Available Today

    Great news!  More betas and release candidates of the System Center 2012 tools are available today for download!  Like many of you, I have been waiting patiently for these gems, and I can’t wait to start playing with them.  These round out the pre-releases of tools that allow you to support private and hybrid cloud solutions.

    Released today:

    • System Center Configuration Manager 2012 Release Candidate (RC)
    • System Center Endpoint Protection 2012 RC
    • System Center Service Manager 2012 Beta
    • System Center App Controller 2012 Beta (formerly codename “Concero”)

    System Center Configuration Manager 2012 has already had a couple of betas, so it’s nice to see that it is now up to Release Candidate status.  Some of the improvements:

    • Diverse mobile device management support through Exchange ActiveSync including support for iOS, Android, Symbian as well as Windows mobile devices.
    • User-centric application delivery across multiple devices
    • Integrated settings management with auto-remediation
    • Integrated desktop antivirus management with System Center 2012 Endpoint Protection
    • An Improved, modern administration user interface with built in search concepts

    Furthermore, new capabilities released in this RC release include:

    • Improved endpoint protection functionality, with integrated setup, management and reporting for System Center 2012 Endpoint Protection.
    • Improved application catalog design that provides a better, more responsive experience when requesting and downloading applications.
    • New support for Windows Embedded devices, including Windows Embedded 7 SP1, POSReady 7, Windows 7 Think PC, and Windows Embedded Compact 7.
    • Improved client health checks for Configuration Manager services and features
    • Improved compliance enforcement and tracking, with the ability to create dynamic collections of baseline compliance and generate hourly compliance summaries.
    • Platform support for deep mobile device management of Nokia Symbian Belle devices. Pending a platform update by Nokia later this calendar year for these devices, customers will be able to try out the management of Nokia devices with ConfigMgr.
    • Additional scalability and performance improvements.

    System Center Endpoint Protection 2012 is a name that you have never heard before today.  That’s because it is an official re-branding of what was Forefront Endpoint Protection.  “This name change better reflects the integration between management and security that we have been working towards…” says it pretty well.  I think the fact that this solution is not only integrated with, it is “built upon” System Center Configuration Manager is a key part of this, too.  It makes sense. 

    “Sounds great.  Which betas are available today?”

    The first of the two betas released today is System Center Service Manager 2012, which is, as the name implies, the service management part of the solution. 

    “You mean, like a help-desk support ticket manager?”

    Yes, and much more.  I know I’m just scratching the surface of all it can do when I say this – but think about how you will provide a self-service interface for your users and the business units you support.  Service Manager is the tool that you will use to create that portal, and then to allow you to act upon requests; and not just manually.  Those requests can be handled by you and your staff, or trigger some automation to take care of the request.  (See System Center Orchestrator 2012).  Also, Service Manager 2012 adds the tracking of SLAs.  So if you have a service that you have promised a certain response time or up-time for, Service Manager can track your actual performance against those goals.

    For a private cloud, you have to grant people the ability to request and acquire services without having to know the details of the underlying architecture (the “fabric” - Storage, Compute, Networking).  Service Manager is one way to provide that interface.  More details about the current version, Service Manager 2010, can be found here.

    Happily managing cloudsAnd the other beta available today, formerly known as codename “Concero”, is System Center App Controller 2012.

    “Let me guess.. It controls your apps?”

    Well.. yeah.. in a sense.  More accurately, though; this is the tool that bridges the gap for the management of private and public cloud services.  Once the fabric is configured, and the clouds are defined; meaning: once you folks in the datacenter have created the building blocks for the services you provide, and have defined the abstracted layer at which your users can request and manage services (See?!  Sometimes it’s just easier to say “cloud”, isn’t it!), or through which you can make changes such as scaling-out your services. For example, adding more instances of the web-tier in a muli-tiered application, for example.

    “Couldn’t I just use System Center Virtual Machine Manager to do that?”

    Sure, you could – if that person has the VMM 2012 admin console installed.  And if all of your clouds are local.  But what the application owner doesn’t have that console installed?  Or what they also manage multi-tiered applications (or call them “services”) running in some other cloud like Windows Azure?

    “Wait.. what?  I can manage my Windows Azure-based services with this?”

    Yep.  As I said, App Controller bridges that gap for you.  It’s a self-service portal where the application owners can go to manage their applications that are running in your private and/or public cloud spaces.  For those folks, it’s not about managing the infrastructure or the plumbing you’ve provided.  For them, “It’s all about the app.” 

    You can find instructions for installing and configuring System Center App Controller 2012 here.

    “So.. the obvious question remains: When will these products be released?”

    They are shooting for the first half of calendar year 2012.


    DOWNLOAD and try these tools out.  But wait just a minute and let me start my download first.

  • Here Are Your Microsoft Virtualization Event Resources

    Happily evaluating Hyper-V and replacing VMwareNovember 2011 - January 2012

    IMPORTANT Software Evaluations – Get ‘em here:

    "WAG" = Windows Administrator Groups (on LinkedIn) with Subgroups for user groups by geography –

    Future Live, In-Person Event Information:
    TechNet Events (for IT Pros):
    MSDN Events (for Software Developers):

    Session Slides:

    Microsoft Virtualization: On Your Terms
    Windows Virtualization Home -

    Hyper-V getting Started Guide -

    Step-by-Step Guide to Getting Started with Hyper-V (download)

    Case Study: Target Corporation

    Case Study: Dartmouth-Hitchcock Medical Center

    Virtual Machines and Supported Guest Operating Systems

    Microsoft server software and supported virtualization environments

    Bing it: “Managing Hyper-V with WMI” / “Managing Hyper-V with PowerShell”

    Hyper-V Security Guide -

    The TechNet Virtualization Scenario Hub -

    Virtualization Community Resources -

    Virtualization Resources in the TechNet Library -

    Windows Virtualization Team Blog -

    VMware Compare: Project Virtual Reality Check -

    VMware Compare: Enterprise Strategy Group and

    “VMLimited” Video -

    Microsoft Virtualization: From the Metal Up

    The Private Cloud Home Page -

    “Ready Now” Private Cloud Solutions -

    The National Institute of Standards and Techology (NIST) Definition of Cloud Computing -

    Microsoft’s Definition of Private Cloud -

    Microsoft Virtual Academy -

    Microsoft Cloud Home Page –

    Online Services -

    Microsoft’s Global Foundation Services (The Datacenter People) -

    Hyper-V Cloud Deployment Guides -

    System Center Virtual Machine Manager 2008 R2 –

    SCVMM Self-Service Portal (SSP) 2.0 SP1 -

    System Center 2012 Evaluation Download Center -

    SCVMM Team Blog -