TFS 2010 email alerts fail unless using “Portfolio Project” instead of “Team Project”

We use TFS 2010 for our work item tracking, source control, and build services.  Beta 2, which is the currently released version of TFS & Visual Studio 2010, is nice and a very strong upgrade for those using 2008.  I’ve spent a lot of time getting familiar with TFS, a lot more than in the past, and I’ve found a few things that threw me off and I will share them with you as time permits.

A key foundation of our escalation process is the ability to do alerting as items are introduced into the system, or modified, or lastly moved to a new state.  We use email for our alerting and I’ve found a little quirk that seems to be “well known” (by the TFS product group) but maybe not to the world as I discovered.  We used several stock queries that we already have built and stored in TFS as the basis for our alert queries.

Turns out, this didn’t work…So here goes some guidance from the pain & sweat that I went through to get it working.

Pre-Req for using Alerting

Prior to using email alerting, you are required to run TFS using a service account.  The service account should be a email-enabled (in our case, Exchange mailbox) so that it can successfully send email in your organization.  To setup your alerts in TFS 2010, you do the following:

1. Open the Team Foundation Server Administrator’s Console (TFS 2010 only)
2. Click Application Tier

1. Click Change Alert Settings
2. Enter your SMTP server account & mailbox-enabled user account

Testing Service Account Email

I recently asked one of my resident TFS experts to help me during this process of troubleshooting alerts & email.  I asked him how one might could troubleshoot that Email alerting has been setup correctly in TFS.  I was surprised that he wasn’t exactly sure what I meant so I realized that it wasn’t a common question.  In my case, I didn’t know whether my alerting problem was email alert configuration setup incorrectly or whether it was the query evaluator so I wanted to “test” email.

Server-side testing for TFS Email

The first steps to try is to get out of TFS and validate that you can successfully, from your TFS application server, connect to your email server and send an email.  Let’s walk through this…

1. Add the Telnet Client on your Server using Server Manager
2. Open an elevated CMD prompt
3. Type telnet “mail server FQDN” 25 (without quotes)
4. Depending on your email configuration for your environment, you can “Auth” or use anonymous (do what fits your environment)
5. Follow the instructions outlined in this article using section titled “Use Telnet on Port 25 to test SMTP Communication”

Ahh, this failed – what now!

Before you freak out, don’t worry.  Wait… you already freaked so don’t throw your keyboard as there might be a logical explanation for this.  The one thing that I see again & again with administrators or Developers is they forget that Windows Server 2008 & above ships with the Windows Firewall enabled.  The typical issue that occurs is that a firewall is blocking the connection out to your SMTP server.  This can also be the problem in locked down network environments so ensure you have enabled SMTP in your network firewall.

To enable SMTP for the Windows Firewall in Server 2008, do the following:

1. Open the Windows Firewall via Start –> Run –> firewall.cpl
2. In the left hand pane, click Advanced settings
3. Click Change Settings
4. Highlight Outbound Rules
5. Under Outbound Rules in the Actions pane, click New Rule
6. Select Port and click Next
7. In the textbox for Specific remote ports, enter 25

Finish the configuration by clicking Next and you will now have the firewall port opened for outbound SMTP.  Unless you are using a Telnet Server, you shouldn’t need an inbound rule.  Try your telnet test again from the steps above.

Successful Alerting using Email requires “Portfolio Project” rather than “Team Project”

The easiest approach for alerting is to start with a very, very simple query such as “when something is assigned to me.”  This is a stock query that can be created by the system and is a good blueprint to what is a working query event to trigger the alert.  Afterwards, you can then start making more complex alerts to meet your business needs.

In my case, I got the very simple stock to work with no problem.  However, when I started creating an alert from a "[Blank Rule]” is when all #@## started happening and nothing was getting emailed.  After doing an evaluation a bit between the working & non-working queries, I noticed that the stock alerts shipped with TFS all used “Portfolio Project” rather than “Team Project”.  This was interesting to me so after making a slight modification to my alert the spam door was opened and it flooded my local mailbox.  After investigating for a few days off & on, let’s just say this was the kind of spam I was happy to see!

To use email alerts in TFS 2010, you must have have the following in your query “Portfolio Project” = ‘TFS Project Name’.  This will cause the event subscription service evaluator to work correctly and now you will start receiving your email.

Cool Troubleshooting Tip for Alerts

Grant Holliday’s blog provided some useful tracing capabilities for testing alerting.  For purposes of this post, I just wanted to give a shout out to his post as it helped quite a bit in getting to the bottom of this issue.  However, I did just use some old troubleshooting steps used by all of us – find one that works and then determine the difference.  This technique works quite often!

-Chris

Understanding IIS Bindings, Websites, Virtual Directories, and lastly Application Pools

In a recent meeting, some folks on my team needed some guidance on load testing the Web application that one of my feature teams are developing.  The questions on load testing subsided rather quickly and prior to pluggin my ears with my headphones for my ZuneHD, I was stepping out of the room when one said, “Can I ask you a question about connections in IIS…”

This simple question led to a 20 minute conversation … which resulted in this blog post and some serious deja vu for me as it was a flash back to my IIS days.

Bindings:  Did you say “Bindings?”

So you’ve been tasked with development of a new Web application to be hosted on IIS (any version)?  The first thing on your mind is usually the design of the Website, how the application will interact with the middle-tier, and usually security.  This is a great start in the design process.  However, let’s not forget that often jumping into this level of design will mean that later on your going to make some other decisions a bit more tricky.

It starts with these questions:

1. Am I going to host everything in one IIS Website?
2. Will I use an “existing” Website like the Default Web Site or create my own?
3. Will some of the site require secure authentication using SSL?

The first thing that often happens with developers posed with these questions are they say these aren’t important but I quickly smile and say, “We’ll see”.

The primary reason that these questions are important are around the fact Websites are accessed by every client using bindings.  The end-user of your Web application(s) don’t know they are using bindings because they are usually hidden behind a nice, pretty “Web address” using DNS.  If you don’t have the answer of how many Websites your Web application will utilize then you are going to be struggling when you are upset that you are limited to “rules” governed by directories.

You see, Websites have something called Server Bindings which represent the underlying address, port, and potentially a host header that your Website is accessed using.  Do you think that HR staff would be happy if their Website is accessed using the same bindings as your company’s intranet?  I would venture to guess the answer is no.

Bindings 101:

A typical binding for Websites are in the form of IP:Port:HostHeader.  For all versions of IIS that anyone reading this in 2010 care about (version 6.0 and higher), the default Web Site binding is set to *:80:* meaning that all requests to that server will land at that site.

Valid Bindings:

For option where you utilize IP address as the “unique” point for access, you will need to disable HTTP.sys default behavior of listening on all IP addresses configured on your server.  For example, if you have 192.168.1.1 and 192.167.1.2 configured as IP addresses on the same server the default behavior “out of the box” is to listen on port 80 no matter if you do the binding in the IIS Manager.

To change this behavior, you will need to configure HTTP.sys’s IPListenList (future blog I guess as there is no MS documentation on the topic) to only listen on a specific address.  This is done via the registry or NetSH depending on what you are most comfortable with.

Figure 1:  Default setting for IPListen (blank equals *:80:*)

In short, if you plan to utilize a Website then know what your bindings will be and where your application will live in production.  If a shared server, you can bet you will need a Host Header or a unique IP address so think ahead and get ‘er going.

Websites versus Application Pools

There are so many reasons that Websites & Application Pools are confused that I don’t have enough time to do a post on it.  I’m not going to try and solve the debate here, but instead, I’m going to try and educate you on what the fundamental difference between the two are.  In discussions with IT Pro’s & Developers, rarely will you have any of them “admit” they know what each is and when to utilize one or the other but my guess is that over 70% of them don’t know.

Thus, I hope for readers out there who used their decision engine (nice plug, ay?) to find this reading will enjoy learning this topic and we can together reduce this 70% to a much lower number…

Websites:  Container of physical and virtual directories

It really is simple.  A website is nothing more than a container of physical and virtual directories that have a unique “Server Binding” for clients to access the content.  The default container in IIS, for years, has been %systemdrive%\inetpub\wwwroot unless you are doing a unattended install in IIS 6.0 which allowed you to put the files where ever you choose.

Path + Server Binding = Website  … It really is easy. 

NOTE:  Their is a serious obmission completely on purpose here.  As you can see, Websites have nothing to do with memory, server processes, bitness, or performance.  They simply are a path + binding.

When to choose a “Website”

With that understanding, you can now make an educated guess as to how to answer the question of whether you should create a new Website or use an existing one.  However, I will make sure to share it in case you missed it - “You decide whether to create a new Website based on whether you would like to have a unique binding for your Website or if you want to use an existing one.”

The path isn’t important in this equation as I can create a 1000 Websites all pointing to exactly the same path and there is absolutely no problems with doing this (of course, why in the heck would you do this is a great question).  The key decision here is that any physical or virtual directory will always use the bindings of the Website so ensure that you understand this.

When to choose directories?

If there is a website which is already running and utilizing a binding that you would prefer to use then you should select this option.  This allows you to utilize the resources of the parent site, if interested, as the server (e.g. IIS) will handle any requests over the same connection(s).  For example, any physical or virtual directory in the IIS path is still considered “/” to the server as it builds out the URI because the bindings are already mapped at the site level.  This means that URLs can be re-written to go various different places within the folder hiearchy over the the same connection since the binding is the “same”…

If you choose to put your Web application in its own Website then you will have to use the HTTP 302 redirection capability (exposed via Server.Transfer or other methods) to push the request elsewhere. 

So, as you can see, thinking ahead of time about whether you are building a Website for your application or whether it is a child directory (physical or virtual) is an important piece of information to have locked early, early on!

Application Pools:  Container of applications

The very nature of application pools is to do the obvious, contain a single or multiple applications.  The introduction of application pools in IIS 6.0 caused some head scratching but in today’s world where IIS 6.0 is very engrain in enterprises and the Web leads to less scratching.  However, again, development teams often make mistakes by not “thinking” about application pools and there impact on their new applications they are building.  Hence the reason we will chat about this some more today…

First Concept…  Windows Process = Application Pools *not* Windows Process = Website

Second Concept… Process Management = Application Pools *not* Process Management = Website

When to create a new Application?

By default, IIS 6.0 or IIS 7.0 must have a single application to run.  If the root application (/) is deleted or corrupted then IIS will fail, as in, not serve your application.  Both products ship with a default application which is assigned to the Default App Pool.  I should not this is only if no other Microsoft features have been installed and instead we have the basic Web server installed.

As you can see, there is also a Classic .NET AppPool but no applications are currently bound to it.  In IIS 7.0, any managed code application can choose to utilize the Integrated Pipeline or to use the classic ASP.NET pipeline which is present in IIS 6.0.

By default, you as a developer of a Web application can choose to simply inherit the settings of the parent Application Pool (/) and choose to not create your own.  This is absolutely fine.  So you might ask, what do I get from choosing this route?  I’m glad you asked because it is important to know that you get all the settings of the parent application pool which in this case is the DefaultAppPool.

These settings include the following:

As you can see, you need make some important decisions early on or you are going to change a lot during the development process. 

When to create a new Application Pool?

Well, it sounds like I’m best to create a new application pool for all my Web applications.  I would say you’ve been suckered and convince that this is the best without all the facts.  The fact is that creating an application pool includes understanding better your strategies for security such as do you run Network Service, a Domain Service Account, etc. that starts to complicate things very quickly.  The one thing that many manage code developers often love to take advantage of is the caching capabilities of processes and manage code.  Each time you create a application, bind it to its own unique application pool then you are limiting your ability to share cache with other .NET applications running on the same box.  For example, if you have the Microsoft Enterprise Library in use all throughout your Web applications then you can often utilize caching to improve performance.  As soon as you break these out into different process boundaries (e.g. App Pools) then you no longer have that benefit.

There are a number of these types of examples listed above that drives the question – Do I use my own application pool or do use another one already running?  I’m happy to get posed a question via comments or email regarding this topic and see what your situation is and make my suggestion :)

Nonetheless, be careful in your planning when utilizing your own Application Pools and share resources where possible is my guidance.  There are absolutely situations where one might choose to always go hard line with creation of app pools for every new Web development project.  I just caution you and say, “Not so fast my friend… “

Summary

In today’s post, I went for a “blast from the past” theme as to just feel the power between… well let’s leave it there.  I spent several years focusing on our IIS product and a great deal more focusing on helping customers struggle with the product with our lest than stellar performance of training on IIS.  You now have learn.iis.net so for that I say woohoo!  So today was fun…

To summarize, I hope that I gave you non-IIS geeks a bit of understanding in one of the fundamental foundations of ‘Getting Started with IIS’ concepts.  I often see many folks fail to understand when to use a Website, or a virtual directory and my aim with this post is to give your brain a bit of a quiz to help you figure out which “way” you want to go when you are developing applications based on IIS. 

In yesterday’s meeting, that is all I did was give my feature team a lot to think about and then I will let them choose what avenue they would like to go…

Thanks,

-Chris

P.S. – I hope that my POW feature team chooses the right selection, but better yet, I hope they can back it up when I quiz them on their decision.  <grin>

Failover Cluster Networks & Local Networks – Phantom Network listed in Failover Cluster Manager

You’ve recently had a request to add a few virtual machines to your Clustered Hyper-V farm and you’ve obliged them and created it using SCVMM’s administrator console.  You’ve created the network as a “Local Only” meaning that the VMs can communicate with each other as well as the Parent partition on the physical host (as opposed to private where the VMs only see other, no physical host).  Your customer is happy…

Phantom Network shows up in Failover Cluster Manager

After adding the few virtual networks for your customer, you’ve now noticed that exactly one network is listed in your list of “available” cluster networks as we show in the following -

You know this has something to do with the recently added networks but the “math” isn’t adding up as you actually added 5 new local only networks.  Why isn’t there five of these listed?

What is this “Cluster Internal Network?”

After some investigation, and I will not lie – reaching out to folks asking what the @#$@ – I’ve found the answer for how “Clustering” see’s these private and local networks.  In my case, I added 5 sandbox environments for our team to test and use Configuration Manager vNext and with each there is a private network created on each host.  Therefore, in SCVMM Admin Console I see the following list of “like” networks -

So why do I only see 1 “Cluster Internal Network” when I added 5 of them?

Cluster Recognizes “Like” Networks when configured on same Subnet

The answer to this question is so simple that it is scary that I couldn’t make sense out of it.  Did you?  I sure didn’t…

Each Internal Network, since local only to the host it is currently running on was configured with the exact same subnet configuration.  Thus, each virtual NIC was configured with the following characteristics:

IP:  192.168.0.X

Subnet:  255.255.255.0

Since this is the exact same “network” at Layer 3, Cluster is aware that all 5 represent the same exact network.  That’s it.  That is why you create 2, 5, or 50 but see only one Internal Network in your cluster networks.  Simple, smart, and very, very logical…

Disable Cluster Communication on these Networks

Because these are local networks, you should disable cluster communication across these NICs.  This is a best practice to control which NICs Cluster should use for cluster communication.

To do this, do the following:

1. Open Failover Cluster Manager
2. Navigate to Networks
3. Select the Local\Private network in question, and right-click and select properties
4. Select the radio button “Do not allow cluster communication on this network”

Summary

Never fear, there’s always an answer somewhere near.  The digging and understanding of new technologies such as clustering + virtualization makes it a blast as we discover just how things work under the hood.  I hope this little nugget of information will help save you time scratching your head.

Thanks,

-Chris

Visual Studio 2008 – Connecting to TFS 2010 Server

We’ve recently in our group migrated to TFS 2010 Beta and in order to support this, we’ve had to make some changes to our local machines to successfully connect to projects hosted on our TFS 2010 server.  In today’s blog, I wanted to do a quick demo of how to correctly connect to the TFS 2010 server from a Visual Studio 2008 client.

Update Visual Studio to Service Pack 1 (for client)

The first step is to update your Visual Studio in case you’ve not already done so.  This is required before you can successfully connect.  The steps are the following -

1. VS 2008 Service Pack 1 (for client)

2. Forward Compat Hotfix for 2008

Adding your Server Connection to a Visual Studio 2010 “Collection” (new concept)

As your already very familiar with, you are required to add a connection to a server before you can load a project.  This is done through the Tools | Connect to Team Foundation Server when you’ve installed Team Explorer on your client.  With Visual Studio 2010’s new collection (a “group” of like projects” then you change your method of connecting from the individual server, and provide the server and collection name. 

Thus, to confirm that you’ve correctly set this up, do the following:

1. Open Visual Studio 2008

2. Click Tools, Connect to Team Foundation Server

 

 

 

 

 

 

 

 

 

 

3.  Click Servers

 

 

 

 

 

 

 

 

 

 

4. Click Add

To correctly access TFS 2010, you are required to enter using the following syntax -

          http://<servername>:<port>/tfs/<collectionname>

5. Enter the URL to your TFS 2010 server into the Add Team Foundation Server textbox -

Summary

I only include this in a blog topic simply because there was confusion on our team as to how to effectively access our new server.  We are hosting our own TFS 2010 server currently until our hosted offering migrates fully to VS 2010.  It has been quite a journey.  Look for more TFS 2010 snippets that we learn over the coming weeks!

-Chris

Failure during Windows Server 2008 System State Backup & System Writer Missing

For some of those new to DPM 2007, you might find yourself in a situation where a Windows Server 2008 (and R2) is failing during backups consistently.  The consistency part is the ironic is that it is always failing and might lead you to review a bit more what is going on.  This problem, as outlined in this post, is often related to missing the Windows Backup feature on Windows Server 2008.

However, there are cases where some servers will fail unless you reset permissions for the System Writer.  This was outlined in this forum post yet no one actually batched this up and shared and since I had quite a few servers that I needed to run this on I batched this up and it corrected the issue.

To correct this problem, simply copy this batch file and execute on your Windows Server 2008 servers in question -

REM  Set ACLs correct for System Writer
REM  *************************************
Takeown /f %windir%\winsxs\filemaps /a
icacls %windir%\winsxs\filemaps  /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps  /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps  /grant "BUILTIN\Users:(RX)"
icacls %windir%\winsxs\filemaps  /grant "Administratoren:(RX)"
Takeown /f %windir%\winsxs\filemaps\* /a
icacls %windir%\winsxs\filemaps\*.*  /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps\*.*  /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps\*.*  /grant "BUILTIN\Users:(RX)"
icacls %windir%\winsxs\filemaps\*.*  /grant "Administrators:(RX)"
Takeown /f %windir%\winsxs\temp\PendingRenames /a
icacls %windir%\winsxs\temp\PendingRenames  /grant "Administrators:(RX)"
icacls %windir%\winsxs\temp\PendingRenames /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\temp\PendingRenames /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\temp\PendingRenames /grant "BUILTIN\Users:(RX)"
Takeown /f %windir%\winsxs\temp\PendingRenames\*.* /a
icacls %windir%\winsxs\temp\PendingRenames\*.*  /grant "Administrators:(RX)"
icacls %windir%\winsxs\temp\PendingRenames\*.* /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\temp\PendingRenames\*.* /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\temp\PendingRenames\*.* /grant "BUILTIN\Users:(RX)"

REM  Restart the Cryptographic Services
REM  **********************************
net stop cryptsvc &;& net start cryptsvc

After executing this, you will be set and ready to re-run your job and off you go.  This is the smallest post in history but it solved a big problem of mine on a couple of servers.

-Chris

Deploying Windows 7 in the Enterprise – A Blueprint for Success

It’s taken nearly a year and four developers, 2 QA, and 3 PMs to build the solution to deploy Windows 7 at Microsoft & other customers.  Code-named “Modena”, this solution offers customers who utilize System Center Configuration Manager 2007 to deploy Windows 7 easily, with little headaches, and hopefully rapidly.  My team was responsible for building the solution and we are putting the finishing touches on Modena and about to release RC3 to Microsoft’s Connect site for our customers to download and begin using.

In the meantime, I wanted to share an update about Modena as we recently released an article titled “Windows 7 and System Center Configuration Manager:  Your Windows 7 Deployment Guide” in the November 2009 edition of TechNet magazine.

What does Modena offer Configuration Manager customers?

A lot of questions are swirling around about ConfigMgr’s OSD feature, the Microsoft Deployment Toolkit (MDT) 2010, and now Modena.  We hope to update folks on the guidance as there are a lot of customers frustrated by not hearing from Microsoft a definitive answer around deploying Windows.

Nonetheless, the purpose of Modena from the outset was to expand on the functionality already offered in OSD as out-of-the-box features simply were not enough for the customers we were responsible for delivering the solution too.  We built a very customizable end-user experience that asks users a set of questions to prepare the system for the migration to Windows 7 and then they walk away and when they return their system has been migrated to Windows 7 with all their data intact.  For administrators, we also spent a lot of time ensuring that we prepared for the unhappy path scenario so that we protect user data and also provide help desk staff enough logs to determine the root cause of the issue.

Modena offers the following:

A end-user experience wizard easily configurable    
  Robust task sequence with built-in error handling that is snap-in and go  
  Key functionality needed to deliver wizard to end-users in their context (ServiceUI.exe)  
  Easily extensible Pre-Flight checks to verify systems are ready for Windows 7, prior to migration  
  Application Discovery Pre-Flight to discover applications already installed and set Modena to re-install them (ConfigMgr packages only)  
  Application selection capabilities that supports local configuration or dynamic Web service to simplify updating of application repository  
  A simple, configurable application called OSD Results that communicates success or failures to end-users at completion of migration

How do I learn more about Modena & Deploying Windows 7?

After you’ve reviewed our article above, then turn your attention to my team’s blog at http://blogs.technet.com/osd where we will continue to share details about our release as well as teach you how to utilize Modena.  We’ve been very, very busy the past few months and haven’t focused as much on sharing information since our last release but expect this to change over the coming months as we turn our attention to helping you successfully deploy Windows 7 at your company.  This is our goal – helping you adopt Windows 7 faster, more reliably, and with confidence. 

If you have any questions about Modena, please do not hesitate to drop me a line!

Thanks,

-Chris

Unable to Extend System Volume & DPM Backup Failure: Change the System State backup location

I recently had a couple of servers where they had rather small system partitions, one was Windows 2003 while the other was Windows 2008 RTM and DPM continued to fail with replica is inconsistent errors.  The DPM documentation and event errors were referencing the replica and volume partition size(s) and all checked out with plenty of space according to the current data size. 

Fix #1:  Expand the available space for the system partition

As a test, I thought about checking the actual system partition size on the protected servers and noticed small free space available on these virtual machines.

For testing purposes, I shut the virtual machine and expanded the system partition virtual hard disk.  To do this, I simply did the following:

1. Shut the virtual machine down in the VMM Administrator Console
2. Right-click and selected properties on the virtual machine
3. Highlighted IDE disk 1 and checked compact and expand disk and selected the expanded space size
4. Click OK
5. Restarted the virtual machine upon the job completion
6. Opened Server Manager (Server 2008 only), elected disk storage, and right-clicked and chose Re-scan disks
7. Right-clicked the system partition and selected to Extend volume

After the volume free space is sufficient (greater than 30+ gigabytes was my goal), then I re-run the consistency check for the replica partition for the server(s) that failed.

To re-check the consistency check, you do the following:

1. Open the DPM Administrator console
2. Click the Protection Tab
3. Locate the protection group, and expand if not already expanded
4. Locate the protected server and expand it to see the current volumes and system state backup
5. Right-click on the failed state item and select “Perform Consistency Check”

Fix #2:  Re-locate the location for the System State backup location on the protected server

The one problem that many might run into is the system partition isn’t expandable using the “extend volume” as it is either not supported (Windows Server 2003) or is a physical disk limitation.  In either case, the choice at this point is less related to creating the relevant space in order to successfully get system state backups to succeed and more pointed at moving the system state backup to another location. 

NOTE:  If you have limited space for your system partition and no other volume with significant space available to host the backup, you have bigger problems than the replica backup failing.  You should fix the limited free space first for this server and then focus on the following fix.

As I wasn’t familiar with how to successfully re-locate system state backup location, I used Bing search to see if others had figured this out already.  I came across this small blog post that was very helpful and for the most part was completely accurate thought I did have to make one minor tweak to finally get things happy (e.g. no failures or errors, just the big old Green OK check mark).

Follow these steps outlined in the post above -

1. Login to the server with credentials that is an Administrator
2. Navigate to the installation directory for the DPM agent, typically %programfiles%\dpm
3. Double-click datasources directory
4. Right-click on the PSDataSourceConfig.xml file and select open with and choose Notepad (or your preferred text editor)
5. Locate the <FilesToProtect> element
6. Change the current path from %systemdrive%\WindowsImageBackup\* (if Server 2008) or %SystemDrive%\DPM_SYSTEM_STATE\%computername%.bkf (if Server 2003) to the location you would like it to reside in (for example, H:\WindowsImageBackup)

As mentioned in the blog post above, DPM’s protection group will now go south on you and it will seem permament.  This is expected since the location of the system state configuration has changed.  The protection group needs to be modified through the wizard and as mentioned it should do so with no changes.  This, unfortunately, didn’t do the case every time and some protection groups were still failing.  This was easily determined using the DPM Events in the event viewer for the server with the following:

“DPM has detected changes in the file locations or volume configurations of protected objects…”

In order to correct this failure, I had to use the modify protection group wizard and remove the System State from the protected server and add it back.  After doing this, the replica consistency started working as planned.

Summary

In the short time I’ve started using DPM 2007, I’ve found it to be rather straight forward to troubleshoot failures with replica backups.  The issue is usually related to volume space not large enough, in my experience thus far.  With some quick cleanup, you can easily turn things around assuming you have the disk space (got to love SAN’s and DAS!) and the DPM console.

Hope this helps,

-Chris

Data Protection Manager 2007: Replica is inconsistent on Windows Server 2008

I recently took some time to learn more about Microsoft’s System Center Data Protection Manager 2007 since it was again a product in my division that I knew very little about.  Beyond that, I had quite a bit of incentive to get our engineering lab which is handled by me as a “side job” besides my Lead PM role for MPSD Engineering.  We recently had a Hyper-V Failover Cluster slight meltdown that was single-handedly caused by my mistake.  The good thing is that I’ve since recovered with less than a day’s worth of productivity lost by Dev, Test, and PMs.

As I continue to post more about DPM 2007, I thought I would kick-off what I noticed was the first glaring thing that showed up once I created my Protection Groups.  For each computer running Windows Server 2008 or above, I noticed that every single System State backup was failing.

Windows Server 2008 Protected Computers Requirements

The first thing I learned quickly was that right in the event viewer was a suggestion that sticks out like a sore thumb.  For any system state to get backed up on Windows Server 2008 (or R2), the Windows Backup Services feature must be installed.  If not, DPM will successfully backup everything other than System State.  This is not a feature that is often enabled on Windows Servers that I’m aware hence the reason it caught me off-guard.

My second “assumption” was that the DPM protection agent would have installed any pre-reqs and this was in fact not the case as it didn’t enable Windows Backup nor did it tell me to.  So off I go to enable this service so that in the case of a disaster recovery, I can restore the server(s) in question.

Batch File to enable Windows Server 2008 Backup Features

To simplify things, I made this quick batch file that is simplified deployment to all my servers that needed backed up.  This is so simple it is scary but I tend to do things simple as they reduce time and complexity.

REM  Install Windows Server Backup
REM  *****************************
ServerManagerCmd -install Backup-Features

This will install the service needed to effectively backup the System State on Windows Server 2008.

Summary

Data Protection Manager 2007 is a pretty slick piece of software.  The primary piece I like the most is the ability to do disk-based backups which is super efficient.  The learning curve seems to be low while the upside is great from a CYA perspective.  The above isn’t the only issue I’ve encountered and you should see some more posts this week outlining what those are until I have this ship running nice and smooth.

Adding IIS Host Headers in DNS using Netdom utliity

An age old question for IIS administrators is how to effectively manage small, medium, or even large Web servers with a minimal set of IP addresses.  The network team often manages layer 3 which includes the IP addresses that are available for Web servers.  If an unlimited supply of IP addresses existed, there would be no need for host headers.

IIS provides a mechanism that supports the creation of multiple Websites (often referred to as virtual servers) using a single IP address through the use of Host Headers.  Host headers are supported in HTTP version 1.1 and more is included in the following Microsoft Knowledge Base article http://technet.microsoft.com/en-us/library/cc753195(WS.10).aspx which isn’t the purpose of my post. 

Instead, the often difficult aspects of the use of host headers in the enterprise is for a Web admin (or any admin that is the non-AD\DNS administrator) to get the host header created.  I was recently in this dilemma and I was completely taken aback when I learned about functionality included in the netdom.exe utility.

Creating Alternate Identities for a Computer using Netdom

By default, any domain-joined Web server already has a single FQDN relative to the actual computer name.  This is created in the DNS server using a A record utilizing dynamic registration.  However, the problem begins when one needs the ability to have an alternate name for the server that doesn’t match the computer name hence creating an alternative DNS A record.

Little did I know that this is possible using a utility on the Server (Windows Server 2008/Windows Server 2008 R2)

Using Netdom /Add to Configure Alternate Identity (Host Headers)

To add the alternate name for the computer name, do the following:

NOTE:  In the following command, the IIS Web Server name is IIsWebServer name and the IIS Host Header desired is hr.contoso.com

1. Open an Elevated Command-Prompt (e.g. Administrator)
2. Type Netdom /? to see the syntax available or netdom /add /? to see more
3. To add the alternate name, type the following:

Netdom computername IIsWebServer /Add:hr.contoso.com

After using this command, reboot the server for the change to successfully complete.

Summary

Amazingly, this is extremely easy and completely successful.  If there are others out there that were aware of this then shame on me but I couldn’t believe I missed this.  Nonetheless, I hope someone else out there finds this as helpful as I did!

-Chris

Failover Cluster Snafu – Forcibly removing Failover Cluster Feature after Cluster Failure

My secondary job is to maintain our engineering lab (part of the mantra of “Do More with Less”) as we don’t have anyone dedicated to this role.  This lab is running completely virtualized minus our SQL infrastructure which is running on a 2-node Failover cluster since so much of our infrastructure relies on SQL.  In our case, we have a 7-node Failover Cluster that utilizes R2’s Clustered Shared Volumes (CSV) and I recently took a vacation.  Whoa, I bet you didn’t see that coming.  What does a 7-node cluster and your vacation have anything to do with each other.

I’m glad you ask … it is completely related to the fact that the 7-node cluster is conveniently triggered to “fail” during my vacation causing me to stop my vacation and take a look.  Recently, I had a node that simply went south during my vacation and I was super-high on the frustrated level as this lab isn’t my primary focus – though it seems to occupy me way too much lately!

What I thought I would share today is completely unsupported I’m certain but luckily you can take my gossip & rants on this blog as “Well, he doesn’t usually do things in the supported fashion anyhow…”

Dang it… I can’t remove the Failover Cluster Feature because it is still a part of a Cluster

What a “cluster” you might have on your hand.  No Pun Intended.  This is exactly the scenario I had.  I had a host go down that, unfornately, didn’t have access to the cluster any longer since it was evicted.  However, the node itself was seriously convinced that it was in fact still a vital part of the family.  I got high on the level of frustration and decided to start the digging process…

NOTE:  DIGGING IN THE REGISTRY FOR LITTLE JEWELS ISN’T RECOMMENDED NOR THE RIGHT IT APPROACH.  IT’S AN APPROACH FOR THOSE WHO ARE WILLING TO GAMBLE EVERYTHING AND CAN SAFELY CYA THEMSELVES IF THE GAMBLE DOESN’T PAY OFF.

<notFortheFaintofHeart> 

How to Force Failover Clustering Feature to be available to Remove

Now you know the warning.  Let me share how I just came across this way to force R2’s Server Manager feature wizard to again forget about Failover Cluster and allow me to move forward.  To do this, go to your broken node and open the Registry.

Backing up your registry right now is a great idea… do it and return.

1. Open Regedit
2. Locate HKLM\CurrentControlSet\Services\ClusDisk & ClusSvc
3. Delete these keys

You have now royally ticked off your R2 server though it is only for a brief moment.  Move to the next step…

Uninstalling Failover Cluster when cluster is unavailable

The next step is to open Server Manager and to remove the Feature for Failover Cluster.  When you do this, Server Manager will remind you that you shouldn’t move forward unless you know that all the services are moved off this cluster.  It is…so choose Yes and move on.

After the removal, it will likely ask for you to reboot which is a pleasant idea.  After rebooting, you can now safely add the feature back and now re-connect to the cluster and start the rebuild process.

</notFortheFaintofHeart> 

Simple.  Easy.  Not recommended…but if you are like me then time sometimes is worth the risk.  If you screw up, you can always rebuild your server. <grin>

Thanks,

-Chris

Differencing Disks & Merging: 80070005 Error … Just one persons Lesson Learned!

Recently, I had a rather simple demo for an audience that required me to have multiple systems running simultaneously and at different phases of a 2 hour migration to Windows 7.  The original plan of attack was to use a single base image, Windows XP SP2, and create differencing disks for each of the 3 virtual machines.  This is seemed to be the most reasonable approach.

In today’s post, I’m going to talk quickly about how I rather abruptly decided to get away from this model (single source, multiple differencing disks) and move to multiple sources using snapshots.  There are most experts that would argue that this is splitting hairs (or tomato vs. tomatoe) as snapshots utilizes the same principles of differencing disks.

When moving to this model, I had to make duplicate copies of the original source VHD (also known as the parent) and then use the Edit Virtual Disk wizard to “merge” the differencing disks (including some snapshots) until I ended up with a single VHD.

Attributes of a Differencing Disk

The one thing that many might miss about differencing disks is that the source disk is marked by the system as Read-only.  This is to avoid any “clobbering” of the file and causing all virtual machines based on the source from self-destructing.  This little tid-bit of information can save a lot of your time when you are attempting to merge to the parent and the system (e.g. Hyper-V) believes there to be additional virtual machines to still be using this source disk.  (This is my theory because I don’t know at what point Hyper-V removes the read-only attribute or if it ever does – I’m trying to follow up with folks internally to see if I can get an answer).  I digress…

Checking Source Disk Attributes

For those not familiar, to check to verify that the Read-only attribute is not enabled, do the the following:

1. Locate the source parent file (NOTE:  Not the snapshot\child file)
2. Right-click on the file, select Properties

If the Attribute for Read Only, un-check. 

It is important to note that when using Snapshots the parent file does not have the Read-only attribute.

Ouch:  General Access Denied Errors (0x8007005) during Merge

So here goes… you are attempting to merge your differencing disk to the parent when Hyper-V throws you a nice little error that says “General Access Denied” which leads most Admins to run to the file system directory and start reviewing the NTFS permissions.  If you were like me, you didn’t panic at all and then checked the permissions.  Wow, the permissions are fine and the panic sets in….

The first steps are to go through the process of merging…

1. In Hyper-V Administrator Console, right-click on the Server and choose Edit Disk
2. Locate the VHD\AVHD you wish to merge (e.g. d:\VM\MyTest.VHD)
3. It will then ask you whether you would like to compact or merge…
4. Select Merge
5. Click Next
6. Select to Merge with Parent
7. Click Finish

…and then it Happens!

Resolution

For protection, some folks set the source VHDs to read-only to avoid anyone using them incorrectly.  As mentioned, I haven’t found a solid repro whereby the read-only attribute is done by the system (aka – Hyper-V\Windows) so for now I’m going to assume humans were involved in the conspiracy.  Nonetheless, the general 8007005 error seems related to the inability of Hyper-V to merge the disks to a single file is the read-only for the source.  This problem doesn’t occur when the merge takes place to another filename whereby Hyper-V copies the two files to a new file.

Remove the read-only and you are in business…

Enjoy!

-Chris

[Info] – SCVMM R2 Documentation Live on the Web

It was recently announced internally at Microsoft that the VMM documentation team has made available via TechNet the documentation to go along with VMM 2008 R2.  For those of us who spent a great deal of time digging, playing, and mostly in “Fire first, Aim second” mode due to some minimal documentation for RTM this is a welcome site.

VMM R2 Product Docs:  Technical Library

*  Courtesy of Microsoft TechNet

Enjoy!

-Chris

Virtualization Tip: Migrating to SAN’s “after the fact” with SCVMM using R2’s Migrate Storage

In today’s post, I thought I would share some insight into how to effectively migrate to Storage Area Network’s (SAN) after you’ve already got an SCVMM environment up and running.  You found yourself with several Windows Server 2008 Hyper-V hosts and you were moving along with very little issues; though, you recently have noticed that downtime is unavoidable if you don’t have your backend storage running on SANs.

You will get overwhelmed when researching the issue and I just thought I would share one person’s perspective who had no SAN, lot’s of physical servers running Hyper-V, and decided to learn second, execute first.  Typical for my personality type…

Existing Environment

Prior to migrating to a SAN, each server had local drives in a RAID 5 configuration.  The volume was dedicated for Virtual Machines and Scratch directories.  Migrations between hosts utilized network transfer and was using BITS and at minimal required that the VMs were in a saved state.  The average transfer time was around 10 minutes.

Preparing for the SAN

The SAN was installed (EMC Clarion AX4-5 with two shelves) and utilizes Fiber to connect to the hosts.  The EMC Clarion is a 3U unit and is connected to all our Hyper-V servers along with our SCVMM server.

After installation, you have to utilize the NaviSphere Express software to configure your Disk Pools and volumes.  This was obviously done prior to the connection to the servers.

For our environment, we have the following configuration:

VM Source Volume – This volume has our read-only source VHDs and is utilized by using Differencing disks to never alter the actual source.  The volume is small with 1 TB usable space in a RAID 5 (optimized for read-only).

VM Storage Volume – This volume is in a RAID 10 configuration and has all the differencing disk(s) for our virtual machines.  Since using RAID 10, this is optimized for Read\Write.  This volume is ~4 TB in size and currently supports about 30 virtual machines with 53% utilization.

VM Library – This volume is large, SATA drives (1 TB each), in a RAID 10 configuration.  This is the VMM library resources share.

Decisions to make prior migration to the SAN

Prior to migrating to the SAN’s, you will need to make some decisions that are very, very important.  Those decisions are the following -

1. Do I utilizes Windows Server 2008 R2’s Clustered Shared Volumes (CSV) to take advantage of simplified management of LUNs?
2. If answer to number 1 is No, then you need to determine your LUN strategy to work around the 1 VM to LUN limit

The decision I made, since I could, was to run all our VMs as Highly Available and utilize CSV so that I can reduce my time in managing the physical disks and volumes in NaviSphere.  As you can see above, I created one single LUN (~4 TB) that would house all of our virtual machines.  In a later post, I will work through everything I did to get clustering up and running and how I enabled CSV’s.  For the purposes of this exercise, though, lets assume that I have a single LUN that is presented to multiple hosts running in a cluster.

Migrating to the SAN

The first thing I was excited to see was System Center Virtual Machine Manager R2’s behavior after each of the servers were a part of a cluster.  Rather than having to break down SCVMM by removing the hosts, VMM quickly realized that the hosts were now “clustered.”  (I lost for a brief moment connectivity and the hosts were in a warning state during this period.

The VMs, though, were not running at the time on the “shared storage” hence each physical host was in the cluster though not utilizing the resources of the cluster.  Prior to the release of SCVMM R2, I would have been required to rebuild my VMs and place the VHDs on the SAN.  SCVMM’s Migrate Storage feature (outlined later), though, was the magic that turned this process into a very simple migration.  Let me explain what I did…

Verifying that the shared storage drives (or mounts) are ready…

This is the first step as the process is required because the physical host where the VM lives already is required to have access to the storage.  To verify this, connect to the physical host that is currently running the VM -

1. Open Server Manager
2. Click Disk Management
3. Locate the volume for the shared storage, verify it is online and initialized

Migrating a Virtual Machine (One at a time)

Because of the sensitive nature, I started off by doing each migration one-by-one as I wasn’t sure of the “outcome” but as I became more comfortable I simply kicked off the process utilizing VMM’s PowerShell interface which made the migration move much quicker.  For now, I will step through the process using the VMM Administrator’s Console -

1. Open the SCVMM Admin Console
2. Locate the Virtual Machine you want to migrate to the SAN (currently running on the local physical host)
3. Right-click on the VM, select Migrate Storage
4. Utilizing the Migrate Storage Wizard, select the CSV volume (C:\ClusterStorage) on the physical host and complete the wizard

To validate the the Virtual Machines are actually utilizing the CSV storage, use the Failover Cluster Admin console and under Services and Applications you will see it listed.  The part I loved about this process is that VMM was intelligent enough to realize this was a clustered shared volume and during the migration *automagically* made the Virtual Machine(s) highly available.  This was verified in the SCVMM Admin Console by doing the following:

1. After migration completes, highlight the virtual machine in SCVMM Admin Console
2. Right-click the VM, select properties
3. Click Hardware configuration tab
4. Scroll to the bottom to the Availability section, validate it lists as “High”

That was it.  As I said, after a couple of these using the manual process it is rather easy to steal the PowerShell code and customize it and poof, your VMs are highly available.

Enjoy migrating!

-Chris

Real World Example of Troubleshooting R2 Live Migration using CSV’s

As I’ve mentioned several times, we run our engineering lab on much of the latest & greatest.  There are many who would argue that this is a luxury (one afforded to those who work at a Software company such as Microsoft) and I wouldn’t have a solid argument against their argument.  However, it doesn’t change the fact that often we are so cutting edge that when “issues” arise we don’t get a solid response internally without development debugging.  There is a challenge to often getting product group developers on tap to help and this leads to “fiddling” around with things which is where I found myself this past week.

We run our lab on a 7-node Hyper-V cluster that has each node attached to our EMC Clarion AX-4 SAN.  This cluster is in a node-majority setup and recently I found that R2’s new Live Migration functionality wasn’t working as designed.  After a little bit of investigation, I determined that only 2 of the 7 hosts were unable to migrate and each time a migration was attempted (for any VM), the following error was thrown in the event viewer -

'Machine name’ live migration did not succeed at the source.

Failed to get the network address for the destination node 'server a': A cluster network is not available for this operation. (0x000013AB)

Unlike many event error messages, the error “A cluster network is not available for this operation” is literally the value for the Win error 000013AB.

Quick Review of the Infrastructure

Each member of the cluster has two physical NICs, one for management and one that is dedicated to Hyper-V and represents multiple networks through VLAN Trunking.  The first thing that most folks, out-of-the-box, will see happen with clustering and networking is the lack of “exact match” when using Virtual Machine Manager (VMM) to manage your cluster.  Basically, the following two items must be identical in order for you to manage any machines in the cluster -

1. Network Name (on the physical NIC)
2. Network Tag

Depending on the number of Nodes in your cluster, this can be a pain in the you know what to determine.  It is very easy though utilizing the following PowerShell script and execute on your VMM server -

#####################################################################

function DisplayNicInfo($VMHostName)

{

  $yy= get-VirtualNetwork -VMHost $VMHostName; 

  $yy | ForEach-object {write-host "    Name      " $_.Name;

                        write-host "    Locations " $_.Locations;

                        write-host "    Tag       " $_.Tag;

                       }

}

#####################################################################

$clusname = read-host "Host Cluster name to check"

Write-Host ""

$VMMServer = get-vmmserver -computername localhost

$Cluster = get-vmhostcluster -name $clusname

$VMHosts = get-vmhost -vmhostcluster $Cluster

$VMHosts | ForEach-object {Write-Host "VMHost: " $_.Name;

  DisplayNicInfo($_.Name);

  Write-Host ""}

NOTE:  This PowerShell script is going to require you to lower your script in order to execute it.  To do this, do the following -

Set-ExecutionPolicy Unrestricted

To execute, you would then just open PowerShell, navigate to the PS1 file, and execute it.  Enter the Cluster Network Name (e.g. cluster.contoso.com), and it will display all nodes current Network Name & Tag configuration.  Fix whatever doesn’t match…

For more information, see the following blog I did about Network Name & Tag.

How do I do “Live Migration” when using Cluster Shared Volumes?

The first thing to note is that Live Migration is possible without the use of Clustering or Clustered Shared Volumes (CSV).  It is accomplished in Virtual Machine Manager (VMM 2008 R2) by right-clicking the virtual machine in the SCVMM Admin Console, select Migrate.  This you are clustering your Virtual Machines in order to get High-Availability then this changes thing.

To successfully migrate, you now move out of the SCVMM Admin Console and find your new best friend called the Failover Cluster Admin Console.  On your VMM server (if not one of your hosts), you can add this using Server Manager and adding it under the Remote Administration Tools.  You will see your highly available virtual machines listed in the console under Services and Applications, and you can right-click any VM and you will see the following option:

Resolving the Live Migration “Cluster Network is not Available”

Now that you are crystal clear on how to “migrate” between nodes let’s talk about how to troubleshoot a bit when you aren’t successfully able to migrate.  This was the case for me.  The first tool in your toolbox is the trusty Event Viewer which is where all events are targeted in case of failures for Hyper-V and High Reliability.  As shown above, you might see the following error screen in your event viewer -

As you can see, the source is Hyper-V-High-Availability and in this case you will see that the failure occurred for the “designation node” which is important as the error tells you whether it was destination or source.  I’ve blanked the Computer name but in this case I was on the source computer and located this error message thus indicating that it was “unable” locate the destination computer. 

This was troublesome after reviewing the cluster networks configuration as all “looked” well as indicated in this screen shot -

Furthermore, if you highlight the network and look at all the servers in the cluster you see the following -

Um, I learned that this screen might “lie” to you every once in awhile and things are not all happy and content in Cluster Live Migration land.  No fear, a little digging around in the network bindings clears life up a bit.

The following steps cleared up my headaches and by no means are they guaranteed to make your life happy.  I do, though, hope that it does save someone else a lot of time as this took a few hours to dig into and determine what was causing the problem.  At least, what I believe was causing the issue.

There are a lot of folks out there who are unclear on how to change the binding order when you have multiple NICs in your workstation or server.  In my case, I didn’t want the first binding to be the unconfigured VLAN NIC as this would certainly cause problems.  Thus, I went digging and ensured that all seven nodes in the cluster had the exact same binding orders and the first in the list was the domain and cluster management network card.

To do this, you do the following:

1. Click Start, locate Networks, and right-click and select Properties
2. In the Network and Sharing Center, click Change adapter settings
3. Hit the ALT key (yes, that’s it)
4. Click Advanced
5. Click Advanced Settings…
6. In the Adapters and Bindings Tab, ensure your Cluster Network "Connection” is the first listed (see screen shot)

After doing this on the node’s that are failing, the last step was to stop the cluster service on these nodes.  This will force a “Quick Migrate” but not a Live migration so there is a possible outage so do so in your outage window.  Keep in mind for those new to clustering (or those like me who haven’t used it since the late 90’s), the cluster service is stopped not in the Services but in the Failover Cluster Manager.

1. Under Nodes, right-click the Nodes
2. Select More Actions…
3. Click Stop Cluster Service

Give it a minute… If there a lot of VMs on the host, they are all going to Save state and then migrating to the new host.  After several minutes, you can go back in and select Start Cluster Service.  You should now (if they same problem as mine) be able to live Migrate back to the broken hosts.  Happy Clustering of VMs are here again!

Enjoy your Labor Day holiday weekend (if you celebrate!)

-Chris

Dynamic Provisioning for ConfigMgr hits the Shelf – TechNet Magazine Sept 2009

I promised early on to not use “marketing” as a ploy in my blog and trust me this is painful to write.  However, if it wasn’t for the fact that many of the blog posts here are directly related, I decided to go ahead with a little bit of selfish plugging.  In a story format, I recently wrote a TechNet Magazine (September 2009) article entitled Dynamically Provision Configuration Manager Roles using VMM and Operations Manager that focuses on much of the topics I outline here in my blog.  The primary difference is that it is a beginning to end approach and lot less scattered than my blog posts.  For those who have attended my presentation  (MMS, etc.) of the same title, this is the “word” format of that same presentation though you get to see all the pieces and visuals of it working.

The article focuses on the following:
   VMM:  Building your Nucleus
   VMM Server Installation
   VMM Agent Installations
   VMM Library:  The Building Blocks of Dynamic Provisioning
   Don’t Send in the Clones
   Working with Profiles and Templates
   WAIK Up
   From Static to Dynamic
   Test and Verify

This article will give you the baseline to understand the concept of dynamic provisioning of Configuration Manager 2007 client-based roles such as MP’s, DP’s, and SUP’s.  This greatly improves your scale unit and ensures that your infrastructure handles the client users in real time instead of running services all the time costing power and management cycles.

To summarize, let me share the “At a Glance” with you -

   Save time with Unattended Installs
   Understanding VMM Libraries
   Dynamic versus Static Provisioning
   Integrating VMM with Operations Manager

I want to give a shout out thanks to Hector Linares (PM on VMM team) as he was the technical reviewer for the article.  From my team, a thanks to Jason Githens, Catherine Campbell, and Angela Schmeil who reviewed the article and “cleaned it up” for me.

Enjoy!

-Chris