Kevin Holman's OpsMgr Blog : Cluster

Getting SQL Admin Studio to connect to your SQL Cluster

kevinhol — Tue, 07 Apr 2009 18:04:48 GMT

I had hell yesterday figuring out what was wrong with this.

I built a new SQL 2005 cluster, on Server 2003x64, to install my SCOM R2 release candidate environment into. I used iSCSI for my cluster, and my nodes are VM guests under Hyper-V, the same basic process that I documented here: Setting up a 2 node Server 2008 failover cluster under HyperV

So – when I was all done – my SQL 2005 Admin Studio would not connect to the remote SQL clustered named instance. This was odd – because this same admin studio WILL connect fine to any of my non-named instances… which are installed on stand-alone machines.

The machine I am running SQL Admin Studio on is a new Windows 2008 Terminal Server… with the Windows 2008 firewall. I know when connecting to a named instance, we first try and establish a connection over 1433, and if it is a named instance, we will talk to the browser service, and the browser service will tell us the dynamic port that SQL has assigned to that named instance. At this point, I could “cheat” and just hard code that port, or create a SQL alias in my SQL client… but that could lead to problems down the road. I would rather solve the problem. At first I thought something was blocking 1434, but basically, what I ended up needing to do – is to allow my “program” of SQL Admin Studio to be granted access, in the Windows 2008 firewall, on the terminal server running SQL admin studio.

The path is: %ProgramFiles% (x86)\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7\IDE\SqlWb.exe

Then allow the connection, any profile, and give it a name. After this – my Admin studio connected right up to the named instance.

The Cluster Service will automatically restart itself

kevinhol — Fri, 20 Mar 2009 17:02:47 GMT

Something I ran across with a customer.

There aren’t many situations where service recoveries run automatically in Microsoft MP’s, but this is one case where they do. The cluster service running is critical to a healthy cluster. In the current cluster MP, the service monitor for the cluster service will automatically start the cluster service on a node, if it detects it stops. There is a recovery action on the monitor to do just that.

As always – if you don't like this intended behavior – you can override just the recovery, and disable it.

Why do you need to know this?

Because – some service packs for clustered applications, require you to stop the cluster service, in order to apply. If you stop this service on a node while doing application maintenance, SCOM will restart it, almost immediately. The correct solution – is to use Maintenance Mode in SCOM, which will unload the monitors, and hence, any automatic recoveries will no long run. So…. make SURE you are effectively using maintenance mode if you ever need to stop your cluster service, or, disable this automatic recovery action.

Applying an OpsMgr hotfix to a RMS Cluster node? Some things to be aware of.

kevinhol — Wed, 25 Feb 2009 05:20:09 GMT

When you apply a SCOM hotfix to a RMS cluster, you need to be aware of some issues, and some workarounds. This is something I have seen several times in the field…

On any server/agent, the Hotfix installer will stop any discovered OpsMgr services, including the SDK, Config, and HealthService. This part is normal. It does this in order to update the files (DLL’s) that are part of the hotfix payload, and then it will start the services again when complete. This all works well, except for on RMS clusters.

The reason for this, is that the Hotfix installer is not 100% cluster aware.

In a RMS cluster… the passive node will have these three services stopped, and the services will be set to Manual Startup. On the active node – the OpsMgr services are also set to Manual Startup, but the services are running, because the Cluster service controls these services now. This is how a clustered service works, and we should not ever stop a clustered service in Service Control Manager, we really should take the resource offline, in Cluster Admin.

So I have two options… I can apply the hotfix to the Active Node… or the Passive node.

If I choose the active node – the hotfix installer will try and stop all the OpsMgr services, and this will cause the Cluster service to try and restart them, or eventually fail them over to the passive node – depending on your Cluster configuration settings. Therefore – it is probably best to patch the passive node first… ensure the hotfix applied correctly, and then move the cluster group and OpsMgr RMS group over to the freshly hotfixed node… and go patch the other one (now passive)

This works – but is not 100% smooth. When we apply the hotfix to the passive node, the hotfix installer will try and start the services at the end of the process, even though they were not running previously. We do NOT want these services trying to run on the passive node – since it does not own the cluster disk resources…. so the services will start, but cannot do anything but log errors.

You will also see an error from the HealthService – not being able to start. It is apparent that this service fails because it cannot access the disk resource, but the SDK and config services WILL start.

What is worse – is that the hotfix installer – changes the config of the service startup types to Automatic – which means these services will continue to try and run on the passive node across reboots.

So – the guidance I have, for RMS clusters – is:

Patch the passive node (we will call this Node 2)
Click ok on the HealthService start failure error.
Ensure the hotfix applied by inspecting the DLL(s) versions as documented in the KB.
Stop the running SDK and Config services on the passive node.
Set any OpsMgr services that were changed to Automatic – BACK to Manual.
Move the cluster resource groups over to the freshly patched Node 2.
On Node 1 (now passive) apply the hotfix, and repeat steps starting at Step 2 above.

NOTE: This is only applicable to OpsMgr specific hotfixes. For OS hotfixes – you would follow your standard clustered OS hotfix routine.

Setting up a 2 node Server 2008 failover cluster under HyperV

kevinhol — Mon, 20 Oct 2008 05:11:26 GMT

For lab testing, we would often set up Microsoft Server 2003 2-node clusters, under virtual server 2005, in order to test OpsMgr configurations, such as clustered RMS, clustered databases, or just testing management packs on clustered applications like SQL and Exchange. Virtual Server 2005 did this quite easily, using the built in SCSI adapter which had cluster support. Here is a good write-up on setting this whole thing up:

http://www.microsoft.com/technet/prodtechnol/virtualserver/deploy/cvs2005.mspx

Enter Server 2008 and HyperV. HyperV does not include SCSI adapter support for clustering. That is antiquated technology anyway, since most companies utilize SAN or iSCSI for shared storage clustering - not direct attached SCSI anymore. This means you won't be able to migrate your virtual clusters built under Virtual Server 2005 to a HyperV host, nor will you be able to use the same process to build a new cluster under HyperV.

I am going to use this post to document building a very simple 2-node Server 2008 (virtualized guest) cluster, using a Server 2008 HyperV host server.

There is a really good blog discussing all the options out there for combining fail-over clustering with HyperV - located at:

http://blogs.technet.com/josebda/archive/2008/06/17/windows-server-2008-hyper-v-failover-clustering-options.aspx

And a great PFE blog essentially doing the same things I am documenting in this post:

http://blogs.technet.com/pfe-ireland/archive/2008/05/16/how-to-create-a-windows-server-2008-cluster-within-hyper-v-using-simulated-iscsi-storage.aspx

And of course the Microsoft step by step for building the cluster on Server 2008:

http://technet.microsoft.com/en-us/library/cc731844.aspx

We will be using something similar to Options 4 and 5 in the above blog post... Only one physical HyperV host, and that same host will serve as the iSCSI target. My post assumes there is already a domain present.

The BIG challenge out there - is picking an iSCSI target to use, for labs, testing, and demos. You have a lot of choices out there... Microsoft Windows Storage Server, and then other third party products.... but in the "free" space... your choices are very limited. There are some free ones out there, but few are supported by Server 2008 fail-over clustering. Best thing is to use an existing in-house iSCSI target - if you have one. If not - then people have reported success using the latest "trial" version of iSCSI target from Starwind http://www.rocketdivision.com/wind.html. I will be using an internal-only version of the Microsoft iSCSI target... cause it's free to me, and doesn't expire. :-)

Here is a high level overview of the process:

1. Setup your Server 2008 HyperV host.

2. Setup two Server 2008 guest OS's.

3. Setup a Microsoft iSCSI target on the HyperV host.

4. Connect the guest OS's to the iSCSI target

5. Setup Microsoft Failover clustering

6. Test the cluster

7. Install the cluster aware application (SCOM, Exchange, SQL, etc...)

Ok - lets get started!

I am going to make some assumptions here - to save time and not detail every part of setting up HyperV on the host, or the guests:

I have a single physical HyperV host, with a name of VS3 (Server 2008 x64). I have a single physical Domain Controller, DC1 (Server 2003 x86). My domain is OPSMGR.NET in this case. I create two guest machines - both Server 2008 x64 Enterprise. Their names will be OMCL1 and OMCL2. These will be my cluster nodes, and they are joined to OPSMGR.NET.

We need to some some quick planning for the cluster... we will need a few things.

Names:

OMCL1 - node 1 name

OMCL2 - node 2 name

OMCLV1 - virtual cluster quorum network name

OMCLSQLV1 ands V2 - virtual cluster server network name(s) for the clustered application instance(s)

VS3 - HyperV host and iSCSI target

IP address:

OMCL1

primary NIC - 10.10.10.50/16

cluster private NIC - 192.168.1.1/24

iSCSI NIC - 192.168.2.10/24

OMCL2

primary NIC - 10.10.10.51

cluster private NIC - 192.168.1.2/24

iSCSI NIC - 192.168.2.11/24

OMCLV1 - 10.10.10.52/16

OMCLSQLV1 - 10.10.10.53/16

OMCLSQLV2 - 10.10.10.53/16

VS3

primary NIC - 10.10.10.12/16

iSCSI NIC - 192.168.2.1/24

Cluster Disks:

Q: quorum

Virtual Networks:

External - External network connected to physical NIC of hyperV host - I use this for all VM and host communications.

Private - Private VM network for cluster heartbeat communication only.

iSCSI - Internal Only network that allows communications for HyperV guest OS and host only.

At this point, I have completed steps 1 and 2.... I installed hyper V, set up the networks required, added virtual network adapters for each network, made sure I can ping all devices (I had to adjust some of the Windows firewall settings on the host to get this working) but essentially - each guest should be able to ping all the other guest and host network interfaces - external, private, and iSCSI.

Then - I installed the iSCSI target - and created a 2GB shared virtual disk file on the host. I set this to allow the two Cluster node initiators by IP address.

Next, on the cluster nodes, I used the MS iSCSI initiator - connected to the host portal, and then connected to the iSCSI disks. Each node connected to the same disk, and I set the drive letters as "Q" on each. One node initialized and formatted the disk.

So I find myself at step 5. Following http://technet.microsoft.com/en-us/library/cc731844.aspx I installed the failover clustering feature on each node, then ran the cluster validation test. Everything passed except warnings from Active directory. This is because I am running my installs using a domain user account with local admin priv on the cluster nodes, but not as a domain admin. I do this on purpose... too many times Microsoft people test everything using a Domain admin account, and dont experience the same pain that our customers do in the field.... this warning was simply stating that the user running setup for the cluster, does not have permissions to create the computer accounts in the domain for the virtual cluster name. Therefore, I will create this manually using my domain admin account, and assign full control permission on the computer account object to my user account doing the cluster installs. Lastly - I need to set the computer account to "disabled" in ADUC.

Ok - now that is done - we will "Create a Cluster". Give the wizard the cluster virtual name, the the IP address we assigned as above, and click Next.

HOLY CRAP. Creating clusters in Server 2008 just got a LOT easier than doing this in Windows Server 2003!!!!

The next step is to manage the cluster, go to Networks, and set the iSCSI network properties to "Do not allow the cluster to use this network". You can also rename the 3 networks to more friendly names at this point.

At this point - you have a functioning cluster.

You should test this by making sure you can ping the virtual cluster resource name, and fail the cluster disk over to each node (stop cluster service or reboot each node)

Now - I am going to add my disks and install SQL 2005. For my SQL cluster - I have created 4 virtual disks on my iSCSI target host. 2 disks are 20GB and 2 disks are 10GB. Each 20GB disk will be for databases, each 10GB disk will be for transaction logs. Then I will install two instances of SQL 2005, and make this a multi-instance (Active/Active) cluster. First - I will connect to the disks with the iSCSI initiator, then bring them online in Disk management, then initialize them, then format them. Then - using the Cluster Management tool, add them to the cluster, and assign them the appropriate drive letters in the cluster admin tool.

From my reading - this is a lot tougher to get going - we will see.

Here is a list of known issues to be familiar with:

http://support.microsoft.com/kb/936302/en-us

http://support.microsoft.com/kb/948208/en-us

To start, I want to list some good resources on understanding clustering, and then adding SQL 2005 to a cluster.

First - a guide on installing a cluster to prepare for SQL clustering (this is Server 2003 based, however, but good tips and make sure you understand this)

http://www.sql-server-performance.com/articles/clustering/cluster_server_2003_p1.aspx

Next, a step-by-step on installing SQL 2005 to a Server 2003 Cluster:

http://www.sql-server-performance.com/articles/clustering/cluster_sql_server_2005_p1.aspx

Background data on SQL clustering:

http://www.sql-server-performance.com/articles/clustering/Cluster_Setup_and_Troubleshooting_Techniques_PartI_p1.aspx

http://www.sql-server-performance.com/articles/clustering/clustering_best_practices_p1.aspx

Here is a GREAT step by step video, of installing SQL 2005 on a 2008 failover cluster, done by a fellow PFE:

http://support.microsoft.com/default.aspx/kb/953170/en-us

OK - because I don't want to deal with the issues defined in http://support.microsoft.com/kb/932897/ I need to create my Cluster Application group manually, and add my disks there. I will create a SQL group for each instance.... naming them the same as my network names will be, to keep it simple. So I create two "Empty Service or Application" groups, naming them OMCLSQLV1 and OMCLSQLV2

Now my cluster config looks like this:

And now my storage looks like this, with a DB and LOG volume for each :

So I make sure all resources are running on OMCL1, and then pop in the SQL 2005 Enterprise x64 CD into my VM. When setup runs the pre-req check - it runs against OMCL1 and OMCL2 - so that is a good start! On the components screen - I am able to check the box "Create a SQL Server failover cluster" so that means that SQL setup detected my cluster. Great news!

I choose a named instance of SQL (inst1) and then give it my virtual server name I want to use (OMCLSQLV1). Basically just fill in each page of the setup wizard from there.... entering the IP address, choosing the correct cluster disk for your data files, picking your domain groups for cluster services, etc....

Oddly enough - it all installed really well! I did run into the issue noted in Problem 5, at http://support.microsoft.com/kb/936302/en-us which basically tells me I need to get SQL 2005 SP2 installed ASAP. I now have a fully functioning SQL 2005 failover cluster on Server 2008 x64. I then will install another SQL instance for the A/A configuration I want, and will apply SQL SP2 to both instances. Then, take a look at the OpsMgr console... and see what it discovers!

Update - Interestingly enough.... before I could event complete the installs... I got an alert about service pack compliance from OMSQLV1\INST1 which means I have already discovered SQL and started monitoring it! Even though we dont have a Server 2008 Failover clustering MP yet... it does appear the current SQL MP detects and and discovers SQL 2005 on a Server 2008 failover cluster.

Update - I could not connect to my SQL servers from the network. The SQL 2005 install is not SQL 2008 Firewall aware - therefore it does not configure the firewall to allow for SQL server access. You can read up on this subject at the following links:

http://blogs.msdn.com/sqlsecurity/archive/2008/07/01/sql-server-and-the-windows-server-2008-firewall.aspx

http://msdn.microsoft.com/en-us/library/cc646023.aspx#BKMK_dynamic_ports

Essentially - I had to create and enable three rules on each node for SQL server - A program based rule for sqlserver.exe for each instance of SQL, and then a rule opening 1434_UDP for the SQL browser service. Then, I could connect to the SQL server using simply OMCLSQLV1\INST1. If I did not open the ports for the browser service - then my application would need to know the port that the named instance of SQL was using, and connect directly with the port in the connection string. Some choose to lock down a named instance to a static port, then set up connection strings to include the fixed port... but I did not choose that route.

Update - one last note.... one of the things I typically do in SQL security, to mirror my customer environments, is to add a global group with my admin account having SA access to SQL, and then I remove BUILTIN\Administrators, and NT AUTHORITY\SYSTEM (local system account). When you did this on a SQL 2005 clustered instance under Windows 2003, you needed to add the cluster domain service account to have access to SQL, or the resource would fail to come online. Well, not in Server 2008 failover clustering - we dont use a domain service account for the cluster, it apparently uses local system, as the cluster service runs under local system. What I found was - if I follow my standard, and remove local system, the cluster resource for SQL fails. I had to go and start sql from an elevated command line on the node: "sqlservr.exe -c -s INST1", connect in SQL mgmt studio, and add the NT AUTHORITY\SYSTEM back. It did not need SA, but did need public. Once this was done the SQL resource would come online and all was well again.

DBcreatewizard or just run good old SetupOM.exe - which should I use to install the Database component of OpsMgr?

kevinhol — Sat, 03 May 2008 14:04:50 GMT

There has always been a bit of confusion on when to run the DBCreateWizard.exe tool, or when to just use SetupOM.exe to create the Operational DB or Data Warehouse DB.

Historically.... in MOM 2005, we used the DBcreate Wizard in order to create the Onepoint database on Active/Active clusters..... or when SQL DBA teams refused to run a MSI based setup on one of their SQL servers. The DB create wizard was a better option for them.... since it did not have to install any binaries on a SQL server. In practice.... it was pretty rare to see this in widespread use.

In OpsMgr 2007, we haven't really documented all the scenarios for when you should run the DBcreate Wizard.... and I will try and do that here.

The DB create wizard is located on the CD - In the \SupportTools folder. It does require some additional files to run it - these don't have to be "installed", just need to be copied over to the SQL DB server where you will run the wizard. Follow: http://support.microsoft.com/kb/938997/en-us

*** Note - the additional files required to run DBCreateWizard.exe are documented in the KB article above. They were also provided on the SP1 Select CD. However - the files provided on CD are for 32bit x86 only. If you are using the DBCreateWizard on a x64 platform - you MUST copy these files listed in the KB article from an x64 server.... any x64 server with the console installed will have them.

Note - there were some significant issues with the RTM version of this tool... in detecting the correct SQL instance on a multi-instance cluster, and leaving some table information blank (http://support.microsoft.com/kb/942865/en-us). When deploying SP1 - Use the SP1 version of this tool. If you MUST deploy the RTM version - I would recommend using SetupOM.exe for all installs.

Ok.... first, you will notice in the OpsMgr Deployment guide, they instruct to use the DBcreateWizard when installing the database on an Active/Passive cluster. That's pretty much our first introduction to this tool. While this isn't required (you can simply run SetupOM.exe on the Active node) it is recommended to use DBCreateWizard. Essentially, our recommendation is that anytime you have a dedicated SQL server for the OpsDB role... with no other OpsMgr role present, then you should use the DBcreateWizard to create the Operational database. The reason for this, from an internal discussion I have been involved in.... is because using SetupOM.exe will create some additional registry entries on the database server... and will change how updates are applied to the server from an OpsMgr perspective. Another scenario to leverage this tool, is anytime your SQL DBA teams refuse to allow you to run a MSI based setup on their SQL servers/clusters.

Below, I will just walk through some of the scenarios where using this stand-alone tool really makes good sense.

Scenarios:

1. All in one role/shared roles. This is where a single server hosts SQL Server 2005 and the Operational Database role, along with the RMS role. In this case.... you might as well just run SetupOM.exe and create the database while installing the management group. You potentially could run the DBcreatewizard first.... but this would be an additional step and provides no value.

2. Split roles: Dedicated SQL server (Server A) and dedicated RMS (Server B). In this scenario - we recommend using DBcreatewizard.exe instead of just running SetupOM.exe on the SQL server. However - you certainly can do either one.... both are fully supported.

3. Split roles - clustered DB: Dedicated cluster for SQL (can be A/P or A/A or multi-instance or multi node.... doesn't matter) In this scenario - we recommend using DBcreatewizard.exe instead of just running SetupOM.exe on the SQL server. That said.... you can run SetupOM.exe on any node that owns the SQL instance you are creating the DB in.... we just favor using DBcreateWizard.

4. Draconian DBA's. In general.... DBA's are used to creating an empty database for an application, then granting permissions to the DB only.... then washing their hands of it. They don't like running setup's... or even running tools on their SQL servers.... If they must have an application create a database as part of that application install - they MUCH prefer that all the DB creation be handled remotely. Unfortunately.... MOM 2005 and OpsMgr 2007 do not support what DBA's would most like to see. We must run our setup or tool on the database server/node in order to install that component. I suppose we could install the OpsDB using the DBcreatewizard in a test lab SQL box, then detach it.... then hand the files to a SQL team and have them drop in into a production environment to make them happier.... but I haven't really done much testing there. Anyway.... the DBcreateWizard is the best option when working with a rigid DBA team. Just follow the KB article listed above... and have the SQL team run the tool to create the database.... then they can delete to tool from the server. We will still require SA priv over the instance to complete the RMS setup.... but once that is done, they can remove these advanced rights, per my previous post: http://blogs.technet.com/kevinholman/archive/2008/04/15/opsmgr-security-account-rights-mapping-what-accounts-need-what-privileges.aspx

5. Multiple Operational Databases in the same SQL instance. It is possible, if you have multiple management groups, that you could place all the Operational DB's into a single SQL instance. Now - these had better be small environments (test/dev) or a beefy SQL server to handle all that I/O.... but just for grins.... lets say you are doing it. If you tried to run SetupOM.exe and install the database component multiple times.... it would detect it was already installed and ask you if you wish to repair or remove OpsMgr. No good. In comes the DBcreateWizard. This tool is the supported method for creating multiple OpsDB's in a single SQL instance.