Where, Oh Where Did All of the Microsoft.com SQL Clusters Go?

re: Where, Oh Where Did All of the Microsoft.com SQL Clusters Go?

Tim Toennies — Mon, 13 Feb 2006 21:54:49 GMT

Wow... this is a really great article. I gotta give you props for being so candid about how you use (and in this case don't use) features of MS' products within MS IT. But, I also have to kinda dig you. This is a blog for microsoft.com, so I don't know the extent you speak for all of MS' operations division but I really hope that someone is running a GeoCluster on SQL 2005 to support a production system at MS!!! If so can you please have them add a comment to your blog & also a link on how they've been successful? If this is not the case you guys are missing a tremendous dog-fooding opportunity.

Thanks.

A great article on real life SQL cluster scenarios - at Microsoft of all places

Aaron Tiensivu's Blog — Tue, 14 Feb 2006 20:18:07 GMT

A great example of how not every tool fits every scenario. It gives a lot of insight into the 'why' and 'why nots' of various technologies.

http://blogs.technet.com/mscom/archive/2006/02/13/419406.aspx

Their blog is usually a good read anyway. If

Microsoft.com SQL Server architecture

SimonS' SQL Server Stuff — Tue, 14 Feb 2006 23:13:21 GMT

I was sent a link recently on the Microsoft.com SQL Aerver Architecture, this is really interesting and...

Tommy’s Blog » It’s hard to have scalable, reliable databases on the Web

Tommy’s Blog » It’s hard to have scalable, reliable databases on the Web — Wed, 22 Feb 2006 09:11:43 GMT

PingBack from http://twwilliams.com/blog/2006/02/21/its-hard-to-have-scalable-reliable-databases-on-the-web/

Microsoft High Availability Secrets Revealed

Toadkillerdog's DogHouse — Thu, 02 Mar 2006 19:12:35 GMT

A really candid and excellent article from the "Group Manager for database engineers" at Microsoft on...

re: Where, Oh Where Did All of the Microsoft.com SQL Clusters Go?

Colin Leversuch-Roberts — Thu, 02 Mar 2006 23:48:41 GMT

I always did think that clustering ( for SQL Server ) sometimes was a double edged sword. I've always strongly supported log shipping, right from 6.0 ( when you had to roll your own ). My experiance with clustering is that it often creates mpore problems than its worth and you have a shed load of expensive hardware sitting around twiddling its thumbs!! Roll on load balanced SQL Server.

re: Where, Oh Where Did All of the Microsoft.com SQL Clusters Go?

Bobby — Fri, 03 Mar 2006 21:31:04 GMT

This article raises several questions -
(1) Does NLB here refer to the silly built-in Win2k3 Enterprise software NLB solution, or, does it refer to a real hardware LB solution (i.e. Foundry, F5 etc.) - and does it matter (potentially not?) Which one is MS using internally (hw or sw)?
(2) if you use 1-way transactional log shipping behind NLB virtual IP in the case of a user-updated database (like a blog system) it would seem you must be ok with 10-20 minute downtimes if there is unexpected error. in our MSCS systems we see about a 15-30 second failover time, so the acceptable downtime for updateable systems within MS seems surprisingly high!
(3) Also - in the case of user-updated databases I presume you are in fact not using NLB (since only one db has the latest info) - or - are you using NLB with one node marked as "off" and then you switch the nodes out after recovery of the secondary?

Hopefully you guys will have a moment to reply :)

Do you need to use SQL Clustertering for High availability

Chris Burrows .NET Blog — Sat, 04 Mar 2006 07:07:20 GMT

I was flicked this blog article from Kevin Jones at work.&nbsp; I must admit it was nice to read the...

TechTalkBlogs (TTB) Post Highlights

Tech Talk Blog — Sat, 04 Mar 2006 10:14:09 GMT

As the word about TTB gets around we've seem more blogs added, and&nbsp; a lot of posts about TTB itself....

re: Where, Oh Where Did All of the Microsoft.com SQL Clusters Go?

MSCOM — Sat, 04 Mar 2006 21:31:05 GMT

I’m to see there have been a few questions to my post, and I’ll do my best to check in and provide clarity or answers.

First, I’d like to address Tim’s question regarding geo clustering. Micorosoft.com Operations is definitely a much smaller group than the operations teams that support our internal systems (MSIT), and MSN Operations. We’re not running clusters, but I know that both MSIT and MSN are. Are they running geo clusters? Great question, and I don’t know the answer to that. I have a request out to other internal Microsoft operations teams to see if I can find someone who is willing to comment here on their experiences.

Now to Bobby’s questions:

1.) Am I talking about NLB that is part of Windows Server? Yep, that’s exactly what I’m talking about and it works great too! To be clear, I’m talking specifically about our backend SQL Servers. This is my question to those who ask why we’re not considering a hardware solution. Why spend the money on hardware when a software solution is already there just waiting to be turned on? Network Load Balancing is included in all Windows Server versions in 2000 and 2003, and it’s very easy for someone who has never used it before to get up to speed quickly. Sure it’s not as sexy as a hardware solution, and no, it does not do some of the cool things you can with hardware, but it does meet our needs and works flawlessly.
2.) Great question about how we use read-only clusters with a single publishing SQL Server. Yes, we can not fail over as quickly as MSCS. No question about it. Latency, or stale data, as you point out, is a huge concern for us in these systems. Here’s how we have tackled this. First, we have to monitor the publishing server closely. If the server, the database or replication fail, and we don’t know about it….lets just say that I would have some tough questions to answer back to our business owners. We typically have planned work that will cause the data to stop flowing, and in this case we have tried to fix the problem with processes. The publishing server will have a log shipping secondary, and NLB will be configured between the two with only the publishing server converged in the cluster. We’ll stop the application from accepting writes, let log shipping sync, and make sure replication has completed to all subscribers. We speed this up, by running these jobs more frequently before we start the work. Once we’re certain we haven’t missed any data we recover the secondary, create the replication publication with the no sync option (all the data is the same at the publisher and subscribers, so there’s no reason to synchronize), then turn NLB off on the old publisher, and ON on the new publisher. Turn the app back on, and setup log shipping the other direction and we’re done. We can do this in about 5 minutes. Most people ask if we have a script to do it, and the unfortunate answer is no. Moving forward we will be replacing most of our log shipping with database mirroring, and this will reduce failover time substantially.
3.) And your last question about using NLB with user updated databases. As I mentioned above, we are indeed using NLB and log shipping. One of the biggest benefits is that there is a single IP or name that our web servers have to point to. This way we don’t have to update connection strings while we do maintenance.

Hopefully this clears things up a little, but if not let me know what you think and I’ll do my best to explain further.

re: Where, Oh Where Did All of the Microsoft.com SQL Clusters Go?

MSCOM — Sat, 04 Mar 2006 23:11:41 GMT

Just to clarify one point in the first sentence of my last post. I'm GLAD to see question and comments. -Brad

Microsoft.com SQL Server architecture

SimonS' SQL Server Stuff — Tue, 23 May 2006 12:10:07 GMT

I was sent a link recently on the Microsoft.com SQL Aerver Architecture, this is really interesting and...

Add a little Development, mix in some Security, a dash of Program Management, apply a liberal amount of IIS and SQL…Part of the Recipe for MSCOM Operations’ Blog

Microsoft.com Operations — Thu, 28 Sep 2006 00:30:11 GMT

Readers of this blog might have noticed (or been puzzled by) the variety of subject matter that we present....

Microsoft.com SQL Server architecture

SimonS SQL Server Stuff — Sun, 12 Aug 2007 02:55:32 GMT

I was sent a link recently on the Microsoft.com SQL Aerver Architecture , this is really interesting

microsoft » Add a little Development, mix in some Security, a dash of Program Management, apply a liberal amount of IIS and SQL???Part of the Recipe for MSCOM Operations??? Blog

Wed, 03 Oct 2007 16:25:03 GMT

PingBack from http://microsoft.whitewitty.info/?p=2243

microsoft » TechTalkBlogs (TTB) Post Highlights

microsoft » TechTalkBlogs (TTB) Post Highlights — Wed, 03 Oct 2007 16:25:09 GMT

PingBack from http://microsoft.whitewitty.info/?p=2242

microsoft » Do you need to use SQL Clustertering for High availability

microsoft » Do you need to use SQL Clustertering for High availability — Wed, 03 Oct 2007 16:25:11 GMT

PingBack from http://microsoft.whitewitty.info/?p=2241

microsoft » TechTalkBlogs (TTB) Post Highlights

microsoft » TechTalkBlogs (TTB) Post Highlights — Wed, 03 Oct 2007 16:25:15 GMT

PingBack from http://microsoft.whitewitty.info/?p=2242

microsoft » Do you need to use SQL Clustertering for High availability

microsoft » Do you need to use SQL Clustertering for High availability — Wed, 03 Oct 2007 16:25:18 GMT

PingBack from http://microsoft.whitewitty.info/?p=2241

microsoft » Microsoft High Availability Secrets Revealed

microsoft » Microsoft High Availability Secrets Revealed — Wed, 03 Oct 2007 16:25:21 GMT

PingBack from http://microsoft.whitewitty.info/?p=2240

microsoft » Tommy’s Blog » It’s hard to have scalable, reliable databases on the Web

Wed, 03 Oct 2007 16:25:23 GMT

PingBack from http://microsoft.whitewitty.info/?p=2239

microsoft » Microsoft High Availability Secrets Revealed

microsoft » Microsoft High Availability Secrets Revealed — Wed, 03 Oct 2007 16:25:31 GMT

PingBack from http://microsoft.whitewitty.info/?p=2240

microsoft » Tommy’s Blog » It’s hard to have scalable, reliable databases on the Web

Wed, 03 Oct 2007 16:25:39 GMT

PingBack from http://microsoft.whitewitty.info/?p=2239

microsoft » Microsoft.com SQL Server architecture

microsoft » Microsoft.com SQL Server architecture — Wed, 03 Oct 2007 16:25:39 GMT

PingBack from http://microsoft.whitewitty.info/?p=2238

microsoft » A great article on real life SQL cluster scenarios - at Microsoft of all places

Wed, 03 Oct 2007 16:25:44 GMT

PingBack from http://microsoft.whitewitty.info/?p=2237

microsoft » Microsoft.com SQL Server architecture

microsoft » Microsoft.com SQL Server architecture — Wed, 03 Oct 2007 16:25:48 GMT

PingBack from http://microsoft.whitewitty.info/?p=2238

microsoft » re: Where, Oh Where Did All of the Microsoft.com SQL Clusters Go?

microsoft » re: Where, Oh Where Did All of the Microsoft.com SQL Clusters Go? — Wed, 03 Oct 2007 16:25:49 GMT

PingBack from http://microsoft.whitewitty.info/?p=2236

microsoft » A great article on real life SQL cluster scenarios - at Microsoft of all places

Wed, 03 Oct 2007 16:25:52 GMT

PingBack from http://microsoft.whitewitty.info/?p=2237

microsoft » re: Where, Oh Where Did All of the Microsoft.com SQL Clusters Go?

microsoft » re: Where, Oh Where Did All of the Microsoft.com SQL Clusters Go? — Wed, 03 Oct 2007 16:25:54 GMT

PingBack from http://microsoft.whitewitty.info/?p=2236

microsoft » A great article on real life SQL cluster scenarios - at Microsoft of all places

Wed, 03 Oct 2007 16:25:56 GMT

PingBack from http://microsoft.whitewitty.info/?p=2237

microsoft » microsoft » Add a little Development, mix in some Security, a dash of Program Management, apply a liberal amount of IIS and SQL???Part of the Recipe for MSCOM Operations??? Blog

Wed, 03 Oct 2007 16:27:05 GMT

PingBack from http://microsoft.whitewitty.info/?p=2600

microsoft » microsoft » Add a little Development, mix in some Security, a dash of Program Management, apply a liberal amount of IIS and SQL???Part of the Recipe for MSCOM Operations??? Blog

Wed, 03 Oct 2007 16:27:37 GMT

PingBack from http://microsoft.whitewitty.info/?p=2600

microsoft » microsoft » microsoft » Add a little Development, mix in some Security, a dash of Program Management, apply a liberal amount of IIS and SQL???Part of the Recipe for MSCOM Operations??? Blog

Mon, 08 Oct 2007 21:42:43 GMT

PingBack from http://microsoft.whitewitty.info/?p=4411

microsoft » microsoft » re: Where, Oh Where Did All of the Microsoft.com SQL Clusters Go?

Mon, 08 Oct 2007 21:43:17 GMT

PingBack from http://microsoft.whitewitty.info/?p=4410

microsoft » microsoft » A great article on real life SQL cluster scenarios - at Microsoft of all places

Mon, 08 Oct 2007 21:43:20 GMT

PingBack from http://microsoft.whitewitty.info/?p=4409

microsoft » microsoft » Microsoft.com SQL Server architecture

microsoft » microsoft » Microsoft.com SQL Server architecture — Mon, 08 Oct 2007 21:43:29 GMT

PingBack from http://microsoft.whitewitty.info/?p=4408

microsoft » microsoft » Tommy’s Blog » It’s hard to have scalable, reliable databases on the Web

Mon, 08 Oct 2007 21:43:31 GMT

PingBack from http://microsoft.whitewitty.info/?p=4407

microsoft » microsoft » Tommy’s Blog » It’s hard to have scalable, reliable databases on the Web

Mon, 08 Oct 2007 21:43:40 GMT

PingBack from http://microsoft.whitewitty.info/?p=4407

microsoft » microsoft » Microsoft High Availability Secrets Revealed

microsoft » microsoft » Microsoft High Availability Secrets Revealed — Mon, 08 Oct 2007 21:43:41 GMT

PingBack from http://microsoft.whitewitty.info/?p=4406

microsoft » microsoft » Do you need to use SQL Clustertering for High availability

Mon, 08 Oct 2007 21:43:40 GMT

PingBack from http://microsoft.whitewitty.info/?p=4405

microsoft » microsoft » microsoft » Microsoft.com SQL Server architecture

microsoft » microsoft » microsoft » Microsoft.com SQL Server architecture — Mon, 08 Oct 2007 21:45:04 GMT

PingBack from http://microsoft.whitewitty.info/?p=4418

microsoft » microsoft » microsoft » A great article on real life SQL cluster scenarios - at Microsoft of all places

Mon, 08 Oct 2007 21:45:06 GMT

PingBack from http://microsoft.whitewitty.info/?p=4417

microsoft » microsoft » microsoft » re: Where, Oh Where Did All of the Microsoft.com SQL Clusters Go?

Mon, 08 Oct 2007 21:45:14 GMT

PingBack from http://microsoft.whitewitty.info/?p=4416

microsoft » microsoft » microsoft » microsoft » Add a little Development, mix in some Security, a dash of Program Management, apply a liberal amount of IIS and SQL???Part of the Recipe for MSCOM Operations??? Blog

Mon, 08 Oct 2007 21:45:20 GMT

PingBack from http://microsoft.whitewitty.info/?p=4415

microsoft » microsoft » microsoft » microsoft » microsoft » Add a little Development, mix in some Security, a dash of Program Management, apply a liberal amount of IIS and SQL???Part of the Recipe for MSCOM

Mon, 08 Oct 2007 22:02:14 GMT

PingBack from http://microsoft.whitewitty.info/?p=4429

microsoft » microsoft » microsoft » microsoft » re: Where, Oh Where Did All of the Microsoft.com SQL Clusters Go?

Mon, 08 Oct 2007 22:02:18 GMT

PingBack from http://microsoft.whitewitty.info/?p=4428

microsoft » microsoft » microsoft » microsoft » A great article on real life SQL cluster scenarios - at Microsoft of all places

Mon, 08 Oct 2007 22:02:37 GMT

PingBack from http://microsoft.whitewitty.info/?p=4427

microsoft » microsoft » microsoft » microsoft » Microsoft.com SQL Server architecture

Mon, 08 Oct 2007 22:02:41 GMT

PingBack from http://microsoft.whitewitty.info/?p=4426

microsoft » microsoft » microsoft » microsoft » microsoft » Microsoft.com SQL Server architecture

Mon, 08 Oct 2007 23:39:45 GMT

PingBack from http://microsoft.whitewitty.info/?p=4452

microsoft » microsoft » microsoft » microsoft » microsoft » A great article on real life SQL cluster scenarios - at Microsoft of all places

Mon, 08 Oct 2007 23:40:01 GMT

PingBack from http://microsoft.whitewitty.info/?p=4451

microsoft » microsoft » microsoft » microsoft » microsoft » re: Where, Oh Where Did All of the Microsoft.com SQL Clusters Go?

Mon, 08 Oct 2007 23:40:14 GMT

PingBack from http://microsoft.whitewitty.info/?p=4450

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » Add a little Development, mix in some Security, a dash of Program Management, apply a liberal amount of IIS and SQL???Part o

Mon, 08 Oct 2007 23:40:18 GMT

PingBack from http://microsoft.whitewitty.info/?p=4449

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » Add a little Development, mix in some Security, a dash of Program Management, apply a liberal amount o

Tue, 09 Oct 2007 00:21:39 GMT

PingBack from http://microsoft.whitewitty.info/?p=4477

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » re: Where, Oh Where Did All of the Microsoft.com SQL Clusters Go?

Tue, 09 Oct 2007 00:21:44 GMT

PingBack from http://microsoft.whitewitty.info/?p=4476

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » A great article on real life SQL cluster scenarios - at Microsoft of all places

Tue, 09 Oct 2007 00:21:55 GMT

PingBack from http://microsoft.whitewitty.info/?p=4475

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » Microsoft.com SQL Server architecture

Tue, 09 Oct 2007 00:21:58 GMT

PingBack from http://microsoft.whitewitty.info/?p=4474

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » Microsoft.com SQL Server architecture

Tue, 09 Oct 2007 00:41:35 GMT

PingBack from http://microsoft.whitewitty.info/?p=4485

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » A great article on real life SQL cluster scenarios - at Microsoft of all places

Tue, 09 Oct 2007 00:41:44 GMT

PingBack from http://microsoft.whitewitty.info/?p=4484

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » re: Where, Oh Where Did All of the Microsoft.com SQL Clusters Go?

Tue, 09 Oct 2007 00:41:54 GMT

PingBack from http://microsoft.whitewitty.info/?p=4483

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » Add a little Development, mix in some Security, a dash of Program Management, ap

Tue, 09 Oct 2007 00:42:03 GMT

PingBack from http://microsoft.whitewitty.info/?p=4482

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » Add a little Development, mix in some Security, a dash of

Tue, 09 Oct 2007 01:45:37 GMT

PingBack from http://microsoft.whitewitty.info/?p=4498

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » A great article on real life SQL cluster scenarios - at Microsoft of all places

Tue, 09 Oct 2007 01:45:47 GMT

PingBack from http://microsoft.whitewitty.info/?p=4496

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » re: Where, Oh Where Did All of the Microsoft.com SQL Clusters Go?

Tue, 09 Oct 2007 01:45:57 GMT

PingBack from http://microsoft.whitewitty.info/?p=4497

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » Microsoft.com SQL Server architecture

Tue, 09 Oct 2007 01:46:19 GMT

PingBack from http://microsoft.whitewitty.info/?p=4495

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » re: Where, Oh Where Did All of the Microsoft.com SQL Clust

Tue, 09 Oct 2007 02:26:58 GMT

PingBack from http://microsoft.whitewitty.info/?p=4507

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » Microsoft.com SQL Server architecture

Tue, 09 Oct 2007 02:27:07 GMT

PingBack from http://microsoft.whitewitty.info/?p=4508

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » A great article on real life SQL cluster scenarios - at Mi

Tue, 09 Oct 2007 02:27:16 GMT

PingBack from http://microsoft.whitewitty.info/?p=4506

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » Add a little Development, mix in som

Tue, 09 Oct 2007 02:27:26 GMT

PingBack from http://microsoft.whitewitty.info/?p=4505

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » Add a little D

Tue, 09 Oct 2007 03:47:16 GMT

PingBack from http://microsoft.whitewitty.info/?p=4522

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft &

Tue, 09 Oct 2007 03:51:13 GMT

PingBack from http://microsoft.whitewitty.info/?p=4534

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » A great article on real life SQL clu

Tue, 09 Oct 2007 03:55:22 GMT

PingBack from http://microsoft.whitewitty.info/?p=4521

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » Microsoft.com SQL Server architectur

Tue, 09 Oct 2007 03:55:24 GMT

PingBack from http://microsoft.whitewitty.info/?p=4520

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » re: Where, Oh Where Did All of the M

Tue, 09 Oct 2007 03:55:35 GMT

PingBack from http://microsoft.whitewitty.info/?p=4519

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » re: Where, Oh

Tue, 09 Oct 2007 04:56:13 GMT

PingBack from http://microsoft.whitewitty.info/?p=4544

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft &

Tue, 09 Oct 2007 04:57:29 GMT

PingBack from http://microsoft.whitewitty.info/?p=4547

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » Microsoft.com

Tue, 09 Oct 2007 04:58:13 GMT

PingBack from http://microsoft.whitewitty.info/?p=4543

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » A great articl

Tue, 09 Oct 2007 04:58:15 GMT

PingBack from http://microsoft.whitewitty.info/?p=4542

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft &

Tue, 09 Oct 2007 04:58:28 GMT

PingBack from http://microsoft.whitewitty.info/?p=4541

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft &

Tue, 09 Oct 2007 05:41:31 GMT

PingBack from http://microsoft.whitewitty.info/?p=4577

microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft » microsoft &

Tue, 09 Oct 2007 05:41:40 GMT

PingBack from http://microsoft.whitewitty.info/?p=4576

Add a little Development, mix in some Security, a dash of Program Management, apply a liberal amount of IIS and SQL…Part of the Recipe for MSCOM Operations’ Blog

Microsoft.com Operations — Fri, 18 Jan 2008 23:59:04 GMT

Readers of this blog might have noticed (or been puzzled by) the variety of subject matter that we present.