ms datacenters : Environmental Sustainability

The Polarization of the Data Center Industry (at least for now): Part 2 of 2

msdcblog — Tue, 08 Dec 2009 04:21:00 GMT

The Inflection Point for Efficiency in the Data Center

By Christian Belady,
Principal Infrastructure Architect
Global Foundation Services, Microsoft Corporation

The Gartner graph (included in Part 1 of this blog) also made me think about the fact-- that had the same survey been asked 5 years ago-- it would have shown100 percent are either looking at internal metrics and/or not considering anything when it comes to energy or efficiency. In my opinion, it wasn’t until 2006 that the industry really did go through a paradigm shift. While there were a few of us who had been pushing efficient computing approaches for over a decade), we had limited success in moving the industry until 2006. What happened in 2006? I think the notion of establishing an industry efficiency metric and data center metrics were born. I thought it would be interesting to recap those milestones based on my perspective:

January 31 2006: Conference on Enterprise Servers and Data Centers: Opportunities for Energy Savings

This conference really brought Efficiency on the main stage of our industry. Andrew Fanara’s (EPA) presence was the key to waking up the industry and his goal was to drive metrics for delivering efficiency into computing. This (to me) was a pivotal day since it marked the first time we collectively got together as an industry to address efficiency. It was no longer a voice of a few, thanks to Andrew.

March 27, 2006: Energy Efficiency Server Benchmark Technical Workshop

This event was a follow-up meeting to the EPA conference in January. I led a work group on the Applicability of Benchmarks to the real world” with members from HP, Intel, AMD, LBNL, Cisco, Dell, Sun, Rumsey Engineers and others. The group concluded that a benchmark should be developed with power on the y-axis and workload on the x-axis. This recommendation is what drove the publication of the paper later that year that John Koomey spearheaded (with many of us in the industry co-authoring) titled “Server Energy Measurement Protocol”. This paper was the foundation of what SPECpower is today.

April 23-26, 2006: High-Density Computing: Trends, Challenges, Benefits, Costs, and Solutions

This symposium was the Uptime Institute’s first and it focused on Density trends. However, it was this conference where I first presented an Efficiency Metric called PUE which seemed to capture the attention of many of the attendees. As a result, I published a paper on PUE with my good friend Chris Malone later in the year at the Digital Power Forum. At this same conference, AMD’s Larry Vertal and Bruce Shaw sat down with Paul Perez (my former VP) and I to discuss the idea of starting a consortium called the Green Grid. Ten months later the Green Grid was officially announced with one of its first whitepapers evangelizing metrics and in particular PUE.

These three events to me really started it all. If you look at the Gartner graph, you can argue that all of the metrics were seeded in these three events and are really making an impact today.

So our industry woke up in 2006 and while the Gartner graph does show we have work ahead of us, I do think we can say that in less than four years the industry has made great strides (and perhaps I shouldn’t complain so much!).

/cb

The Polarization of the Data Center Industry (at least for now): Part 1 of 2

msdcblog — Tue, 08 Dec 2009 04:01:00 GMT

By Christian Belady,

Principal Infrastructure Architect

Global Foundation Services, Microsoft Corporation

I read a lot about what others are doing in the industry, and a recent post by Rich Miller of Data Center Knowledge caught my attention. It was titled “Gartner: Energy Woes to Worsen in 2010” and he posted a Graph from Gartner showing that only about 50 percent of the IT professionals are using some form of energy metrics in the data center.

At first I was not surprised, because as many of you heard me complain in the past, “this industry is moving too slow when it comes to efficiency”. Clearly, the adoption of efficiency metrics is quite low according to the graph. Less than 18 percent of those surveyed are using PUE/DCiE, less than 3 percent are using SPEC Power and perhaps 7 percent or so are using Energy Star. A little more than 20 percent are using other internal metrics…and I couldn’t help but wonder what they are. Additionally, why are the remaining 50 percent not measuring anything, particularly in a time when sustainability seems to be top of mind for everyone. Why do these folks seem to not care?

Perhaps the answer lies in the fact that maybe the respondents should have been weighted by their relative data center sizes. I would argue that any company – where a bulk of their costs are driven by their IT operations (which by the way are likely to be huge operations such as Microsoft, Yahoo, Ebay, Amazon, Google, etc.) – are further to the left on the graph because it has such a large impact on profitability. Similarly, those who have either small operations or their IT costs are low relative to other costs are further to the right. So in essence, we have a polarization of the data center industry: the Leftists and the Rightists.

The leftists are made up of large internet businesses that live and die with online commerce and consumer services. These guys are obsessed with driving cost out of their operations, are willing to break the rules and standards, believe they can change trends, quickly adopt new ideas, and take risks to substantially change their TCO models for the better. Because of their size, they can also hire specialists to engineer and optimize their data centers to levels never seen before.

Conversely, the rightists are made up of the more traditional enterprises where the IT department is just a very small part of their cost structure. In these organizations, the IT department is just expected to be there. It’s a cost that people accept because it is small and as long as there are no issues everyone is happy. The motivation for the operation’s folks is to simply make sure nothing goes wrong. As a result, there are no incentives to improve efficiency (especially if there is a perceived risk) and their priority is uptime. As a result, there is no reason to go to outside air economization, there is no reason to add new processes to monitor and measure efficiency, and there is no reason to look at new data center architectures that could risk downtime. This all makes sense if you look at the following graphs in which I compare two hypothetical companies, one with most of their costs in IT Operations (Leftists) and one with only 10 percent revenue dollars are eaten up by IT operations costs (Rightists):

Click image to view larger

This shows me why it may be that companies are still not motivated to focus on efficiency. In the case of the Leftist company - spending resources for a 20 percent IT Operations improvement in efficiency would return an improvement of 160 percent in profitability. However, that same improvement in the IT Operations for the Rightist company only yields an improvement of 20 percent in profitability. In their case, it would make sense to focus on improving their other costs. So the reality is we can’t expect everyone to think efficiency is so important! Ultimately it will boil down to how sensitive a business is to the costs of its IT operations. So in the short term, there appears to be this polarization that has occurred. However, to Gartner’s point, this could all change if energy costs increase substantially or carbon taxes emerge. In addition, what would happen if the big online providers (Leftists) hyper-optimized their operations and can offer cloud services at a fraction of the cost of running IT Operations that the Rightists can do themselves? Would this make all of those who did not focus on efficiency move to the cloud to survive?

Gartner’s survey really has made me think about this. In fact, I hope that the next time Gartner runs this survey they also track the following:

1) What percentage of the costs are in IT Operations?

2) How would the business be impacted if IT costs would double or even triple?

Tune in tomorrow for my thoughts on this issue and the “inflection point for efficiency”.

\cb

Presenting our Datacentre Strategy at Tech-Ed EMEA

msdcblog — Wed, 04 Nov 2009 19:41:00 GMT

Just a short post to let you know that I’ll be in Berlin for Tech-Ed EMEA from November 9 - 13 and I hope to see some of you there. I’ll be providing deeper insight into our new Dublin and Chicago datacentres, including our approaches to delivering on the scale and efficiency required to meet the capacity to support Microsoft’s global Software+Services and cloud computing business objectives.

I’ll be delivering two sessions:

· ITS207 – Microsoft Data Centres: From Buildings to Building Blocks

o In this session I’ll be providing the audience with a virtual tour of our new datacentres in Chicago and Dublin, exposing the evolution from traditional colocations to our Generation 3 mega-data-centres. I’ll talk about the design goals of these two facilities and will provide some insight into our upcoming Generation 4 modular datacentres, and our mission to deliver breakthroughs in flexible, highly efficient scale.

· MGT19-IS – Microsoft’s Mission for Efficient Datacentres

o This session will focus on the broader aspects of Microsoft’s efforts to improve the efficiency of our datacentres across the globe, as well as pointers to resources and tools that any datacentre operator or IT manager can take advantage of. Additionally, I’ll invite the audience to share their own ideas, questions and experiences in managing the capacity, scale and efficiency of their datacentre capacity. I’m really looking forward to this discussion.

More information on Tech-Ed EMEA can be found at http://www.microsoft.com/europe/teched/ Tech-Ed is a good venue for any and all technology professionals interested in exploring a broad set of current and upcoming Microsoft technologies, tools, platforms, and services.

Whether you can make it to these sessions or not, I’m looking forward to seeing you in Berlin. Feel free to tap me on the shoulder and say hello.

Catch you again shortly,

John Dwyer,

International Data Centre Manager

Global Foundation Services, Microsoft

Rolling out the Green Carpet for Dublin’s Neighbors

msdcblog — Mon, 26 Oct 2009 07:01:00 GMT

A few weeks back we had the privilege of hosting a data centre visit and tour for the residential neighbours of the Dublin data centre. Given the long projected lifespan of our new facility, we’re going to be neighbours for a long time, and so we felt...(read more)

Microsoft Celebrates Chicago Data Center Grand Opening

msdcblog — Mon, 28 Sep 2009 20:14:00 GMT

Containers Increase IT Efficiency of Microsoft’s Cloud Computing Infrastructure

Microsoft’s cloud computing infrastructure takes another big step forward this week with the grand opening of our Chicago data center. At more than 700,000 square feet, this facility significantly expands our ability to meet the demand generated from our Live, Online, and Cloud Computing services offerings for our customers. Combined with the grand opening of our Dublin, Ireland, data center last week to expand our capacity and network throughout EMEA, the opening in Chicago demonstrates how Microsoft is expanding capacity around the world to support its online businesses and customers.

Investing in turbulent economic times is always a tough choice – one that Microsoft is clearly making by investing in technology and innovation in a thoughtful and measured manner with an eye towards long-term growth for our customers, our shareholders, and our business. Microsoft’s data centers represent the infrastructure foundation of the company’s cloud services offerings and demonstrate how Microsoft is positioning itself to compete and succeed with an approach we call Software plus Services. Microsoft’s Software plus Services strategy is designed to create computing experiences that offer both the best of client technology and the best of the Web to connect people, data, devices, and applications.

One of many reasons we decided to hold a grand opening of the Chicago facility is to share our best practices. While in one sense our best practices are competitive advantages for Microsoft, we hope they will also help others in the industry make the cloud a safer and more reliable place that companies can trust for their operations. Very few companies can make the infrastructure investment that Microsoft has, so we think it is important to share what we’ve learned with the industry.

The first phase of the Chicago data center represents 30 megawatts of critical power. An additional 30 megawatts is pre-positioned for future growth. This incremental approach means customers today will enjoy top-notch performance and availability while we control costs for Microsoft and its shareholders.

One aspect of this data center that I’d like to talk about in some detail here involves our continued focus on environmental best practices.

Through the use of pre-manufactured, standard shipping containers, each of which house approximately 1,800 to 2,500 servers (as we’ve noted in previous blogs), we are able to realize greater conservation of energy and deliver new advancements in power efficiency. In addition, the isolated nature of containers enables Microsoft and its vendors to research new approaches around power and cooling alternatives to reduce energy consumption even more in the future.

Inside view of a container in the Chicago data center.

The Chicago facility is one of the largest data centers in the world to use containers. The entire first floor is devoted to parking stalls for containers, which will eventually house more than two-thirds of all the servers in the data center. The containers plug into standard interfaces called “CBlox” that we have developed with our partners. This interface provides a kind of “plug and play” for data center containers (for those of you who have been around long enough to remember when Microsoft developed “Plug and Play” for Windows).

See the video of containers being installed in the new Chicago facility.

Containers provide further environmental benefits in that they don’t require additional packaging materials or external form factors for the thousands of servers that they house. They also require less cabling and other equipment that all add up to unnecessary waste in traditional data center designs. Containers are also a very efficient way to quickly deploy capacity.

Another best practice we are using in Chicago is water-side economization, which enables us to cool the facility without requiring the high levels of electricity typically needed to power large chillers. Environmental sustainability is in Microsoft’s DNA and the Chicago data center serves as a great example.

I’d also like to talk about the incredible quality and amount of work that went into this data center, as well as the investments Microsoft has made in the local community throughout this project. Building this state-of-the-art facility generated approximately 3,000 construction-related jobs with a peak workforce of around 1,100 workers. More than 1.5 million man-hours of labor went into the project, and the total investment in the facility will top $500 million over time.

This week’s grand opening provides us the opportunity to thank those who helped turn our vision for the Chicago data center into reality. The facility actually began operations on July 20, delivering online services to our customers. It is gratifying to reflect on this achievement as we plan our next advances to move data center sustainability and efficiency forward for Microsoft, our customers, and the industry.

Arne

Arne Josefsberg,

General Manager of Infrastructure Services

Global Foundation Services

Microsoft

Dublin Data Center Celebrates Grand Opening

msdcblog — Thu, 24 Sep 2009 04:00:00 GMT

Environmental advances and expanded international computing delivered

This is a big week for Microsoft’s online, live, and cloud services as we celebrate the grand opening of our new data center in Dublin, Ireland. The Dublin facility delivers two key advances for Microsoft’s Software plus Services initiatives. One is expanded support for all our customers in the Europe, Middle East, and Africa region, thanks to Microsoft’s first mega data center built outside of the U.S. The other is dramatically improved environmental sustainability, resulting from innovative technology that takes advantage of the naturally cool climate in Ireland.

Regarding expanded international support, the new data center is already playing an important role in helping meet growing customer demand for Microsoft services in Europe and beyond. The facility, which began operations on July 1, 2009, currently covers 303,000 square feet, with 5.4 mega watts of critical power available to deliver services to consumers and business customers. Over time, the data center can expand to a total of 22.2 mega watts of critical power to support our growing cloud services. Combined with our other new mega data center in Chicago, Illinois—which we’ll celebrate the grand opening of next week on September 30—Microsoft is taking significant steps forward in our cloud computing capabilities.

It is important to note that at both facilities we will ramp up capacity incrementally to meet increases in customer demand. We are making thoughtful, measured investments in order to contain costs at the same time that we build out our infrastructure to deliver a secure cloud foundation, with robust performance and availability for services around the world.

Regarding the environment and Green IT factors, Microsoft is leveraging a combination of natural environmental factors to dramatically improve the environmental sustainability of the Dublin data center as compared to traditionally-built data centers. The average temperature range year round in Dublin is between 23 to 80 degrees F (-5 to 27 degrees C) and that, combined with the use of air-side economization, results in “free cooling” (chiller-free) operations 100% year round under normal operating conditions. This in turn significantly reduces water consumption and the use of chemicals required to treat cooling towers, which are common throughout the data center industry but not required in our Dublin facility.

Air-handling units on the roof of the data center (see photos above and below) draw outside air down into the facility to cool the server rooms, and then return hot air back out to the roof. Traditional data centers, on the other hand, cool server rooms with chillers, which consume a great deal of power and water. No chillers are used in the Dublin data center. The outside air that cools the facility is usually lower than the 95 degree F limit for our server rooms. If it ever exceeds that temperature, or in the extremely rare event of external air quality issues such as a nearby fire, Direct eXpansion (DX) cooling will be used. DX is a simpler means of mechanical cooling that is normally used for residential, automotive, or light commercial applications.

As a result of these measures the Dublin facility will use less than 1% of the water that traditional data center facilities typically use on an annual basis, and will improve Power Usage Effectiveness (PUE) by approximately 50%.

These improvements are part of our continued corporate commitment to environmental sustainability. Microsoft is a proud participant and endorser of the European Union (EU) Code of Conduct for Data Centres, a voluntary program that encourages organizations responsible for the operations of data centers to utilize technologies, systems, and processes that maximize the efficient use of electricity. We are one of the largest online service providers to sign the EU’s Code of Conduct for Data Centres. Our participation includes a commitment to continuing to develop and employ innovative systems and processes that deliver computing scale to meet our evolving Software plus Services business initiatives with the lowest possible consumption of power and natural resources. We also work actively with The Green Grid and Climate Savers Computing industry consortiums and the U.S. Environmental Protection Agency to advance sustainability and reduce the carbon footprint of data centers globally.

On September 16 we were honored to be recognized as a Best Practice in the European Commission's Sustainable Energy Europe Campaign for innovative achievements in design and operational strategies that are driving greater sustainability and efficiency at the Dublin facility.

While our best practices are competitive advantages for Microsoft, we hope they will also help others in the industry make the cloud a safer and more reliable place that companies can trust for their operations. Very few companies can make the infrastructure investment that Microsoft has, so we think it is important to share our best practices with the industry. This is one of many reasons we decided to hold a grand opening of the facility today.

I’d like to close by noting how this week’s grand opening helps celebrate and build upon Microsoft’s long-term investment in operations in the Republic of Ireland. Microsoft Ireland is now nearing the 25-year anniversary mark, and I believe the building of the Dublin data center will long be remembered as a key milestone of our relationship with this community. The $500 million facility is one of the largest construction projects in Ireland over last 12 months and has generated approximately 1 million man-hours of work with a peak workforce of around 2,100 workers. The data center will also provide approximately 35-50 jobs in the Dublin area for years to come.

It is exciting to unveil this marvellous facility and I’d like to extend my thanks to all the people inside and outside Microsoft who helped make it happen.

Arne

Arne Josefsberg,

General Manager of Infrastructure Services

Global Foundation Services

Microsoft

Microsoft Brings Two More Mega Data Centers Online in July

msdcblog — Mon, 29 Jun 2009 16:55:00 GMT

I am really excited that our team is now reaching another key milestone in data center innovations. July marks the launch of our two newest mega data centers in Chicago and Dublin. Our Dublin facility will go live on July 1, followed by our Chicago facility on July 20 to support our growing Online, Live, and Cloud services.

Together these Generation 3 facilities demonstrate Microsoft’s continuing commitment to improving data center efficiency with a focus on environmental sustainability.

The Dublin, Ireland, data center is our first mega data center built outside of the United States. This building covers 303,000 square feet, with 5.4 mega watts of critical power available now. Over time, the data center can expand to a total of 22.2 mega watts of critical power, growing with our business and customer demand. The facility makes extensive use of outside air economization to cool the facility year round, resulting in greater power efficiency with a resultant reduction in carbon footprint.

Dublin Data Center

The Chicago, Illinois facility covers over 700,000 square feet—approximately the size of 16 football fields—with critical power of 60 megawatts. Phase 1 represents 30 mega watts of critical power and the rest is pre-positioned for future growth. Two-thirds of the Chicago data center is optimized for housing containerized servers. Containers conserve energy and will help us realize new advancements in power efficiency with a PUE yearly average calculated at 1.22. These prepackaged units (with up to 1,800 to 2,500 servers each) can be wheeled into the facility and made operational within hours, so they represent important advances in the ability to quickly and efficiently provision capacity. The density inside the containers can exceed 10 times that of traditional data centers.

It is truly exciting to bring these two data centers online. We take great pride in the innovations they deliver to move the data center industry forward and to extend Microsoft’s online services to customers globally. As the company’s Software-plus-Services strategy progresses, these data centers will play a key supporting role.

I’d like to extend my congratulations and thanks to everyone at Microsoft who was involved in creating these state-of-the-art facilities and to all the local and regional trades people and organizations that helped make them possible. For more information on our cloud infrastructure strategy and services please visit our web site.

Arne

Arne Josefsberg,

General Manager of Infrastructure Services

Global Foundation Services

Microsoft’s Infrastructure Services Team Welcomes Kevin Timmons

msdcblog — Mon, 22 Jun 2009 17:00:00 GMT

Building an organization around exceptional leaders with the deepest industry expertise is core to how we evolve our Global Foundation Services organization. One such leader is Kevin Timmons, who joined Microsoft Global Foundation Services (GFS) today to head up our Data Center Services organization. Kevin brings a wealth of knowledge and passion in this space, most recently serving as vice president of Operations at Yahoo!, where he led the build-out of their data centers and infrastructure. Before that he was a director of Operations at GeoCities, and prior to that he served as a senior software engineer at Marconi Dynamics.

Kevin is known as a hands-on leader with a great grasp on the issues in his field and a keen interest in increasing energy efficiency. One of the key ways he has approached that challenge was by closely measuring efficiency at each data center and using PUE (Power Usage Effectiveness) as a key metric—a strategy that helped build more efficient data centers. Kevin also brings valuable experience and know-how in the field of data center site selection. Anyone familiar with our areas of focus in Microsoft data centers, especially around environmental sustainability, will recognize the great fit Kevin’s experiences bring to the team. In sitting down with him and exploring these areas in depth, I’ve become increasingly excited about the industry experience Kevin brings to our team.

In addition to bringing Kevin on board, we’ve recently restructured our Infrastructure Services team within GFS. In mid May we aligned the organization around five teams: Shared Infrastructure, Programmable Infrastructure, Platform Hardware and Standards, Global Network Services, and the Data Center Services team that Kevin now heads up. These changes will help us align our teams with Microsoft’s evolving cloud computing business and position our infrastructure for the upcoming year and beyond. We look forward to sharing more about these teams in upcoming postings.

Arne

Arne Josefsberg,

General Manager of Infrastructure Services

Global Foundation Services

Microsoft’s Top 10 Business Practices for Environmentally Sustainable Data Centers (celebrating Earth Day)

msdcblog — Tue, 21 Apr 2009 10:00:00 GMT

How to Reduce Energy Consumption, Waste, and Costs while Increasing Efficiency and ROI

What does environmental sustainability mean to data centers, industry operators, and IT businesses? A lot of managers in these areas may barely notice the Earth Day global event on April 22 as they struggle to support their businesses in the face of budget cuts and uncertainty about the future. But the fact is, being “lean and green” is good for both the business and the environment, and organizations that focus their attentions accordingly will see clear benefits. Reducing energy use and waste improves a company’s bottom line and boosting the use of recycled materials is a proven way to demonstrate good corporate citizenship to your customers, employees, and the communities you do business in.

That said, with so much on the plates of data center and IT professionals these days it isn’t always easy to know where to begin in moving to greener and more efficient operations. With that in mind—along with Microsoft’s commitment to share best practices with the rest of the data center industry—we asked some of the senior members our Global Foundation Services’ Infrastructure Services team to send us their top ten best business practices for environmentally sustainable data centers and IT.

Their favorite practices are reflected in our new “Microsoft’s Top 10 Business Practices for Environmentally Sustainable Data Centers” paper.

You can explore other data center resources on our GFS web site here.

Microsoft announced in March 2009 that it is taking a proactive corporate approach to reduce our carbon emissions per unit of revenue by at least 30 percent below 2007 levels by 2012. The data centers managed by GFS are a significant component of Microsoft’s carbon footprint, and our organization plays a vital role in Microsoft’s efforts to meet this corporate goal.

As you’ll read in the list of best practices we’ve compiled, companies can make major gains in energy efficiency by increasing server utilization and moving to virtualization, plus you can employ a wide range of smaller initiatives that collectively add up to significant gains. Microsoft has been using these practices for several years now and found that in addition to helping us protect the environment, they make best use of our resources and help us stay tightly aligned with our core strategies and goals. We hope they work for you as well.

Below is an overview of the best practices list.

1. Provide incentives that support your primary goals.

2. Focus on effective resource utilization.

3. Use virtualization to improve server utilization and increase operational efficiency.

4. Drive quality up through compliance.

5. Embrace change management.

6. Invest in understanding your application workload and behavior.

7. Right-size your server platforms to meet your application requirements.

8. Evaluate and test servers for performance, power, and total cost of ownership.

9. Converge on as small a number of stock-keeping units (SKUs) as you can.

10. Take advantage of competitive bids from multiple vendors to foster innovation and reduce costs.

Beyond the business practices listed above, Microsoft’s data center team is taking significant steps in four areas important to environmental sustainability:

· Using recycled resources whenever practical: The Microsoft data center in San Antonio, Texas, for example, uses approximately eight million gallons of recycled water a month from the city’s waste water system during peak cooling months.

· Using renewable resources whenever available: In the process we have developed for deciding where to build our data center facilities, renewable energy sources play a key role. For instance, the Microsoft data center in Quincy, Washington, uses 100 percent renewable hydropower from the Columbia Basin River. The San Antonio facility obtains part of its electricity from wind power. And the company’s Dublin, Ireland, data center will use outside air for cooling, thus reducing the need for energy-intensive coolers.

· Reducing waste in operations: One example of Microsoft’s focus on reducing waste is the company’s transition to using standard shipping containers to house thousands of servers apiece. Ordering servers by the truckload eliminates the need for large amounts of packaging and other materials previously required when servers were delivered individually or in racks.

· Actively participating in industry environmental groups: Microsoft plays leadership roles in Climate Savers Computing Initiative and The Green Grid—industry organizations focused on improving computer systems and data center energy efficiency and establishing a standard methodology for measuring Power Usage Effectiveness (PUE) accurately and consistently. We also feel it’s important for us to openly share information and best practices around data center energy efficiency, because we believe the data center industry needs to work together to make dramatic gains toward environmental sustainability.

The teams that contributed to this white paper are represented by our power and cooling architect Christian Belady, our distinguished engineer Dileep Bhandarkar, our Data Center Services director Daniel Costello (“the father” of our Generation 4 modular data center vision and the leader of the team of engineers that created it), our Data Center Software Services general manager Jawaid Ekram, and myself.

Their backgrounds include expertise in data center electrical and mechanical engineering, power and cooling architecture and design, research and development, and business operations and administration. It has been my distinct pleasure to work with and lead these subject matter experts and to share their thoughts with the industry via our blogs and participation in industry events.

Arne

Arne Josefsberg,

General Manager of Infrastructure Services

Global Foundation Services

Our Vision for Generation 4 Modular Data Centers – One way of Getting it just right . . .

msdcblog — Tue, 02 Dec 2008 05:13:00 GMT

The following posting originally appeared on Michael Manos' Loose Bolts blog.

Data Centers are a hot topic these days. No matter where you look, this once obscure aspect of infrastructure is getting a lot of attention. For years, there have been cost pressures on IT operations and this, when the need for modern capacity is greater than ever, has thrust data centers into the spotlight. Server and rack density continues to rise, placing DC professionals and businesses in tighter and tougher situations while they struggle to manage their IT environments. And now hyper-scale cloud infrastructure is taking traditional technologies to limits never explored before and focusing the imagination of the IT industry on new possibilities.

At Microsoft, we have focused a lot of thought and research around how to best operate and maintain our global infrastructure and we want to share those learnings. While obviously there are some aspects that we keep to ourselves, we have shared how we operate facilities daily, our technologies and methodologies, and, most importantly, how we monitor and manage our facilities. Whether it’s speaking at industry events, inviting customers to our “Microsoft data center conferences” held in our data centers, or through other media like blogging and white papers, we believe sharing best practices is paramount and will drive the industry forward. So in that vein, we have some interesting news to share.

Today we are sharing our Generation 4 Modular Data Center plan. This is our vision and will be the foundation of our cloud data center infrastructure in the next five years. We believe it is one of the most revolutionary changes to happen to data centers in the last 30 years. Joining me, in writing this blog are Daniel Costello, my director of Data Center Research and Engineering and Christian Belady, principal power and cooling architect. I feel their voices will add significant value to driving understanding around the many benefits included in this new design paradigm.

Our “Gen 4” modular data centers will take the flexibility of containerized servers—like those in our Chicago data center—and apply it across the entire facility. So what do we mean by modular? Think of it like “building blocks”, where the data center will be composed of modular units of prefabricated mechanical, electrical, security components, etc., in addition to containerized servers.

Was there a key driver for the Generation 4 Data Center?

If we were to summarize the promise of our Gen 4 design into a single sentence it would be something like this: “A highly modular, scalable, efficient, just-in-time data center capacity program that can be delivered anywhere in the world very quickly and cheaply, while allowing for continued growth as required.” Sounds too good to be true, doesn’t it? Well, keep in mind that these concepts have been in initial development and prototyping for over a year and are based on cumulative knowledge of previous facility generations and the advances we have made since we began our investments in earnest on this new design.

One of the biggest challenges we’ve had at Microsoft is something Mike likes to call the ‘Goldilock’s Problem’. In a nutshell, the problem can be stated as:

The worst thing we can do in delivering facilities for the business is not have enough capacity online, thus limiting the growth of our products and services.

The second worst thing we can do in delivering facilities for the business is to have too much capacity online.

This has led to a focus on smart, intelligent growth for the business — refining our overall demand picture. It can’t be too hot. It can’t be too cold. It has to be ‘Just Right!’ The capital dollars of investment are too large to make without long term planning. As we struggled to master these interesting challenges, we had to ensure that our technological plan also included solutions for the business and operational challenges we faced as well.

So let’s take a high level look at our Generation 4 design

Are you ready for some great visuals? Click here for the Microsoft 4th Gen Video. It’s a concept video that came out of my Data Center Research and Engineering team, under Daniel Costello, that will give you a view into what we think is the future.

From a configuration, construct-ability and time to market perspective, our primary goals and objectives are to modularize the whole data center. Not just the server side (like the Chicago facility), but the mechanical and electrical space as well. This means using the same kind of parts in pre-manufactured modules, the ability to use containers, skids, or rack-based deployments and the ability to tailor the Redundancy and Reliability requirements to the application at a very specific level.

Our goals from a cost perspective were simple in concept but tough to deliver. First and foremost, we had to reduce the capital cost per critical Mega Watt by the class of use. Some applications can run with N-level redundancy in the infrastructure, others require a little more infrastructure for support. These different classes of infrastructure requirements meant that optimizing for all cost classes was paramount. At Microsoft, we are not a one trick pony and have many Online products and services (240+) that require different levels of operational support. We understand that and ensured that we addressed it in our design which will allow us to reduce capital costs by 20%-40% or greater depending upon class.

For example, non-critical or geo redundant applications have low hardware reliability requirements on a location basis. As a result, Gen 4 can be configured to provide stripped down, low-cost infrastructure with little or no redundancy and/or temperature control. Let’s say an Online service team decides that due to the dramatically lower cost, they will simply use uncontrolled outside air with temperatures ranging 10-35 C and 20-80% RH. The reality is we are already spec-ing this for all of our servers today and working with server vendors to broaden that range even further as Gen 4 becomes a reality. For this class of infrastructure, we eliminate generators, chillers, UPSs, and possibly lower costs relative to traditional infrastructure.

Applications that demand higher level of redundancy or temperature control will use configurations of Gen 4 to meet those needs, however, they will also cost more (but still less than traditional data centers). We see this cost difference driving engineering behavioral change in that we predict more applications will drive towards Geo redundancy to lower costs.

Another cool thing about Gen 4 is that it allows us to deploy capacity when our demand dictates it. Once finalized, we will no longer need to make large upfront investments. Imagine driving capital costs more closely in-line with actual demand, thus greatly reducing time-to-market and adding the capacity Online inherent in the design. Also reduced is the amount of construction labor required to put these “building blocks” together. Since the entire platform requires pre-manufacture of its core components, on-site construction costs are lowered. This allows us to maximize our return on invested capital.

In our design process, we questioned everything. You may notice there is no roof and some might be uncomfortable with this. We explored the need of one and throughout our research we got some surprising (positive) results that showed one wasn’t needed.

In short, we are striving to bring Henry Ford’s Model T factory to the data center. http://en.wikipedia.org/wiki/Henry_Ford#Model_T. Gen 4 will move data centers from a custom design and build model to a commoditized manufacturing approach. We intend to have our components built in factories and then assemble them in one location (the data center site) very quickly. Think about how a computer, car or plane is built today. Components are manufactured by different companies all over the world to a predefined spec and then integrated in one location based on demands and feature requirements. And just like Henry Ford’s assembly line drove the cost of building and the time-to-market down dramatically for the automobile industry, we expect Gen 4 to do the same for data centers. Everything will be pre-manufactured and assembled on the pad.

And did we mention that this platform will be, overall, incredibly energy efficient? From a total energy perspective not only will we have remarkable PUE values, but the total cost of energy going into the facility will be greatly reduced as well. How much energy goes into making concrete? Will we need as much of it? How much energy goes into the fuel of the construction vehicles? This will also be greatly reduced! A key driver is our goal to achieve an average PUE at or below 1.125 by 2012 across our data centers. More than that, we are on a mission to reduce the overall amount of copper and water used in these facilities. We believe these will be the next areas of industry attention when and if the energy problem is solved. So we are asking today…“how can we build a data center with less building”?

We have talked openly and publicly about building chiller-less data centers and running our facilities using aggressive outside economization. Our sincerest hope is that Gen 4 will completely eliminate the use of water. Today’s data centers use massive amounts of water and we see water as the next scarce resource and have decided to take a proactive stance on making water conservation part of our plan.

By sharing this with the industry, we believe everyone can benefit from our methodology. While this concept and approach may be intimidating (or downright frightening) to some in the industry, disclosure ultimately is better for all of us.

Gen 4 design (even more than just containers), could reduce the ‘religious’ debates in our industry. With the central spine infrastructure in place, containers or pre-manufactured server halls can be either AC or DC, air-side economized or water-side economized, or not economized at all (though the sanity of that might be questioned). Gen 4 will allow us to decommission, repair and upgrade quickly because everything is modular. No longer will we be governed by the initial decisions made when constructing the facility. We will have almost unlimited use and re-use of the facility and site. We will also be able to use power in an ultra-fluid fashion moving load from critical to non-critical as use and capacity requirements dictate.

Finally, we believe this is a big game changer. Gen 4 will provide a standard platform that our industry can innovate around. For example, all modules in our Gen 4 will have common interfaces clearly defined by our specs and any vendor that meets these specifications will be able to plug into our infrastructure. Whether you are a computer vendor, UPS vendor, generator vendor, etc., you will be able to plug and play into our infrastructure. This means we can also source anyone, anywhere on the globe to minimize costs and maximize performance. We want to help motivate the industry to further innovate—with innovations from which everyone can reap the benefits.

To summarize, the key characteristics of our Generation 4 data centers are:

Scalable
Plug-and-play spine infrastructure
Factory pre-assembled: Pre-Assembled Containers (PACs) & Pre-Manufactured Buildings (PMBs)
Rapid deployment
De-mountable
Reduce TTM
Reduced construction
Sustainable measures
Map applications to DC Class

We hope you join us on this incredible journey of change and innovation!

Long hours of research and engineering time are invested into this process. There are still some long days and nights ahead, but the vision is clear. Rest assured however, that we as refine Generation 4, the team will soon be looking to Generation 5 (even if it is a bit farther out). There is always room to get better.

So if you happen to come across Goldilocks in the forest, and you are curious as to why she is smiling you will know that she feels very good about getting very close to ‘JUST RIGHT’.

Generations of Evolution – some background on our data center designs

We thought you might be interested in understanding what happened in the first three generations of our data center designs. When Ray Ozzie wrote his Software plus Services memo it posed a very interesting challenge to us. The winds of change were at ‘tornado’ proportions. That “plus Services” tag had some significant (and unstated) challenges inherent to it. The first was that Microsoft was going to evolve even further into an operations company. While we had been running large scale Internet services since 1995, this development lead us to an entirely new level. Additionally, these “services” would span across both Internet and Enterprise businesses. To those of you who have to operate “stuff”, you know that these are two very different worlds in operational models and challenges. It also meant that, to achieve the same level of reliability and performance required our infrastructure was going to have to scale globally and in a significant way.

It was that intense atmosphere of change that we first started re-evaluating data center technology and processes in general and our ideas began to reach farther than what was accepted by the industry at large. This was the era of Generation 1. As we look at where most of the world’s data centers are today (and where our facilities were), it represented all the known learning and design requirements that had been in place since IBM built the first purpose-built computer room. These facilities focused more around uptime, reliability and redundancy. Big infrastructure was held accountable to solve all potential environmental shortfalls. This is where the majority of infrastructure in the industry still is today.

We soon realized that traditional data centers were quickly becoming outdated. They were not keeping up with the demands of what was happening technologically and environmentally. That’s when we kicked off our Generation 2 design. Gen 2 facilities started taking into account sustainability, energy efficiency, and really looking at the total cost of energy and operations. No longer did we view data centers just for the upfront capital costs, but we took a hard look at the facility over the course of its life. Our Quincy, Washington and San Antonio, Texas facilities are examples of our Gen 2 data centers where we explored and implemented new ways to lessen the impact on the environment. These facilities are considered two leading industry examples, based on their energy efficiency and ability to run and operate at new levels of scale and performance by leveraging clean hydro power (Quincy) and recycled waste water (San Antonio) to cool the facility during peak cooling months.

As we were delivering our Gen 2 facilities into steel and concrete, our Generation 3 facilities were rapidly driving the evolution of the program. The key concepts for our Gen 3 design are increased modularity and greater concentration around energy efficiency and scale. The Gen 3 facility will be best represented by the Chicago, Illinois facility currently under construction. This facility will seem very foreign compared to the traditional data center concepts most of the industry is comfortable with. In fact, if you ever sit around in our container hanger in Chicago it will look incredibly different from a traditional raised-floor data center. We anticipate this modularization will drive huge efficiencies in terms of cost and operations for our business. We will also introduce significant changes in the environmental systems used to run our facilities. These concepts and processes (where applicable) will help us gain even greater efficiencies in our existing footprint, allowing us to further maximize infrastructure investments.

This is definitely a journey, not a destination industry. In fact, our Generation 4 design has been under heavy engineering for viability and cost for over a year. While the demand of our commercial growth required us to make investments as we grew, we treated each step in the learning as a process for further innovation in data centers. The design for our future Gen 4 facilities enabled us to make visionary advances that addressed the challenges of building, running, and operating facilities all in one concerted effort.

/Mm/Dc/Cb

Out of the Box Paradox – Manifested (aka Chicago Area Data Center begins its journey)

msdcblog — Mon, 20 Oct 2008 18:06:00 GMT

The following posting originally appeared on Michael Manos' Loose Bolts blog.

With modern conventional thinking and untold management consultants coaching people to think outside the box, I find it humorous that we have actually physically manifested an “Out of the Box Paradox” in Chicago.

What is an Out of the Box Paradox you ask? Well I will refer to Wikipedia on this one for a great example:

“The encouragement of thinking outside the box, however, has possibly become so popular that thinking inside the box is starting to become more unconventional. This kind of “going against the grain means going with the grain” mentality causes a paradox in that there may be no such thing as conventionality when unconventionality becomes convention.”

The funny part here is that we are actually doing this with….you guessed it…..boxes. Today we finished the first phase of construction and we are rolling into the testing of container-based deployments. Our facility in Chicago is our first purpose-built data center to accommodate containers on a large scale. It has been an incredibly interesting journey. The challenges of solving things that have never been done before are many. We even had to create our own container specification, one specifically with the end-user in mind to ensure we maximized the cost and efficiency gains possible, not to mention standard blocking and tackling issues like standardizing power, water, network and other interfaces. All sorts of interesting things have been discovered, corrected, and perfected. From electrical harmonics issues to streamlining materials movement, to whole new operational procedures.

The facility is already simply amazing and it’s a wonder to behold. Construction kicked off only one year ago and when completed it will have the capacity to scale to hundreds of thousands of servers which can be deployed (and de-commissioned as needed) very quickly. The joke we use internally is that this is not your mother’s data center. You get that impression from the first moment you step into the “hangar bay” on the first floor. The “hangar’s” first floor will house the container deployments and I can assure you it is like no data center you have ever seen. It’s one more step to the industrialization of the IT world, or at least the cloud-scale operations space. To be fair, and it’s important to note, only one half of the total facility is ready at this point, but even half of this facility is significant in terms of total capacity.

That “Industrialization of IT” is one of the core tenets of my mission at Microsoft. Throwing smart bodies at dumb problems is not really smart at all. The real quest is how to drive innovation and automation into everything that you do to reduce the amount of work that needs to be performed by humans. Dedicate your smart people for solving hard problems. It’s more than a mission, it’s a philosophy deeply rooted in our organization. Besides, industry numbers tell us that humans are the leading cause of outages in data center facilities. Our Chicago facility is a huge step forward to driving that industrialization increasingly forward. It truly represents an evolution and demonstrates what could happen when you blend the power of software and breakthrough innovative design and engineering. Even for buildings!

I have watched with much interest the back and forth on containers in the media, in the industry, and the interesting uses being proposed by the industry. The fact of the matter is that Containers are a great “Out of the Box Paradox” that really should not be terribly shocking to the industry at large.

The idea of “containment” is almost as old as mechanical engineering and thermodynamics itself. Containment gives you the ability to manage the heat or lack thereof more effectively in individual ecosystems. Forward looking designers have been doing “containment” for a long time. So going back to the paradox that “out of the box, is in the box thinking” shift, the concept is not terribly new. It’s the application at our scale and specifically to the data center world which is most interesting.

It allows us to get out of the traditional decision points common to the data center industry in that certain infrastructure decisions actually reside in the container itself, which allows for a much quicker refresh cycle of key components and the ability to swap out for the next greatest technology rapidly. Therefore, by default it allows us to deploy our capital infrastructure costs much more closely aligned with actual need versus the large step functions one normally sees in data center construction (build a large expensive facility, and fill it up over time versus build capacity out as you need it). This allows you to better manage costs, better manage your business, and give you the best possible ramp for technology refresh. You don’t particularly care if its AC or DC, if it’s water cooled or air cooled. Our metrics are simple – Give us the best performing, most efficient, lowest TCO technology to meet our needs. If today that’s AC, great. Tomorrow DC? Fantastic. Do I want to be able to do a bake-off between the two? Sure. I don’t have to reinvest huge funds in my facilities to make those changes.

For those of you with real lives and have not been following the whole container debates here is a quick recap -

Microsoft is using standard 40 foot shipping containers for the deployment of servers in support of the software + services strategy and in support of our cloud services infrastructure initiatives.
The containers can house as many as 2500 servers achieving a density of 10 times the amount of compute in the equivalent space in a traditional data center.
We believe containers offer huge advantages at scale in terms of both initial capital and ongoing operating costs.
This idea has met some resistance in the industry. As highlighted by my interesting back and forth with Eric Lai from Computerworld magazine. Original article can be found here, with my “Anthills” response found here.
Chicago represents one of the first purpose-built container-built facilities ever.

To be clear, as I have said in the past, containers are not for everyone, but they are great for us.

The other thing which is important is the energy efficiency of the containers. Now I want to be careful here as the reporting of efficiency numbers can be a dangerous exercise in the blogo-sphere. But our testing shows that our containers in Chicago can deliver an average PUE of 1.22 with an AVERAGE ANNUAL PEAK PUE of 1.36. I break these two numbers out separately because there is still some debate (at least in the circles I travel in) on which of these metrics is more meaningful. Regardless of your position on which is more meaningful, you have to admit those numbers are pretty darn compelling.

For the purists and math-heads out there, Microsoft includes house lighting and office loads in our PUE calculation. They are required to run the facility so we count them as overhead.

On the “Sustainability” side of containers it’s also interesting to note that shipping 2500 servers in one big container has a positive reduction on the CO2 related to transportation, let alone the amount of packaging material eliminated.

So in my mind, containers are driving huge cost and efficiency (read also as cost benefits in addition to “green” benefits) gains for the business. This is an extremely important point, as Microsoft expands its data center infrastructure, it is supremely important that we follow an established smart growth methodology for our facilities that is designed to prevent overbuilding—and thus avoid associated costs to the environment and to our shareholders. We are a business after all. We must do all of this while also meeting the rapidly growing demand for Microsoft’s Online and Live services.

Containers, and this new approach is definitely a change in how facilities have traditionally been developed, and as a result many people in our industry are intimidated by it. But they shouldn’t be. Data center’s have not changed in fundamental design for decades. Sometimes change is good. The exposure to any new idea is always met with resistance, but with a little education things change over time.

In that vein we are looking at holding our second Microsoft Data Center Experience (MDX) event in Chicago in the Spring/Summer 2009. Our first event held in San Antonio, was basically an opportunity for a couple hundred Microsoft enterprise customers to tour our facilities, ask all the questions they wanted, interact with our Data Center experts (mechanical, electrical, operations, facilities management, etc.), and generally get a feel to our approach. It’s not that ours is the right way, or the wrong way…..just our way. Think of an Operations event for Operations people, by Operations people.

It’s not glamorous, there are no product pitches, no slick brochures, no hardware hunks or booth babes, but hopefully it’s interesting. That first event was hugely successful with incredible feedback from our customers. As a result, we decided to do the same thing in Chicago with the very first container data center. Which of course makes things a bit tricky. While the facility will be going through a vigorous testing phase from effectively now moving forward, we thought it better to ensure that any and all construction activity be formally complete before we go moving large groups of people through our facility to ensure safety. Plus, I don’t think I have enough hard hats and safety gear for you all.

So if you attended MDX-San Antonio and really want to drill deeper in on Containers, in a facility custom built for them, or would like to attend just to ask questions, look for details on it from your Microsoft account management team or your local Microsoft sales office for details next Spring. (Although it’s not a sales event, you are more likely to reach someone there faster than calling into Global Foundation Services directly, after all we have a global infrastructure to run.)

/Mm

DataCenter Think Tanks Sheepish on Weightloss

msdcblog — Fri, 15 Aug 2008 18:15:00 GMT

The following posting originally appeared on Michael Manos' Loose Bolts blog.

Matt Stansbury over at Data Center Facilities Pro posted an interesting post regarding a panel containing Uptime’s Ken Brill. The note warns folks on the use of PUE as a benchmarking standard between data centers.

I can’t say I really disagree with what he says. In my mind, self measurement is always an intensely personal thing. To me, PUE is a great self-measurement tool to drive towards power efficiency in your data center. Do you include lighting? Do you include your mechanical systems? To me those questions are not all that dissimilar to the statement, “I am losing weight”. Are you weighing yourself nude? in the your underwear? with your shoes on?

I do think the overall PUE metric could go a little farther to fully define what *MUST* be in the calculation, especially if you are going to use it comparatively. But those who want to use this metric in some kind of competitive game are completely missing the point. This is ultimately about using the power resources you have to its highest possible efficiency. As I have stated over and over, and as recently as the recent Data Center Dynamics conference in Seattle. Every Data Center is different. If I tried to compare the efficiency of one of our latest generation facilities in San Antonio or Dublin to a facility built 10 years ago, assuming we made sure that we were comparing apples to apples with like systems included, of course the latest generation facilities would be better off. A loss of 5 pounds on an Olympic runner with 4% body fat compared to a loss of 5 pounds on professional sumo wrestler have dramatically different effects (or non effects).

Everyone knows I am a proponent of PUE/DCiE. So when you read this understand where my baggage is carried. To me the use of either, or, or both of these is a matter of audience. Engineers love efficiency. Business Managers understand overhead. Regardless the measurement is consistent and more importantly the measurement is happening with some regularity. This is more important than anything.

If we are going to attempt to use PUE for full scale facility comparison a couple of things have to happen. At Microsoft we measure PUE aggressively and often. This speaks to the time element that Ken mentions in his talk in the post. It would be great for the Green Grid or Uptime or anyone to produce the “Imperial Standard”. One could even think that these groups could earn some extra revenue by certifying facilities to the “Imperial PUE standard”. This would include minimum measurement cylces (once a day, twice a day, average for a year, peak for a year, etc). Heavens knows it would be a far more useful metric for measuring data centers than the current LEEDS certifications. But thats another post for another time. Seriously, the time element is hugely important. Measuring your data center once at midnight in January while experiencing the coldest winter on record might make great marketing, but it doesnt mean much.

As an industry we have to face the fact that there are morons amongst us. This, of course is made worse if people are trying to advertise PUE as a competitive advantage due mostly to the fact that this means that they have engaged marketing people to “enhance” the message. Ken’s mention of someone announcing that their PUE of .8 should instantly flag that person as an idiot and you should hand them an Engvallian sign. But even barring these easy to identify examples we must remember that any measurement can be gamed. In fact, I would go so far as to say that gaming measurements is the national pastime of all businesses.

Ultimately I just chalk this up to another element of “Green-washing” that our industry is floating in.

Ken also talks about the use of the word “Power” being incorrect and that because it is a point in time measurement versus an over time measurement and that we should be focused on “Energy”. According to Ken this could ultimately doom the measurement on the whole. I think this is missing the point entirely on two fronts. First whether you call it power or energy, the naming semantics dont really matter. They matter to english professors and people writing white papers, but it terms of actually doing something, it has no effect. The simple act of measuring is the most critical concept here. Measure something, get better. Whether you like PUE, DCIE or whether you want to adopt “Energy” and call it EUE and embrace a picture of a sheep with power monitoring apparatus attached to its back, the name doesnt really matter. )Though I must admit, a snappy mascot might actually drive more people to measure. Just do something!

My informal polling at speaking engagements continues on the state of the industry and I am sad to say, the amount of people actively measuring power consumption remains less than 10% (let alone measuring for efficiency!), and if anything the number seems to be declining.

In my mind, as an end-user, the thrash that we see coming from the standards bodies and think tank organizations like Uptime, Green Grid, and others should really stop bickering over whose method of calculation is better or has the best name. We have enough challenge getting the industry to adopt ANY KIND of measurement. To confuse matters more and argue the finer points of absurdity is only going to further magnify this thrash and ensure we continue to confuse most data center operators into more non-action . As an industry we are heading down a path with our gun squarely aimed at out foot. If we are not careful, the resultant wound is going to end up in amputation.

- MM