ms datacenters : Gen 4.0 Data Center

The Polarization of the Data Center Industry (at least for now): Part 2 of 2

msdcblog — Tue, 08 Dec 2009 04:21:00 GMT

The Inflection Point for Efficiency in the Data Center

By Christian Belady,
Principal Infrastructure Architect
Global Foundation Services, Microsoft Corporation

The Gartner graph (included in Part 1 of this blog) also made me think about the fact-- that had the same survey been asked 5 years ago-- it would have shown100 percent are either looking at internal metrics and/or not considering anything when it comes to energy or efficiency. In my opinion, it wasn’t until 2006 that the industry really did go through a paradigm shift. While there were a few of us who had been pushing efficient computing approaches for over a decade), we had limited success in moving the industry until 2006. What happened in 2006? I think the notion of establishing an industry efficiency metric and data center metrics were born. I thought it would be interesting to recap those milestones based on my perspective:

January 31 2006: Conference on Enterprise Servers and Data Centers: Opportunities for Energy Savings

This conference really brought Efficiency on the main stage of our industry. Andrew Fanara’s (EPA) presence was the key to waking up the industry and his goal was to drive metrics for delivering efficiency into computing. This (to me) was a pivotal day since it marked the first time we collectively got together as an industry to address efficiency. It was no longer a voice of a few, thanks to Andrew.

March 27, 2006: Energy Efficiency Server Benchmark Technical Workshop

This event was a follow-up meeting to the EPA conference in January. I led a work group on the Applicability of Benchmarks to the real world” with members from HP, Intel, AMD, LBNL, Cisco, Dell, Sun, Rumsey Engineers and others. The group concluded that a benchmark should be developed with power on the y-axis and workload on the x-axis. This recommendation is what drove the publication of the paper later that year that Jon Koomey spearheaded (with many of us in the industry co-authoring) titled “Server Energy Measurement Protocol”. This paper was the foundation of what SPECpower is today.

April 23-26, 2006: High-Density Computing: Trends, Challenges, Benefits, Costs, and Solutions

This symposium was the Uptime Institute’s first and it focused on Density trends. However, it was this conference where I first presented an Efficiency Metric called PUE which seemed to capture the attention of many of the attendees. As a result, I published a paper on PUE with my good friend Chris Malone later in the year at the Digital Power Forum. At this same conference, AMD’s Larry Vertal and Bruce Shaw sat down with Paul Perez (my former VP) and I to discuss the idea of starting a consortium called the Green Grid. Ten months later the Green Grid was officially announced with one of its first whitepapers evangelizing metrics and in particular PUE.

These three events to me really started it all. If you look at the Gartner graph, you can argue that all of the metrics were seeded in these three events and are really making an impact today.

So our industry woke up in 2006 and while the Gartner graph does show we have work ahead of us, I do think we can say that in less than four years the industry has made great strides (and perhaps I shouldn’t complain so much!).

/cb

For more information on Microsoft's datacenter best practices visit our external web site at www.globalfoundationservices.com and check out Rob Bernard's feature “Using the Power of IT to Drive Environmental Sustainability” that highlights the effort Microsoft is making in environmental sustainability, as well as how that directly relates to what world leaders are currently discussing in Copenhagen for the United Nations Climate Change Conference 2009.

Presenting our Datacentre Strategy at Tech-Ed EMEA

msdcblog — Wed, 04 Nov 2009 19:41:00 GMT

Just a short post to let you know that I’ll be in Berlin for Tech-Ed EMEA from November 9 - 13 and I hope to see some of you there. I’ll be providing deeper insight into our new Dublin and Chicago datacentres, including our approaches to delivering on the scale and efficiency required to meet the capacity to support Microsoft’s global Software+Services and cloud computing business objectives.

I’ll be delivering two sessions:

· ITS207 – Microsoft Data Centres: From Buildings to Building Blocks

o In this session I’ll be providing the audience with a virtual tour of our new datacentres in Chicago and Dublin, exposing the evolution from traditional colocations to our Generation 3 mega-data-centres. I’ll talk about the design goals of these two facilities and will provide some insight into our upcoming Generation 4 modular datacentres, and our mission to deliver breakthroughs in flexible, highly efficient scale.

· MGT19-IS – Microsoft’s Mission for Efficient Datacentres

o This session will focus on the broader aspects of Microsoft’s efforts to improve the efficiency of our datacentres across the globe, as well as pointers to resources and tools that any datacentre operator or IT manager can take advantage of. Additionally, I’ll invite the audience to share their own ideas, questions and experiences in managing the capacity, scale and efficiency of their datacentre capacity. I’m really looking forward to this discussion.

More information on Tech-Ed EMEA can be found at http://www.microsoft.com/europe/teched/ Tech-Ed is a good venue for any and all technology professionals interested in exploring a broad set of current and upcoming Microsoft technologies, tools, platforms, and services.

Whether you can make it to these sessions or not, I’m looking forward to seeing you in Berlin. Feel free to tap me on the shoulder and say hello.

Catch you again shortly,

John Dwyer,

International Data Centre Manager

Global Foundation Services, Microsoft

Designing Generation 4.0 Data Centers: The Engineers’ Approach to Solving Business Challenges…continued

msdcblog — Mon, 04 May 2009 10:01:00 GMT

Part # 2

(A couple of years ago, when our Data Center Services’ Research & Engineering team within Microsoft’s Global Foundation Services (GFS) group kicked off the Generation 4.0 Data Center design project, we began with the question of: What are the primary business challenges facing data center deployments today? ) Here’s the rest of the story….

Question Everything

Often the application of technology involves as much innovation as the technology itself. Rather than reinvent the wheel we looked at our industry’s journey thus far. We started by questioning everything up to and including the roof and the very definition of a data center. With a lot of prior art in modularization, some of which has already been applied to the IT industry, we saw a good fit. The military has been deploying portable ground stations with IT servers and communications equipment for decades. And for some time now telecom companies have deployed pre-manufactured buildings which are then assembled on site as central offices.

As part of our design project we met to discuss the modular solution space with a group of folks from our Infrastructure Services team within GFS , including members of our hardware, data center operations, development, engineering, risk management, and security teams, as well as our internal product groups. (The original Gen 4.0 team from the two-day session is 100 percent intact and still working at Microsoft, by the way.) We knew we could modularize the server room, which we now call Server PACs. The challenge became how to modularize the entire facility. So we created other PACs: Generator PACS, Medium Voltage Switchgear PACs, UPS PACs, etc. Next, we developed the system electrical one-line diagrams and mechanical schematics for our four data center classes.

Then we had to take these single lines and schematics and break them into logical modules for the components to reside in. This may seem easy but represents a shift in thinking from a building where, for instance, we would have a UPS room and associated equipment and switchgear manufactured by multiple vendors and put it physically in sometimes separate modules. The challenge became how to shift from a traditional construction mindset to the new, modularized manufacturing mindset. Maintainability is a large part of reliability in a facility, and became a key differentiator between the four classes. Our A Class infrastructure, which is not concurrently maintainable and is on basically street power and unconditioned air, will require scheduled downtime for maintenance. The cost, efficiency, and time-to-market targets for A Class are very aggressive and a fraction of what the industry has come to see as normal today. We realized that standardization and reuse of components from one class to the next was a key to improving cost and efficiency. Our premise was that the same kit of parts (or modules) should be usable from class to class. These modules (in this new mindset) can be added to other modules to transition within the data center from one class to the next.

Standardize to Differentiate

With the standardization of a kit of modular parts it is possible to supply different types of facilities—large-scale, mega-data centers and edge or mini-data centers—from the same supply chain. A pre-manufactured set of solutions can thus reduce costs through economy of scale. Traditionally, because of network and site service costs it has been more cost effective to deploy 40 megawatts of capacity in a single location than to build 40 individual 1-megawatt facilities. In a pre-manufactured model that isn’t necessarily the case. The use of a production-line approach, while at the same time simplifying interconnections through the right modularization of the components, can drive costs down for mini- as well as mega-data center facilities. Of course the fixed costs of site development must be considered, but we are finding that the production-line approach is significantly narrowing the difference in cost per megawatt for mega- and mini-data centers.

We realize that innovation will come from many sources, and so we are partnering with technology leaders in the vendor community, as well as our customers, to help drive and deliver the benefits of this new approach. We believe it is crucially important to develop this technology now due to the many constraints on our environment and economy globally. However, we do not have unlimited resources to drive this alone. Through industry partnership and collaboration we will be able to accelerate the adoption of modular data centers and deliver the benefits of technology, software, and services to more people.

Imagine the possibilities of bringing affordable computing to corners of the world that have no power or IT grid to speak of today. This could be done using modular power plants such as fuel cells and modular data centers in partnership with wireless technology where there is a lack of existing infrastructure (for example). Applications could become virtual in the local modular data center, or the cloud, and the thousands of services and applications it holds could be provided via low-cost devices. Great opportunities like being able to provide a $100 or a lot less laptop (or new access devices not yet invented) per child could then be achieved sooner. A modular data center could quickly be deployed to support millions of virtual applications and at the same time access the power of the Internet and cloud services for a large population that does not have that opportunity today. Personally, I am very excited about these possibilities and believe I am fortunate to be working on this technology at this time.

We look forward to continued collaboration with our industry participants.

You can read part 1 of this 2 part series in my earlier blog on April 29, 2009

/dc

Daniel Costello, director of Data Center Services

Global Foundation Services

Daniel Costello is the director for Data Center Services at Microsoft, responsible for data center research and engineering, standards and technologies, data center technology roadmap, Generation 4 data center engineering, data center automation and integration with IT hardware, operating systems and applications. Daniel works closely with Microsoft Research on proof of concepts in support of the data center of the future and manages a team of facility engineers and service architects.

Designing Generation 4.0 Data Centers: The Engineers’ Approach to Solving Business Challenges

msdcblog — Wed, 29 Apr 2009 09:00:00 GMT

Part 1:

A couple of years ago, when our Data Center Services’ Research & Engineering team within Microsoft’s Global Foundation Services (GFS) group kicked off the Generation 4.0 Data Center design project, we began with the question of: What are the primary business challenges facing data center deployments today?

Challenge #1: Time to Market

We kept coming back to a couple of leading issues. The first focused on time to market and meeting a variable demand profile. The issue is simple: it takes months to years to build data centers, but sometimes businesses need to move faster.

This issue isn’t entirely new, of course. When we designed our Chicago facility we made significant gains in this area by devoting more than half the facility to housing containers that served as modularized server rooms, or Pre-Assembled Components (PACs). Server PACs dramatically reduce time to market because they are assembled at the same time as the site infrastructure and building for the data center. When the site preparation is complete all we have to do is roll the containers into the new facility, connect a few cables, and we’re up and running.

This advance was the heart of our Generation 3 data center. While we saw it as a great step forward, it almost immediately led us to ask ourselves whether we could take the gains even further by modularizing the entire facility. That led us to the crux of Generation 4, where we created further PACs: Generator PACS, Medium Voltage Switchgear PACs, UPS PACs, etc. By moving to PACs for all these key data center systems, we created a design where almost everything we need to add new capacity can be pre-assembled in parallel and then brought together in a matter of weeks. The fact that the components all come in their own modular containers eliminated the need for much of the on-site construction—which can be the most time-consuming, expensive, and environmentally unfriendly element of building a data center.

Perhaps most importantly, with Generation 4 we can quickly add capacity incrementally in response to demand. Gone are the days when we had to wait 12-18 months for a large data center to be built, only to use a small portion of its capacity while we waited for demand to catch up to capacity. In short, our Generation 4 design delivers a revolution in terms of time to market that the data center industry has never seen before.

Challenge #2: Cost

The next business challenge (after time to market) is cost. We looked at several areas impacting cost, capital efficiency, and return on invested capital (ROIC), which affects cash flow and is calculated using Net Present Value (NPV). Capital outlay itself is now widely measured by our industry in dollar value ($) per watt, versus square foot.

Operational costs that impact Cost of Goods Sold (COGS) are measured in $ per kilowatt per month and are impacted largely by depreciation, energy costs, and operations staffing. A couple of years ago we moved to a chargeback model for power versus real estate, and as Christian Belady pointed out in his blog, that incentive-based approach proved effective in turning the corner on power usage and costs.

To tackle other costs in the data center, the use of server PACs is proving equally effective. Using this approach we project our Chicago facility will deliver cost savings of approximately 30 percent and enable a more efficient cash flow because we will not build out the modularized server PACS until they are required. That’s just the beginning of the cost benefits as we move to Generation 4 and fully modularize the data center. The traditional raised floor is not where the majority of the money or lead time is spent. Instead, it is in the electrical and mechanical systems. Moving to PACs in these areas will reduce costs and free large amounts of capital previously required to construct huge facilities that we might not fully utilize for several years.

Challenge #3: Efficiency

The discussion on the impact of energy costs led us to the next business challenge: efficiency. Efficiency has been called the “fifth fuel” and is regarded as a source of energy in itself. Today the industry is beginning to use Power Usage Effectiveness (PUE) as the recognized metric for data center efficiency. If we take a look at the larger picture beyond just operational power consumption, the Total Cost of Energy (TCOE) also needs to be considered to address the full lifecycle of the data center–from component manufacturer to transportation to construction and on-site assembly, and even end of life.

When we considered the goal of making our data centers more energy efficient, our team’s debate focused on the inefficiency of using redundant hardware systems to provide backup and failover capabilities. Previously many data centers have been built with the same level of reliability to the highest common denominator. Our Generation 4 team no longer believes that hardware redundancy is the best way to ensure service reliability. Therefore, we began with a fresh approach to this industry problem by looking at the latest technology capabilities and then establishing multiple classes of service. Each class was assigned a differentiated chargeback model that would encourage our properties to move to the lowest cost and most efficient service level that meets their business needs. To support this we are developing software that moves reliability across all applications higher up the Open Systems Interconnect (OSI) model all the way to the operating system and application layer. The four classes of data center service we have created are still being researched and we may consolidate them over time. Regardless, our goal around software-based reliability matched to hardware levels of reliability is driving the right discussions.

Challenge #4: Flexibility and Density

The last major challenge we identified was to enable data centers to be flexible and host multiple form factors and levels of density. With traditionally built facilities, the density of the data center is normally set during design. That density then remains unchanged for 15 years or longer during the facility’s lifecycle. Measured in watts per square foot, density can lead to capacity planning challenges. Build with too low a density and the data center will be less energy efficient and take up more real estate than is necessary, which can have a big impact when land is expensive. Build too high a density and you can strand power and cooling, which is where the majority of the costs are.

The Goals our Engineering Team Set

· Reduce time-to-market and deliver the facility at the same time as the computing infrastructure

· Reduce capital cost per megawatt and reduce COGS per kilowatt per month by class

· Increase ROIC and minimize the up-front investment for data centers

· Differentiate reliability and redundancy by data center class and design the system to be flexible to accommodate any class of service in the same facility

· Drive data center efficiency up while lowering PUE, water usage, and overall TCOE

· Develop a solution to accept multiple levels of density and form factors, such as racks, skids, or containers

Next week, I will talk about the process we used to develop these ideas.

/dc

Daniel Costello, director of Data Center Services

Global Foundation Services

Daniel Costello is the director for Data Center Services at Microsoft, responsible for data center research and engineering, standards and technologies, data center technology roadmap, Generation 4 data center engineering, data center automation and integration with IT hardware, operating systems and applications. Daniel also works closely with Microsoft Research on proof of concepts in support of the data center of the future and manages a team of facility engineers and service architects.

Interested in learning more about Green IT? Check out Microsoft’s Webcast Series on TechNet!

The series will feature a number of well known speakers, including Christian Belady and Daniel Costello. The full session listing is provided below:

The series will feature a number of well known speakers, including Christian Belady and Daniel Costello.

The full session listing is provided below:

Title	Date	Time (PST)	URL
Transforming the Data Center with Energy Efficiency (Level 200)	5/1/2009	11:00 AM	http://bit.ly/12CFzf
Hyper-Green Virtualization: Scaling Enterprise IT for Energy Efficiency (Level 200)	5/5/2009	10:00 AM	http://bit.ly/TFwxV
Cloud Computing Futures: Creating Greener Clouds with Microsoft Research (Level 200)	5/6/2009	9:00 AM	http://bit.ly/18hsgU
Improving Energy Efficiency w/ Windows 7 Power Management (Level 200)	5/8/2009	10:00 AM	http://bit.ly/fplU

Microsoft’s Top 10 Business Practices for Environmentally Sustainable Data Centers (celebrating Earth Day)

msdcblog — Tue, 21 Apr 2009 10:00:00 GMT

How to Reduce Energy Consumption, Waste, and Costs while Increasing Efficiency and ROI

What does environmental sustainability mean to data centers, industry operators, and IT businesses? A lot of managers in these areas may barely notice the Earth Day global event on April 22 as they struggle to support their businesses in the face of budget cuts and uncertainty about the future. But the fact is, being “lean and green” is good for both the business and the environment, and organizations that focus their attentions accordingly will see clear benefits. Reducing energy use and waste improves a company’s bottom line and boosting the use of recycled materials is a proven way to demonstrate good corporate citizenship to your customers, employees, and the communities you do business in.

That said, with so much on the plates of data center and IT professionals these days it isn’t always easy to know where to begin in moving to greener and more efficient operations. With that in mind—along with Microsoft’s commitment to share best practices with the rest of the data center industry—we asked some of the senior members our Global Foundation Services’ Infrastructure Services team to send us their top ten best business practices for environmentally sustainable data centers and IT.

Their favorite practices are reflected in our new “Microsoft’s Top 10 Business Practices for Environmentally Sustainable Data Centers” paper.

You can explore other data center resources on our GFS web site here.

Microsoft announced in March 2009 that it is taking a proactive corporate approach to reduce our carbon emissions per unit of revenue by at least 30 percent below 2007 levels by 2012. The data centers managed by GFS are a significant component of Microsoft’s carbon footprint, and our organization plays a vital role in Microsoft’s efforts to meet this corporate goal.

As you’ll read in the list of best practices we’ve compiled, companies can make major gains in energy efficiency by increasing server utilization and moving to virtualization, plus you can employ a wide range of smaller initiatives that collectively add up to significant gains. Microsoft has been using these practices for several years now and found that in addition to helping us protect the environment, they make best use of our resources and help us stay tightly aligned with our core strategies and goals. We hope they work for you as well.

Below is an overview of the best practices list.

1. Provide incentives that support your primary goals.

2. Focus on effective resource utilization.

3. Use virtualization to improve server utilization and increase operational efficiency.

4. Drive quality up through compliance.

5. Embrace change management.

6. Invest in understanding your application workload and behavior.

7. Right-size your server platforms to meet your application requirements.

8. Evaluate and test servers for performance, power, and total cost of ownership.

9. Converge on as small a number of stock-keeping units (SKUs) as you can.

10. Take advantage of competitive bids from multiple vendors to foster innovation and reduce costs.

Beyond the business practices listed above, Microsoft’s data center team is taking significant steps in four areas important to environmental sustainability:

· Using recycled resources whenever practical: The Microsoft data center in San Antonio, Texas, for example, uses approximately eight million gallons of recycled water a month from the city’s waste water system during peak cooling months.

· Using renewable resources whenever available: In the process we have developed for deciding where to build our data center facilities, renewable energy sources play a key role. For instance, the Microsoft data center in Quincy, Washington, uses 100 percent renewable hydropower from the Columbia Basin River. The San Antonio facility obtains part of its electricity from wind power. And the company’s Dublin, Ireland, data center will use outside air for cooling, thus reducing the need for energy-intensive coolers.

· Reducing waste in operations: One example of Microsoft’s focus on reducing waste is the company’s transition to using standard shipping containers to house thousands of servers apiece. Ordering servers by the truckload eliminates the need for large amounts of packaging and other materials previously required when servers were delivered individually or in racks.

· Actively participating in industry environmental groups: Microsoft plays leadership roles in Climate Savers Computing Initiative and The Green Grid—industry organizations focused on improving computer systems and data center energy efficiency and establishing a standard methodology for measuring Power Usage Effectiveness (PUE) accurately and consistently. We also feel it’s important for us to openly share information and best practices around data center energy efficiency, because we believe the data center industry needs to work together to make dramatic gains toward environmental sustainability.

The teams that contributed to this white paper are represented by our power and cooling architect Christian Belady, our distinguished engineer Dileep Bhandarkar, our Data Center Services director Daniel Costello (“the father” of our Generation 4 modular data center vision and the leader of the team of engineers that created it), our Data Center Software Services general manager Jawaid Ekram, and myself.

Their backgrounds include expertise in data center electrical and mechanical engineering, power and cooling architecture and design, research and development, and business operations and administration. It has been my distinct pleasure to work with and lead these subject matter experts and to share their thoughts with the industry via our blogs and participation in industry events.

Arne

Arne Josefsberg,

General Manager of Infrastructure Services

Global Foundation Services

What’s the Upside to a Downturn? Recessions Heighten Focus on Efficiency!

msdcblog — Wed, 04 Feb 2009 21:01:00 GMT

By Christian Belady

What’s the upside to a downturn? As the economy shrinks and budget and program cutting expands throughout our industry, how can data center facility and IT infrastructure teams identify opportunities that create value for their companies? In other words, how do we emerge stronger from a recession? For most business services professionals (people in any organization for that matter), acknowledging inefficiencies is the first step towards taking action, and with action comes insight, evidence, and real options. Like any well-managed business, we routinely check our assumptions and planning needs against our assessment of the economic environment. As part of this process, we look at many scenarios and options to improve efficiencies, reduce costs, and increase the return on our investments.

We have spoken at many conferences in the past couple of years and shared our experiences and ideas on how this industry could collectively improve to become more efficient, in part by reducing our carbon footprint. The uncertainty presented in our industry today – regarding the environment, the economy, and how people react to change – reconfirms the importance of developing a data center strategy that can react to unpredictable and dynamic business needs. In this blog we would like to share some of our experiences over the past couple years and the lessons we’ve learned about how to measure and increase efficiency.

We’d also like to turn up the dial on our invitation to the rest of the industry to work together with us in these areas. An important example involves our belief that data centers could operate at much higher temperatures, which would end much of the need for expensive, energy inefficient cooling equipment. To that end we urge all companies to encourage ASHRAE to further open its recommended operating temperature ranges beyond the adjustments they recently made. Additionally, if we all work towards recommending improvements to hardware specifications, server manufacturers could increase the efficiency of their products. With so much at stake--both in helping our companies through difficult financial times and in protecting our global environment--it’s time to work together. We should be striving to run more efficiently as an industry, not just as separate corporations.

Sharing Microsoft’s Energy Efficiency Best Practices for Data Center Operations

So what are some of the elements in our efficiency program that have reduced our needs? Here are some examples:

1) To begin with, we applied our own best practices on energy efficiency that we published about a year ago. These were all things that we felt were pretty important for foundational improvement in our data center operations efficiency. They included:

1. Engineer the data center for cost and energy efficiency

2. Optimize the design to assess multiple factors

3. Optimize provisioning for maximum efficiency and productivity

4. Monitor and control data center performance in real time

5. Make data center operational excellence part of organizational culture

6. Measure power usage effectiveness (PUE)

7. Use temperature control and airflow distribution

8. Eliminate the mixing of hot and cold air

9. Use effective air-side or water-side economizers

10. Share and learn from industry partners

2) We also invested in a team led by computer industry veteran Dr. Dileep Bhandarkar (a distinguished engineer in our organization), to optimize our server performance, cost, and power. These technologists are turning over every stone to squeeze efficiency out of our hardware. The focus has been on working with hardware suppliers to increase the efficiency of power supplies and right-size the system, including the processor, memory, and storage. More importantly, the team has focused on selecting hardware based not only on total cost of ownership but also on the best performance per watt for the particular application. As a result we have seen a 13% drop in average server power in operations, driven by an even steeper drop in power for the servers we added in the past year. During that same period, interestingly, ASHRAE power curves actually showed an increase in server power by 4%.

3) But perhaps the overarching thing that drove the above elements was when we changed our internal chargeback model about 1.5 years ago from charging for space to charging for power allocated/consumed. We presented on this for the first time in July 2008 to the EPA. This proved to help reduce energy and also reduce carbon emissions. We later blogged about this and how efficiency is not only a technology problem but perhaps even more a behavior problem. If you are interested in more details on how we do our chargeback models, visit our Power of Software blog on this topic.

4) Also, it’s in our DNA to use software—especially Microsoft software—to achieve our goals. Virtualization technology such as Hyper-V and our new Windows Azure platform in Windows Server is being utilized to reduce energy use, and using Microsoft System Center management products to improve efficiency throughout our operations. The purpose of this blog isn’t to promote these products, and we realize many data centers are built on other platforms. But the point is that smart use of software can give you a lot more bang for your hardware buck, and at the same time help you make significant gains in reducing your carbon footprint.

So our key takeaways from this experience are:

1) All of the best practices and technology we are pursuing help improve efficiency and reduce carbon footprint, but only if adoption occurs. As we have seen within the industry, adoption has been slow. However, with the right incentives, adoption can and will happen.

2) Our chargeback models provided incentives that improved efficiency far beyond our expectations and drove our total mega watt needs for our operations much lower than expected.

3) New software solutions are enabling data center teams to make dramatic efficiency improvements, primarily in the areas of virtualization, automation, and streamlined management functions. Moving to new software may be a tough sell in this economic climate, but the right products will make the purchases well worth the investment.

We feel it’s important for us to openly share information and best practices around energy efficiency because we believe the data center industry as a whole needs to work together in order to make the dramatic gains needed to make a difference for our companies and the planet. We invite other data center teams to explore our best practices and approaches, enhance them with your own, and apply them in your situation as you see fit.

We certainly don’t pretend to have all the answers, and we’re very interested in considering approaches that have worked for other companies, or ideas that may bear fruit in the future. The data center industry is an exciting place to work right now because there is so much focus on what we do and so much opportunity for our work to make the world a better place—whether that’s by reducing our carbon footprint and the amount of water we use, or by enabling further advances in online services that take our societies to new levels of collaboration and opportunities for all.

Please join us in sharing information about what your company or team is doing to increase efficiency, or comment on the opinions we’ve expressed here, so that we can engage in a dialogue that benefits us all.

Thank you, in advance, for helping to advance the industry as a whole.

Also, be sure to visit our team’s blog in a couple of weeks when Daniel Costello, director of Research & Engineering, will be sharing more information on how we are advancing our efficiency best practices within our Generation 4 Modular Data Center plans.

/cb

Christian Belady, power and cooling architect, Global Foundation Services, Microsoft

Microsoft's Generation 4 Data Center Vision - the Architects' Perspective

msdcblog — Mon, 08 Dec 2008 09:44:00 GMT

Microsoft's Generation 4 Data Center Vision - the Architects' Perspective

By David Gauthier, Data Center Infrastructure Architect and Christian Belady, Principal Power and Cooling Architect, Microsoft Corp.

On Tuesday, December 2, our Global Foundation Services team went public with our Generation 4 Modular Data Center Vision and over the past week a lot of great discussions and questions have been posed from our industry colleagues. Today, we wanted to address some of those questions and share more insight on our Gen 4 plan via a video interview we did with Adam Bomb, a Technical Evangelist at Microsoft's TechNet Edge.

Some people got the impression that this announcement was solely about a containerized server room rather than a re-thinking of the entire infrastructure. The goal of Gen 4 is to modularize not only the server and storage components, which a number of companies are already doing, but also to modularize the infrastructure, namely the electrical and mechanical systems. The real innovation is around the commonality, manufacturing, supply chain and integration of these modules to provide a plug-and-play infrastructure along with modularized server environments. In addition, it is focused on scaling the infrastructure with the business demands, smoothing capital investment, and driving costs down as shown by the following chart.

While we expect these modular innovations to reduce capital investments by 20%-40% or more depending on class, we also expect considerable reductions in operating expenses related to electricity and water consumption. Designing from the start for environmental sustainability has allowed us to focus on using less construction material up front, less energy and water during operation, and also allows us to recycle and reuse components at the end of their useful life. No longer will we be governed by the initial decisions made when constructing the facility. We will have almost unlimited use and re-use of the facility and site. We will also be able to use power in an ultra-fluid fashion moving load from critical to non-critical as use and capacity requirements dictate.

More importantly, any IT or infrastructure vendor's products that meet our specs can be used interchangeably, thus providing a means for global sourcing and a competitive and innovative supplier landscape. As our team has already discussed pretty thoroughly, Microsoft is already reaping some of the benefits of modularity with server containers and we are confident that further modularization will afford us additional benefits when applied to the 'back of house'.

Some people have questioned whether an open air facility is as safe against natural disasters as a traditional concrete and steel structure. We believe that it can be and we are currently taking these factors into account as part of our heat map criteria site selection activities. Of course, we also expect that our modular approach affords tremendous flexibility in addressing site specific conditions as needed. Another significant benefit modularity offers is a smaller system failure zone versus a traditional data center. For example, a fire in a data center could bring the whole facility down in traditional construction (or have the fire department activate the emergency power off). Alternatively, in a modularized data center the fire could actually be isolated to only a subset of modules and thus provide greater resiliency.

We recognize that these types of architectures may not work for every application or every data center provider out there. We're not saying this will be the right way or the wrong way, just our way - and it may not be for everyone. We are confident about this direction meeting the needs of our data center environment and we hope that it sets the stage for continued healthy and dynamic dialog and sharing in this industry.

For more background information on our modular data center vision, please visit the December 2^nd blog of "Our Vision for Generation 4 Modular Data Centers - One way of getting it just right...," as well as the Gen 4 video posted at Soapbox.

cb/dg

Our Vision for Generation 4 Modular Data Centers – One way of Getting it just right . . .

msdcblog — Tue, 02 Dec 2008 05:13:00 GMT

The following posting originally appeared on Michael Manos' Loose Bolts blog.

Data Centers are a hot topic these days. No matter where you look, this once obscure aspect of infrastructure is getting a lot of attention. For years, there have been cost pressures on IT operations and this, when the need for modern capacity is greater than ever, has thrust data centers into the spotlight. Server and rack density continues to rise, placing DC professionals and businesses in tighter and tougher situations while they struggle to manage their IT environments. And now hyper-scale cloud infrastructure is taking traditional technologies to limits never explored before and focusing the imagination of the IT industry on new possibilities.

At Microsoft, we have focused a lot of thought and research around how to best operate and maintain our global infrastructure and we want to share those learnings. While obviously there are some aspects that we keep to ourselves, we have shared how we operate facilities daily, our technologies and methodologies, and, most importantly, how we monitor and manage our facilities. Whether it’s speaking at industry events, inviting customers to our “Microsoft data center conferences” held in our data centers, or through other media like blogging and white papers, we believe sharing best practices is paramount and will drive the industry forward. So in that vein, we have some interesting news to share.

Today we are sharing our Generation 4 Modular Data Center plan. This is our vision and will be the foundation of our cloud data center infrastructure in the next five years. We believe it is one of the most revolutionary changes to happen to data centers in the last 30 years. Joining me, in writing this blog are Daniel Costello, my director of Data Center Research and Engineering and Christian Belady, principal power and cooling architect. I feel their voices will add significant value to driving understanding around the many benefits included in this new design paradigm.

Our “Gen 4” modular data centers will take the flexibility of containerized servers—like those in our Chicago data center—and apply it across the entire facility. So what do we mean by modular? Think of it like “building blocks”, where the data center will be composed of modular units of prefabricated mechanical, electrical, security components, etc., in addition to containerized servers.

Was there a key driver for the Generation 4 Data Center?

If we were to summarize the promise of our Gen 4 design into a single sentence it would be something like this: “A highly modular, scalable, efficient, just-in-time data center capacity program that can be delivered anywhere in the world very quickly and cheaply, while allowing for continued growth as required.” Sounds too good to be true, doesn’t it? Well, keep in mind that these concepts have been in initial development and prototyping for over a year and are based on cumulative knowledge of previous facility generations and the advances we have made since we began our investments in earnest on this new design.

One of the biggest challenges we’ve had at Microsoft is something Mike likes to call the ‘Goldilock’s Problem’. In a nutshell, the problem can be stated as:

The worst thing we can do in delivering facilities for the business is not have enough capacity online, thus limiting the growth of our products and services.

The second worst thing we can do in delivering facilities for the business is to have too much capacity online.

This has led to a focus on smart, intelligent growth for the business — refining our overall demand picture. It can’t be too hot. It can’t be too cold. It has to be ‘Just Right!’ The capital dollars of investment are too large to make without long term planning. As we struggled to master these interesting challenges, we had to ensure that our technological plan also included solutions for the business and operational challenges we faced as well.

So let’s take a high level look at our Generation 4 design

Are you ready for some great visuals? Click here for the Microsoft 4th Gen Video. It’s a concept video that came out of my Data Center Research and Engineering team, under Daniel Costello, that will give you a view into what we think is the future.

From a configuration, construct-ability and time to market perspective, our primary goals and objectives are to modularize the whole data center. Not just the server side (like the Chicago facility), but the mechanical and electrical space as well. This means using the same kind of parts in pre-manufactured modules, the ability to use containers, skids, or rack-based deployments and the ability to tailor the Redundancy and Reliability requirements to the application at a very specific level.

Our goals from a cost perspective were simple in concept but tough to deliver. First and foremost, we had to reduce the capital cost per critical Mega Watt by the class of use. Some applications can run with N-level redundancy in the infrastructure, others require a little more infrastructure for support. These different classes of infrastructure requirements meant that optimizing for all cost classes was paramount. At Microsoft, we are not a one trick pony and have many Online products and services (240+) that require different levels of operational support. We understand that and ensured that we addressed it in our design which will allow us to reduce capital costs by 20%-40% or greater depending upon class.

For example, non-critical or geo redundant applications have low hardware reliability requirements on a location basis. As a result, Gen 4 can be configured to provide stripped down, low-cost infrastructure with little or no redundancy and/or temperature control. Let’s say an Online service team decides that due to the dramatically lower cost, they will simply use uncontrolled outside air with temperatures ranging 10-35 C and 20-80% RH. The reality is we are already spec-ing this for all of our servers today and working with server vendors to broaden that range even further as Gen 4 becomes a reality. For this class of infrastructure, we eliminate generators, chillers, UPSs, and possibly lower costs relative to traditional infrastructure.

Applications that demand higher level of redundancy or temperature control will use configurations of Gen 4 to meet those needs, however, they will also cost more (but still less than traditional data centers). We see this cost difference driving engineering behavioral change in that we predict more applications will drive towards Geo redundancy to lower costs.

Another cool thing about Gen 4 is that it allows us to deploy capacity when our demand dictates it. Once finalized, we will no longer need to make large upfront investments. Imagine driving capital costs more closely in-line with actual demand, thus greatly reducing time-to-market and adding the capacity Online inherent in the design. Also reduced is the amount of construction labor required to put these “building blocks” together. Since the entire platform requires pre-manufacture of its core components, on-site construction costs are lowered. This allows us to maximize our return on invested capital.

In our design process, we questioned everything. You may notice there is no roof and some might be uncomfortable with this. We explored the need of one and throughout our research we got some surprising (positive) results that showed one wasn’t needed.

In short, we are striving to bring Henry Ford’s Model T factory to the data center. http://en.wikipedia.org/wiki/Henry_Ford#Model_T. Gen 4 will move data centers from a custom design and build model to a commoditized manufacturing approach. We intend to have our components built in factories and then assemble them in one location (the data center site) very quickly. Think about how a computer, car or plane is built today. Components are manufactured by different companies all over the world to a predefined spec and then integrated in one location based on demands and feature requirements. And just like Henry Ford’s assembly line drove the cost of building and the time-to-market down dramatically for the automobile industry, we expect Gen 4 to do the same for data centers. Everything will be pre-manufactured and assembled on the pad.

And did we mention that this platform will be, overall, incredibly energy efficient? From a total energy perspective not only will we have remarkable PUE values, but the total cost of energy going into the facility will be greatly reduced as well. How much energy goes into making concrete? Will we need as much of it? How much energy goes into the fuel of the construction vehicles? This will also be greatly reduced! A key driver is our goal to achieve an average PUE at or below 1.125 by 2012 across our data centers. More than that, we are on a mission to reduce the overall amount of copper and water used in these facilities. We believe these will be the next areas of industry attention when and if the energy problem is solved. So we are asking today…“how can we build a data center with less building”?

We have talked openly and publicly about building chiller-less data centers and running our facilities using aggressive outside economization. Our sincerest hope is that Gen 4 will completely eliminate the use of water. Today’s data centers use massive amounts of water and we see water as the next scarce resource and have decided to take a proactive stance on making water conservation part of our plan.

By sharing this with the industry, we believe everyone can benefit from our methodology. While this concept and approach may be intimidating (or downright frightening) to some in the industry, disclosure ultimately is better for all of us.

Gen 4 design (even more than just containers), could reduce the ‘religious’ debates in our industry. With the central spine infrastructure in place, containers or pre-manufactured server halls can be either AC or DC, air-side economized or water-side economized, or not economized at all (though the sanity of that might be questioned). Gen 4 will allow us to decommission, repair and upgrade quickly because everything is modular. No longer will we be governed by the initial decisions made when constructing the facility. We will have almost unlimited use and re-use of the facility and site. We will also be able to use power in an ultra-fluid fashion moving load from critical to non-critical as use and capacity requirements dictate.

Finally, we believe this is a big game changer. Gen 4 will provide a standard platform that our industry can innovate around. For example, all modules in our Gen 4 will have common interfaces clearly defined by our specs and any vendor that meets these specifications will be able to plug into our infrastructure. Whether you are a computer vendor, UPS vendor, generator vendor, etc., you will be able to plug and play into our infrastructure. This means we can also source anyone, anywhere on the globe to minimize costs and maximize performance. We want to help motivate the industry to further innovate—with innovations from which everyone can reap the benefits.

To summarize, the key characteristics of our Generation 4 data centers are:

Scalable
Plug-and-play spine infrastructure
Factory pre-assembled: Pre-Assembled Containers (PACs) & Pre-Manufactured Buildings (PMBs)
Rapid deployment
De-mountable
Reduce TTM
Reduced construction
Sustainable measures
Map applications to DC Class

We hope you join us on this incredible journey of change and innovation!

Long hours of research and engineering time are invested into this process. There are still some long days and nights ahead, but the vision is clear. Rest assured however, that we as refine Generation 4, the team will soon be looking to Generation 5 (even if it is a bit farther out). There is always room to get better.

So if you happen to come across Goldilocks in the forest, and you are curious as to why she is smiling you will know that she feels very good about getting very close to ‘JUST RIGHT’.

Generations of Evolution – some background on our data center designs

We thought you might be interested in understanding what happened in the first three generations of our data center designs. When Ray Ozzie wrote his Software plus Services memo it posed a very interesting challenge to us. The winds of change were at ‘tornado’ proportions. That “plus Services” tag had some significant (and unstated) challenges inherent to it. The first was that Microsoft was going to evolve even further into an operations company. While we had been running large scale Internet services since 1995, this development lead us to an entirely new level. Additionally, these “services” would span across both Internet and Enterprise businesses. To those of you who have to operate “stuff”, you know that these are two very different worlds in operational models and challenges. It also meant that, to achieve the same level of reliability and performance required our infrastructure was going to have to scale globally and in a significant way.

It was that intense atmosphere of change that we first started re-evaluating data center technology and processes in general and our ideas began to reach farther than what was accepted by the industry at large. This was the era of Generation 1. As we look at where most of the world’s data centers are today (and where our facilities were), it represented all the known learning and design requirements that had been in place since IBM built the first purpose-built computer room. These facilities focused more around uptime, reliability and redundancy. Big infrastructure was held accountable to solve all potential environmental shortfalls. This is where the majority of infrastructure in the industry still is today.

We soon realized that traditional data centers were quickly becoming outdated. They were not keeping up with the demands of what was happening technologically and environmentally. That’s when we kicked off our Generation 2 design. Gen 2 facilities started taking into account sustainability, energy efficiency, and really looking at the total cost of energy and operations. No longer did we view data centers just for the upfront capital costs, but we took a hard look at the facility over the course of its life. Our Quincy, Washington and San Antonio, Texas facilities are examples of our Gen 2 data centers where we explored and implemented new ways to lessen the impact on the environment. These facilities are considered two leading industry examples, based on their energy efficiency and ability to run and operate at new levels of scale and performance by leveraging clean hydro power (Quincy) and recycled waste water (San Antonio) to cool the facility during peak cooling months.

As we were delivering our Gen 2 facilities into steel and concrete, our Generation 3 facilities were rapidly driving the evolution of the program. The key concepts for our Gen 3 design are increased modularity and greater concentration around energy efficiency and scale. The Gen 3 facility will be best represented by the Chicago, Illinois facility currently under construction. This facility will seem very foreign compared to the traditional data center concepts most of the industry is comfortable with. In fact, if you ever sit around in our container hanger in Chicago it will look incredibly different from a traditional raised-floor data center. We anticipate this modularization will drive huge efficiencies in terms of cost and operations for our business. We will also introduce significant changes in the environmental systems used to run our facilities. These concepts and processes (where applicable) will help us gain even greater efficiencies in our existing footprint, allowing us to further maximize infrastructure investments.

This is definitely a journey, not a destination industry. In fact, our Generation 4 design has been under heavy engineering for viability and cost for over a year. While the demand of our commercial growth required us to make investments as we grew, we treated each step in the learning as a process for further innovation in data centers. The design for our future Gen 4 facilities enabled us to make visionary advances that addressed the challenges of building, running, and operating facilities all in one concerted effort.

/Mm/Dc/Cb