...building hybrid clouds that can support any device from anywhere
Contoso Labs Series - Table of Contents
Now that Cisco has been chosen as the vendor for our network, we need to identify the layers of our network fabric, and the devices to be used in those roles.
We'll have a 3-level network topology when done. Some devices will act as leaf node top-of-rack switches. More capable devices will be core+edge routers. Finally, we'll need an aggregator level to connect the two, and isolate storage traffic from the core. We'll detail the network configuration and the traffic shaping considerations that went into it at a later time. For now, let's just identify the devices we're using, and why they suit our needs.
For leaf top-of-rack switches, we'll be using the Cisco Nexus 3048. This device has good port density, (48x1GbE RJ-45 ports, 4x10GbE SFP+ ports) and supports all of the important functions needed, like OMI and port-channels. Given our node count, we’ll end up needing 32 of these total, so their relative affordability is another asset to us. A simple, solid choice for our purposes.
The spine aggregator switches will be Cisco Nexus 3064-X devices. These are much more capable 10GbE switches, of which we’ll need 8. Each has 48 SFP+ 10GbE ports, as well as four QSFP+ 40GbE ports. Extremely low latency and line-speed switching combined with Layer-3 routing allows us to create a very high speed spine/aggregator layer for our racks. This is critical in our overall architecture because we’re using a converged fabric design, where our Ethernet fabric has to carry all of our combined I/O. Isolating storage traffic off of the core will keep performance acceptable for everyone, and we need a high speed intermediate layer to pull that off.
The edge of our network will be served by two Cisco Nexus 6001 devices. While from the outside these would appear almost identical to the 3064-X’s, the network capabilities provided by the 6001 are much greater. Larger lookup and routing tables, and more sophisticated controls are available that make it better suited to sit at the center of a network that will be hosting 300+ physical nodes and thousands of virtual machines operating on NVGRE virtual networks.
That covers our purchasing of net new equipment for this project. Combined with our existing assets, we have everything we need to design and deploy our private cloud. Starting on Wednesday, we’ll start describing how we integrated these components, and what our deployment is going to look like.
Great information. I would enjoy it even more with a network diagram. :)
I decided to create an alternate design based on the requirements you specified. This design is also a spine-leaf CLOS architecture, however I decided to use Cisco Nexus 6001 with Cisco Nexus 2248 fabric extenders (FEX).
Spine = 2 x Cisco Nexus 6001
Leaf (includes border leaf) = (6 x Cisco Nexus 6001) + (32 x Cisco Nexus 2248)
Here are some of the benefits:
1) Lower cost -
Cisco sells a bundle (N6001P-8FEX-1G) that includes one 6001 with 8 x 2248 FEX, and it includes the SFP lasers to interconnect these devices, all for a street price of $64,000. So those cost (4 x $64,000) + (4 x $25,000) for more Cisco 6001 + (200 x $600) for
more SFP lasers. The total cost is in the ballpark of $476,000. (You could lower your costs more by placing your 6001s all in the same rack and using low-cost DAC cables versus SFP lasers).
My estimate for your design is (8 x $15,500) for Nexus 3064 + (32 x $7,500) for Nexus 3048 + (2 x 25,000) for Nexus 6001 + (272 x $600) for SFP lasers. The total cost would be around $577,000 (Note: I don't understand how you will use 8 x 3064. I would have
thought you only needed 4 since the 3048 only has 4 x 10 Gbps uplinks?)
2) Fewer devices to manage - The fabric extenders are configured via the parent 6001 switches.
3) Advanced features of Cisco Nexus 6001 are available to the entire data center.
Some people might be concerned that the Nexus 2248 doesn't perform local switching (all traffic is forwarded to 6001). This problem is minor if you consider that only 24 of 288 servers will be on the same switch. This means that local switching would only be
helpful for about 8% of the servers.
Here is a diagram of the design:
@Tristan We did consider going with a FEX-based design when we picked Cisco, but there were a few factors that tilted us in the other direction. One mundane one was cost. Microsoft's buying habits meant the prices on some Cisco parts are different than others, and very different from the typical street prices. That's not helpful to everyone, but this project is under a real budget so I needed to find efficiencies and savings where possible.Your alternate design has given me some new things to talk about though. I might move up the discussion of our network design to this week to keep this thread going.