Selecting the number of nodes for your Scale-Out File Server Cluster

Selecting the number of nodes for your Scale-Out File Server Cluster

  • Comments 21
  • Likes

I recently got a stream of e-mails and questions about the maximum number of cluster nodes you can have in a Scale-Out File Server cluster. For the record, we test and support up to 8 nodes per file server cluster. This is the case for both Windows Server 2012 (which introduced the Scale-Out File Server cluster feature) and Windows Server 2012 R2.

 

image

 

However, the real question is usually "How many file server nodes do I need for my Scale-Out File Server cluster?" The most common scenarios we see involve the deployment of 2 to 4 file server nodes per cluster, with just a few people considering 8 nodes. Here are some arguments for each cluster size:

  • 2 is the bare minimum for achieving continuous availability (transparent failover). We see a lot of these two-node clusters out there.
  • 3 is a good idea to allow you to still have continuous availability even when you're doing maintenance on a node.
  • 4 allows you to upgrade the cluster without extra hardware by evicting two nodes, installing the new operating system and using the Copy Cluster Roles Wizard.
  • 8 will allow you to combine the network throughput and computing power of the many nodes to create an amazing file sharing infrastructure.

On the topic of performance, keep in mind that the Scale-Out concept means that you have the ability to linearly scale the cluster by adding more nodes to achieve higher IOPS and throughput. We have proved it in our test labs. However, we have also shown that a single file server cluster node can deliver over 200,000 IOPs at 8KB each or over 2GB/sec throughput, with a fairly standard server configuration (two modern Intel CPUs with a few cores each, dual 10GbE RDMA network interfaces, two SAS port each with four lanes of 6Gbps SAS shared storage with an SSD tier). We have also shown high-end file server configurations achieving over a million IOPs and over 16.5GB/sec from a single node. So, there are only a few scenarios that would require anything close to 8 nodes solely to achieve specific performance goals.

Also, keep in mind that the reasons for adopting a Scale-Out File Server usually include a combination of availability, performance and scalability of the resulting file service. While the number of nodes is an important ingredient to achieve that, you should never overlook related configuration decisions like the number and model of CPU you're using, the number and type of network interfaces, the number and speed of the SAS ports, the proper deployment of tiered storage with the right class of SSDs and even the generation and number of lanes of the PCIe slots you use. Two well sized and properly fitted file server nodes can easily beat eight nodes that were poorly put together.

Here a few additional references for those looking to dig deeper:

I'm not sure what's driving this wave of questions about the Scale-Out File Server cluster limits, but I hope this helps clarify the topic. Have you ever encountered a scenario where you would need more than 8 nodes? We would love to hear it...

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • <p>Another important item to consider with SAS storage is whether the server supports PCI express 3.0</p> <p>you need PCIe 3.0 x8 to support the bandwidth a dual port SAS card like the LSI 9207-8e can generate</p>

  • <p>@Michael</p> <p>PCI Gen3 x8 goes up to about ~6.7 GB/sec, which can keep up with 2 ports of 6Gbps/4-lane SAS (total of 4.4 GB/sec from both ports combined). If using PCIe Gen2 x8 or PCIe Gen3 x4, the bus becomes a bottleneck.</p> <p>See tables at the end of <a rel="nofollow" target="_new" href="http://blogs.technet.com/b/josebda/archive/2013/02/04/hyper-v-over-smb-performance-considerations.aspx">blogs.technet.com/.../hyper-v-over-smb-performance-considerations.aspx</a> for further details on speeds and feeds...</p>

  • <p>Hey Jose,</p> <p>Great article!!</p> <p>From my point of view the amount of RAM is another important decision for each node since you can setup up to 80% of RAM exclusively for cache on CSV volume. </p> <p>A question for you: &nbsp;I guess is strongly recommended to have all nodes with the same hardware configuration to avoid unbalanced configuration, and surprises during fail-over process, am I right?</p>

  • <p>@Andre</p> <p>Yes, RAM for CSV caching is important and balanced nodes is always a good idea to make things more predictable.</p> <p>To be fair, whatever passes Failover Cluster Validation is supported, but I would keep them configured exactly the same, like monozygotic twins.</p>

  • <p>Dear Jose,</p> <p>I have a question out of the above scope please.</p> <p>You mentioned that always we need to choose the smaller number of the two tiers SSD+HDD for the NumberOfColumns.</p> <p>The question if I added more disks in the JBOD to expand the storage space in the future.</p> <p>The example we have 4 disks in the SSD tier and 8 disks in the HDD tier for a mirrored space.</p> <p>I will add 4 SSDs and 4 HDDs (Total= 8 disks in the SSD tier and 12 disks in the HDD tier).</p> <p>How this will reflect the NumberOfColumns?</p> <p>Before we expand the JBOD, the NumberOfColumns was set to 2 for the mirrored space (4 SSD/2).</p> <p>Do I need to run a new cmdlet below and modify the NumberOfColumns from 2 to 4?</p> <p>Get-StoragePool Pool1 | Set-ResiliencySetting -Name Mirror -NumberOfColumnsDefault 4</p> <p>I am setting Storage Tiering on a new HP server with 25 Bays mixed between HDD and SATA SSD.</p> <p>I noticed that the Storage Pool detect the media type for the SSD disks as SSD, but for the HDD is showing as UNKNOWN.</p> <p>Is it safe to SET the PhysicalDisk -MediaType = HDD in production and not for lab test?</p> <p>Is it recommended that the Server must detect the media type of the disks by default or we can set it in PowerShell if it showing Unknown?</p> <p>Thank you in advance.</p> <p>Regards.</p>

  • <p>Can I use Single Share for Scale-Out File Server with minimam Redirection of CSV Data/Metadata at Mirrored Storage Space ?</p> <p>In your article &quot;Automatic SMB Scale-Out Rebalancing in Windows Server 2012 R2&quot;,</p> <p>at section &quot;Hyper-V over SMB with Windows Server 2012 R2 and Mirrored Storage Spaces (asymmetric)&quot;</p> <p>&quot; In this new release,</p> <p> &nbsp;the SMB client gained the flexibility to connect to different Scale-Out File Server cluster nodes</p> <p> &nbsp;for each independent share that it needs to access&quot;</p> <p>AND, at section &quot;Key configuration requirements for asymmetric storage in Windows Server 2012 R2&quot;</p> <p>&quot; To be clear,</p> <p> &nbsp;if you have a single CSV in such asymmetric configuration </p> <p> &nbsp;in Windows Server 2012 R2 Scale-Out File Server cluster,</p> <p> &nbsp;only one node will be actively accessed by SMB clients&quot;</p> <p>I want to use Mirrored Stoage Space, and 4 Hyper-V Cluster node, 10 Guest OS per node,</p> <p>How many Share do I need ?</p> <p>Regards,</p> <p>I like Windows Server 2012 R2 Storage Space</p> <p>Yoshihiro Kawabata</p>

  • <p>@Yoshihiro </p> <p>If you use a two-node Scale-Out File Server with a single share backed by a mirrored storage space in Windows Server 2012 R2, all clients (Hyper-V hosts) running Windows Server 2012 R2 will be redirected to a single file server node, the owner node for that single CSV. The other file server node will be idle waiting for a potential failover.</p> <p>This is not about the number of Hyper-V nodes, but the number of file server nodes in the configuration. You mentioned you had 4 Hyper-V nodes. Assuming you have 2 file server cluster nodes, you should use at least 2 shares, each backed by a different CSV on a mirrored storage space. Each of the 4 Hyper-V nodes will connect to the file server node that owns the CSV for that share. If a single Hyper-V node is using both shares, that node will connect to both file server nodes and use the best path for each share.</p>

  • <p>@Sharbel</p> <p>It&#39;s safe to set the media type property if it&#39;s not automatically detected. Just make sure you&nbsp; set it to the correct value (HDD or SSD) and don&#39;t place any data on it before doing it.</p> <p>You can&#39;t change the number of columns for an existing space (the number of columns is a property of the storage space, not the storage pool). You can add disks to the pool to get more capacity and you can increase the size of the space (using Resize-VirtualDisk), even if it&#39;s tiered (use Resize-StorageTier). However, the number of columns has to remain the same, since it dictates how data is laid out on the disk.</p> <p>You can find the Storage Spaces FAQ at <a href="http://social.technet.microsoft.com/wiki/contents/articles/11382.storage-spaces-frequently-asked-questions-faq.aspx" target="_parent">http://social.technet.microsoft.com/wiki/contents/articles/11382.storage-spaces-frequently-asked-questions-faq.aspx</a></p>

  • <p>What a brilliant post, much appreciated Jose a tonne of information there :)</p>

  • <p>@Jose Barreto -- Thank you for clear response.</p> <p>I will continue to try Windows Server 2012 R2 Scale Out File Server.</p> <p>Regards,</p>

  • <p>What you&#39;re doing with &nbsp;SMB is refreshing. I look forward to seeing the move towards leveraging the physical desktop and I hope youre involved.</p>

  • <p>Thank you dear Jose for the clear response,</p> <p>I am having an issue to (Resize-StorageTier) and expand the SSD_Tier:</p> <p>I still have 7GB free on the SSD Tier:</p> <p><span style="font-family:courier new,courier;font-size:small;">PS C:\Get-StorageTierSupportedSize MirroredTieredSpace1_SSD_Tier -ResiliencySettingName Mirror | FT -AutoSize</span></p> <p><span style="font-family:courier new,courier;font-size:small;">SupportedSizes TierSizeMin TierSizeMax TierSizeDivisor</span></p> <p><span style="font-family:courier new,courier;font-size:small;">-------------- &nbsp; &nbsp; &nbsp; &nbsp;----------- &nbsp; ----------- ---------------</span></p> <p><span style="font-family:courier new,courier;font-size:small;">{} &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1073741824 &nbsp;7516192768 &nbsp; &nbsp; &nbsp;1073741824</span></p> <p><span style="font-family:courier new,courier;font-size:small;">When I try to expand the SSD tier using below cmdlet:</span></p> <p><span style="font-family:courier new,courier;font-size:small;">PS C:\Resize-StorageTier -InputObject (Get-StorageTier -FriendlyName &quot;MirroredTieredSpace1_SSD_Tier&quot;) -Size 3GB</span></p> <p><span style="font-family:courier new,courier;font-size:small;">Resize-StorageTier : Invalid Parameter</span></p> <p><span style="font-family:courier new,courier;font-size:small;">At line:1 char:1</span></p> <p><span style="font-family:courier new,courier;font-size:small;">+ Resize-StorageTier -InputObject (Get-StorageTier -FriendlyName &quot;MirroredTieredSp ...</span></p> <p><span style="font-family:courier new,courier;font-size:small;">+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</span></p> <p><span style="font-family:courier new,courier;font-size:small;">&nbsp; &nbsp;+ CategoryInfo &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;: InvalidArgument: (StorageWMI:ROOT/Microsoft/...SFT_StorageTier) [Resize-StorageTier], CimException</span></p> <p><span style="font-family:courier new,courier;font-size:small;">&nbsp; &nbsp;+ FullyQualifiedErrorId : StorageWMI 5,Resize-StorageTier</span></p> <p><span style="font-family:courier new,courier;font-size:small;">PS C:\Get-StorageTier *Space1*</span></p> <p><span style="font-family:courier new,courier;font-size:small;">ObjectId &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : {1}\\ISCKC-HV01\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StorageTier.ObjectId=&quot;{6fa45e22-498a-11e3-80b4-806e6f6e6963}:ST:{5c3a3723-4ab2-11e3-80bb-38eaa78c3d44}{5c3a37ae-4ab2-11e3-80bb-38eaa78c3d44}&quot;</span></p> <p><span style="font-family:courier new,courier;font-size:small;">PassThroughClass &nbsp; &nbsp; :</span></p> <p><span style="font-family:courier new,courier;font-size:small;">PassThroughIds &nbsp; &nbsp; &nbsp; :</span></p> <p><span style="font-family:courier new,courier;font-size:small;">PassThroughNamespace :</span></p> <p><span style="font-family:courier new,courier;font-size:small;">PassThroughServer &nbsp; &nbsp;:</span></p> <p><span style="font-family:courier new,courier;font-size:small;">UniqueId &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : {5c3a37ae-4ab2-11e3-80bb-38eaa78c3d44}</span></p> <p><span style="font-family:courier new,courier;font-size:small;">Description &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;:</span></p> <p><span style="font-family:courier new,courier;font-size:small;">FriendlyName &nbsp; &nbsp; &nbsp; &nbsp; : MirroredTieredSpace1_SSD_Tier</span></p> <p><span style="font-family:courier new,courier;font-size:small;">MediaType &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;: SSD</span></p> <p><span style="font-family:courier new,courier;font-size:small;">Size &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : 375809638400</span></p> <p><span style="font-family:courier new,courier;font-size:small;">PSComputerName &nbsp; &nbsp; &nbsp; :</span></p> <p><span style="font-family:courier new,courier;font-size:small;">ObjectId &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : {1}\\ISCKC-HV01\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StorageTier.ObjectId=&quot;{6fa45e22-498a-11e3-80b4-806e6f6e6963}:ST:{5c3a3723-4ab2-11e3-80bb-38eaa78c3d44}{5c3a37af-4ab2-11e3-80bb-38eaa78c3d44}&quot;</span></p> <p><span style="font-family:courier new,courier;font-size:small;">PassThroughClass &nbsp; &nbsp; :</span></p> <p><span style="font-family:courier new,courier;font-size:small;">PassThroughIds &nbsp; &nbsp; &nbsp; :</span></p> <p><span style="font-family:courier new,courier;font-size:small;">PassThroughNamespace :</span></p> <p><span style="font-family:courier new,courier;font-size:small;">PassThroughServer &nbsp; &nbsp;:</span></p> <p><span style="font-family:courier new,courier;font-size:small;">UniqueId &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : {5c3a37af-4ab2-11e3-80bb-38eaa78c3d44}</span></p> <p><span style="font-family:courier new,courier;font-size:small;">Description &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;:</span></p> <p><span style="font-family:courier new,courier;font-size:small;">FriendlyName &nbsp; &nbsp; &nbsp; &nbsp; : MirroredTieredSpace1_HDD_Tier</span></p> <p><span style="font-family:courier new,courier;font-size:small;">MediaType &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;: HDD</span></p> <p><span style="font-family:courier new,courier;font-size:small;">Size &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; : 2394444267520</span></p> <p><span style="font-family:courier new,courier;font-size:small;">PSComputerName &nbsp; &nbsp; &nbsp; :</span></p> <p>Thank you in advance.</p> <p>Regards,</p>

  • <p>@Sharbel </p> <p>On the Resize cmdlet, you have to specify the new total size, not the amount you want to increase.</p> <p>You can find an example of how to use Resize-StorageTier (and the required Update-Disk) in item 22 of this blog post: &nbsp;<a rel="nofollow" target="_new" href="http://blogs.technet.com/b/josebda/archive/2013/08/28/step-by-step-for-storage-spaces-tiering-in-windows-server-2012-r2.aspx">blogs.technet.com/.../step-by-step-for-storage-spaces-tiering-in-windows-server-2012-r2.aspx</a></p>

  • <p>Hey Jose,</p> <p>we want to use 2 Deployment units ( as shown in TechNet Wiki Storage Spaces Overview) with 2 nodes and 1 JBOD encosure each. Now my question: can I pool disc from both deployment units ?</p> <p>And if not: hoe can i mirror the data between the units?</p> <p>Regards</p> <p>Günter Herrwaldt</p>

  • <p>@Günter </p> <p>You can have a mirror with disks on the same JBOD.</p> <p>To mirror across 2 different JBODs, they must be connected to the same hosts.</p> <p>If there are different hosts connected to different JBODs, you can use Hyper-V Replica.</p>