A blog by Jose Barreto, a member of the File Server team at Microsoft.
All messages posted to this blog are provided "AS IS" with no warranties, and confer no rights.
Information on unreleased products are subject to change without notice.
Dates related to unreleased products are estimates and are subject to change without notice.
The content of this site are personal opinions and might not represent the Microsoft Corporation view.
The information contained in this blog represents my view on the issues discussed as of the date of publication.
You should not consider older, out-of-date posts to reflect my current thoughts and opinions.
© Copyright 2004-2012 by Jose Barreto. All rights reserved.
Follow @josebarreto on Twitter for updates on new blog posts.
This blog post is obsolete. For the latest post on this topic, see http://blogs.technet.com/b/josebda/archive/2012/07/31/deploying-windows-server-2012-with-smb-direct-smb-over-rdma-and-the-intel-neteffect-ne020-card-using-iwarp-step-by-step.aspx
We have covered the basics of SMB Direct and some of the use cases in previous blog posts and TechNet articles. If you haven’t seen those, here are a few pointers: Windows Server 2012 Beta SMB Overview High-Performance, Continuously Available File Share Storage for Server Applications Technical Preview Deploying Fast and Efficient File Servers for Server Applications Windows Server 2012 beta - Test cases for Hyper-V over SMB (includes PowerShell examples) Building Your Cloud Infrastructure: Converged Data Center with File Server Storage However, I get a lot of questions about specifically which cards work with this new feature and how exactly you set those up. This is one of a few blog posts that cover specific instructions for RDMA NICs. In this blog post, we’ll cover all the details to deploy the Intel NetEffect NE020 card, which use the iWARP “flavor” of RDMA.
We have covered the basics of SMB Direct and some of the use cases in previous blog posts and TechNet articles.
If you haven’t seen those, here are a few pointers:
However, I get a lot of questions about specifically which cards work with this new feature and how exactly you set those up. This is one of a few blog posts that cover specific instructions for RDMA NICs.
In this blog post, we’ll cover all the details to deploy the Intel NetEffect NE020 card, which use the iWARP “flavor” of RDMA.
To implement and test this technology, you will need: Two or more computers running Windows Server 2012 beta One or more Intel NetEffect Ethernet Adapter (NE020) card for each server One or more 10GbE switches Two or more cables required for the NE020 (typically using SFP+ connectors) Intel states support for Windows Server 2012 SMB Direct and Kernel-mode RDMA capabilities on the following adapter models: NetEffect™ Ethernet Server Cluster Adapter CX4 (Dover) NetEffect™ Ethernet Server Cluster Adapter SFP+SR (Argus) NetEffect™ Ethernet Server Cluster Adapter DA (Argus) You can find more information about these adapters on Intel’s web site at http://www.intel.com/Products/Server/Adapters/Server-Cluster/Server-Cluster-overview.htm. Important note: You should use NE020 cards with the chip version X2A1 (see picture below). If you have an old card with a X2A chip, it won't work with SMB Direct in Windows Server 2012.
To implement and test this technology, you will need:
Intel states support for Windows Server 2012 SMB Direct and Kernel-mode RDMA capabilities on the following adapter models:
You can find more information about these adapters on Intel’s web site at http://www.intel.com/Products/Server/Adapters/Server-Cluster/Server-Cluster-overview.htm.
Important note: You should use NE020 cards with the chip version X2A1 (see picture below). If you have an old card with a X2A chip, it won't work with SMB Direct in Windows Server 2012.
Windows Server 2012 Beta includes an inbox driver for the Intel NE020. However, Intel provides an updated driver for download. You should be able to use the inbox driver to access the Internet to download the updated driver. The latest Intel NE020 driver can be downloaded from: http://www.intel.com/support/go/network/adapter/ne020/win8. The driver is provided to you as a single ZIP file that you should extract to a specific folder. It will contain a few files, including one or more SYS files with the driver itself, an INF text file with the required information to install the driver and a few supporting files. To update to the new driver, follow these steps: Open “Device Manager” and find the NE020 device, under “Network Adapters” Right click the device and select “Update Driver Software” Click on “Browse my computer for driver software” Point to the folder where you extracted the ZIP file you downloaded Follow the wizard to complete the installation
Windows Server 2012 Beta includes an inbox driver for the Intel NE020. However, Intel provides an updated driver for download. You should be able to use the inbox driver to access the Internet to download the updated driver.
The latest Intel NE020 driver can be downloaded from: http://www.intel.com/support/go/network/adapter/ne020/win8. The driver is provided to you as a single ZIP file that you should extract to a specific folder. It will contain a few files, including one or more SYS files with the driver itself, an INF text file with the required information to install the driver and a few supporting files.
To update to the new driver, follow these steps:
After you have the drivers in place, you should configure the IP address for your NIC. If you’re using DHCP, that should happen automatically, so just skip to the next step. For those doing manual configuration, assign an IP address to your interface using either the GUI or something similar to the PowerShell below. This assumes that the interface is called RDMA1, that you’re assigning the IP address 192.168.1.10 to the interface and that your DNS server is at 192.168.1.1. Set-NetIPInterface -InterfaceAlias RDMA1 -DHCP Disabled Remove-NetIPAddress -InterfaceAlias RDMA1 -AddressFamily IPv4 -Confirm:$false New-NetIPAddress -InterfaceAlias RDMA1 -AddressFamily IPv4 -IPv4Address 192.168.1.10 -PrefixLength 24 -Type Unicast Set-DnsClientServerAddress -InterfaceAlias RDMA1 -ServerAddresses 192.168.1.1 Note: The Intel NE020 does not support iWARP with IPv6 addresses. You should disable IPv6 with this specific card if you’re planning to use the RDMA capability of the NIC. You can disable IPv6 for that interface using the following PowerShell cmdlet: Disable-NetAdapterBinding -InterfaceAlias RDMA1 -ComponentID ms_tcpip6
After you have the drivers in place, you should configure the IP address for your NIC. If you’re using DHCP, that should happen automatically, so just skip to the next step.
For those doing manual configuration, assign an IP address to your interface using either the GUI or something similar to the PowerShell below. This assumes that the interface is called RDMA1, that you’re assigning the IP address 192.168.1.10 to the interface and that your DNS server is at 192.168.1.1.
Set-NetIPInterface -InterfaceAlias RDMA1 -DHCP Disabled Remove-NetIPAddress -InterfaceAlias RDMA1 -AddressFamily IPv4 -Confirm:$false New-NetIPAddress -InterfaceAlias RDMA1 -AddressFamily IPv4 -IPv4Address 192.168.1.10 -PrefixLength 24 -Type Unicast Set-DnsClientServerAddress -InterfaceAlias RDMA1 -ServerAddresses 192.168.1.1
Note: The Intel NE020 does not support iWARP with IPv6 addresses. You should disable IPv6 with this specific card if you’re planning to use the RDMA capability of the NIC. You can disable IPv6 for that interface using the following PowerShell cmdlet:
Disable-NetAdapterBinding -InterfaceAlias RDMA1 -ComponentID ms_tcpip6
iWARP uses TCP/IP for communications, so you need to configure the Firewall to allow that traffic. You essentially need to add a firewall rule to the SMB Server to allow incoming traffic from the SMB Direct clients. In Windows Server 2012, SMB Direct with iWARP uses TCP port 5445, in addition to the traditional 445 port used for SMB. Here’s an example of how you would configure your firewall using a PowerShell cmdlet on the SMB server to allow access by the clients: New-NetFirewallRule –Name SMBiWarp -DisplayName "SMB iWARP In" -Enabled True -Profile Any -Direction Inbound -Action Allow -Protocol TCP -LocalPort 5445 If you have multiple SMB servers, you will need to create and enable this firewall rule on every server that will use SMB Direct with iWARP. Note: These instructions are for the Windows Server 2012 beta release only. We’re working to add a built-in firewall rule for iWARP before the final release. When that happens, you will be able to simply enable the pre-existing rule instead of creating a new one.
iWARP uses TCP/IP for communications, so you need to configure the Firewall to allow that traffic. You essentially need to add a firewall rule to the SMB Server to allow incoming traffic from the SMB Direct clients. In Windows Server 2012, SMB Direct with iWARP uses TCP port 5445, in addition to the traditional 445 port used for SMB.
Here’s an example of how you would configure your firewall using a PowerShell cmdlet on the SMB server to allow access by the clients:
New-NetFirewallRule –Name SMBiWarp -DisplayName "SMB iWARP In" -Enabled True -Profile Any -Direction Inbound -Action Allow -Protocol TCP -LocalPort 5445
If you have multiple SMB servers, you will need to create and enable this firewall rule on every server that will use SMB Direct with iWARP.
Note: These instructions are for the Windows Server 2012 beta release only. We’re working to add a built-in firewall rule for iWARP before the final release. When that happens, you will be able to simply enable the pre-existing rule instead of creating a new one.
One of the main advantages of iWARP RDMA technology is the ability to be routed across different subnets. While the most common setup is a single subnet (or maybe even single rack) deployment, you can use the Intel NE020 to connect computers across subnets. However, this capability is disabled by default on Windows Server 2012. To enable Network Direct (and therefor SMB Direct) in this fashion, you do need to configure every system (SMB Servers and SMB Clients) to allow routing RDMA across subnets. This is done using the following PowerShell cmdlet: Set-NetOffloadGlobalSetting -NetworkDirectAcrossIPSubnets Allow Disable-NetAdapter -InterfaceAlias RDMA1 Enable-NetAdapter -InterfaceAlias RDMA1 Note: Disabling and re-enabling the interface makes the settings change effective without a reboot. We recommend that you apply the configuration change above before creating any shares. If you do happen to apply it (or make any other major network configuration changes), the SMB client will re-evaluate its connections when new interfaces are detected or every 10 minutes. You can also tell SMB to update its connections immediately by using the following cmdlet on the SMB clients: Update-SmbMultichannelConnection
One of the main advantages of iWARP RDMA technology is the ability to be routed across different subnets. While the most common setup is a single subnet (or maybe even single rack) deployment, you can use the Intel NE020 to connect computers across subnets. However, this capability is disabled by default on Windows Server 2012.
To enable Network Direct (and therefor SMB Direct) in this fashion, you do need to configure every system (SMB Servers and SMB Clients) to allow routing RDMA across subnets. This is done using the following PowerShell cmdlet:
Set-NetOffloadGlobalSetting -NetworkDirectAcrossIPSubnets Allow Disable-NetAdapter -InterfaceAlias RDMA1 Enable-NetAdapter -InterfaceAlias RDMA1
Note: Disabling and re-enabling the interface makes the settings change effective without a reboot.
We recommend that you apply the configuration change above before creating any shares. If you do happen to apply it (or make any other major network configuration changes), the SMB client will re-evaluate its connections when new interfaces are detected or every 10 minutes. You can also tell SMB to update its connections immediately by using the following cmdlet on the SMB clients:
Update-SmbMultichannelConnection
Follow the steps below to confirm everything is working as expected:
Use the following PowerShell cmdlets to verify Network Direct is globally enabled and that you have NICs with the RDMA capability. Run on both the SMB server and the SMB client. Get-NetOffloadGlobalSetting | Select NetworkDirect Get-NetAdapterRDMA
Use the following PowerShell cmdlets to verify Network Direct is globally enabled and that you have NICs with the RDMA capability. Run on both the SMB server and the SMB client.
Get-NetOffloadGlobalSetting | Select NetworkDirect Get-NetAdapterRDMA
Use the following PowerShell cmdlets to make sure SMB Multichannel is enabled, confirm the NICs are being properly recognized by SMB and that their RDMA capability is being properly identified. On the SMB client, run the following PowerShell cmdlets: Get-SmbClientConfiguration | Select EnableMultichannel Get-SmbClientNetworkInterface On the SMB server, run the following PowerShell cmdlets: Get-SmbServerConfiguration | Select EnableMultichannel Get-SmbServerNetworkInterface netstat.exe -xan | ? {$_ -match "445"} Note: The NETSTAT command confirms if the File Server is listening on the RDMA interfaces.
Use the following PowerShell cmdlets to make sure SMB Multichannel is enabled, confirm the NICs are being properly recognized by SMB and that their RDMA capability is being properly identified.
On the SMB client, run the following PowerShell cmdlets:
Get-SmbClientConfiguration | Select EnableMultichannel Get-SmbClientNetworkInterface
On the SMB server, run the following PowerShell cmdlets:
Get-SmbServerConfiguration | Select EnableMultichannel Get-SmbServerNetworkInterface netstat.exe -xan | ? {$_ -match "445"}
Note: The NETSTAT command confirms if the File Server is listening on the RDMA interfaces.
On the SMB client, start a long-running file copy to create a lasting session with the SMB Server. While the copy is ongoing, open a PowerShell window and run the following cmdlets to verify the connection is using the right SMB dialect and that SMB Direct is working: Get-SmbConnection Get-SmbMultichannelConnection netstat.exe -xan | ? {$_ -match "445"} Note: If you have no activity while you run the commands above, it’s possible you get an empty list. This is likely because your session has expired and there are no current connections.
On the SMB client, start a long-running file copy to create a lasting session with the SMB Server. While the copy is ongoing, open a PowerShell window and run the following cmdlets to verify the connection is using the right SMB dialect and that SMB Direct is working:
Get-SmbConnection Get-SmbMultichannelConnection netstat.exe -xan | ? {$_ -match "445"}
Note: If you have no activity while you run the commands above, it’s possible you get an empty list. This is likely because your session has expired and there are no current connections.
On the SMB client, open a PowerShell window and run the following cmdlets to view the SMB events that confirm that you have an SMB Direct connection. If there are any RDMA-related connection errors, you will also see them: Get-WinEvent -LogName Microsoft-Windows-SMBClient/Operational | ? Message -match "RDMA"
On the SMB client, open a PowerShell window and run the following cmdlets to view the SMB events that confirm that you have an SMB Direct connection. If there are any RDMA-related connection errors, you will also see them:
Get-WinEvent -LogName Microsoft-Windows-SMBClient/Operational | ? Message -match "RDMA"
I hope this helps you with your testing of the Intel NE020. I wanted to covered all different angles to make sure you don’t miss any relevant steps. I also wanted to have enough troubleshooting guidance here to get you covered for any known issues. Let us know how was your experience with the beta by posting a comment.