Server for NFS Diagnostics

Server for NFS Diagnostics

  • Comments 2
  • Likes

In this post, we will discuss the instrumentation available in Server for NFS in Windows Server 2012 and how it can be used to detect and diagnose any deployment and operational issues.

Event Viewer

There are quite few changes in the Server for NFS event model for Windows Server 2012. In the previous releases of Windows Server, Server for NFS logged events in to the System channel. In Windows Server 2012, Server for NFS logs the events into its own channel. The event IDs are unchanged; however event channels and provider GUIDs are different. The following figure displays the layout of the event channel, where Server for NFS logs events.



Activity Logging

Server for NFS logs the events for some of the NFS operations into Operational channel which includes:

  • Read and Write
  • Lock and Unlock
  • Mount and Unmount
  • Create and Delete

The activity logging can be enabled using the PowerShell cmdlet Set-NfsServerConfiguration.

For example, the following command enables the activity logging for mount, read, and write operations.


PS C:\> Set-NfsServerConfiguration –LogActivity mount,read,write
The activity logging can also be enabled through the Services for Network File System management snap-in. Follow these steps to enable Activity Logging in Server for NFS.
  1. Open Server Manager and then click Services for Network File System (NFS) from the Tools menu.
  2. In Services for Network File System, right-click on Server for NFS and select Properties.
  3. Switch to Activity Logging tab and select the activities you want to be logged.

 Identity Mapping Events

Server for NFS logs identity mapping related events into the IdentityMapping channel. The following are some of the critical events to watch for when local file based identity mapping is configured as the identity mapping source.

Event ID

Level

Message

Resolution

4025

Error

A duplicate ID value <UID/GID number> was found when loading <FileName>. The file will not be used as a mapping source

Server for NFS performs some validations against the passwd and group files. This event is logged if multiple user accounts in the passwd file have the same user identifier (UID) or multiple group accounts in the group file have the same group identifier (GID). To resolve this issue, edit the passwd/group files to change the UID/GID on the conflicting user/group account having this issue. Use Get-NfsMappedIdentity PowerShell cmdlet to retrieve the list of users/groups having the UID/GID mentioned in the event.

4026

Error

A duplicate name <AccountName> was found when loading <FileName>. The file will not be used as a mapping source

Edit the file specified in the event and remove the duplicate account name.

4027

Error

A syntax error was found on line <LineNumber> when loading <FileName>. The file will not be used as a mapping source

The passwd/group file specified in the event is not following the correct syntax required by Server for NFS. Edit the file and check for any errors at the line number mentioned.

4029

Warning

Mapping source update detected. File <FileName> not found

Server for NFS looks for the passwd and group files at following location,  %windir%\system32\drivers\etc. Make sure that the files are present at this location and NfsService has permission to read these files.[MCJ1] 

4030

Error

<FileName> has no data. The file will not be used as a mapping source.

The passwd/group file is empty. Make sure that you have right files stored at location %windir%\system32\drivers\etc or remove the files from this location if it was not intended to use mapping files as identity mapping source.

4032

Error

<FileName>. Memory allocation failed when processing the file. It will not be used as a mapping source.

The system is overloaded and there is not enough memory available to process the request. Close some of the applications that are not required to free the memory.

4033

Error

<FileName>. Failed to process the file. The file will not be used as a mapping source.

Unexpected error encountered while opening the file specified in the event. Check the file for correct syntax.

 

The following are some of the critical events when the identity mapping store is Active Directory, Active Directory Lightweight Directory Services or other RFC2307 compliant LDAP store.

Event ID

Level

Message

Resolution

4012

Error

Active Directory Domain Services(R) contains multiple users which match attribute <AttributeName>.  Only one Windows(R) user should be associated with each UNIX UID. With multiple Windows users associated with one UNIX UID, Server for NFS cannot determine which Windows user is requesting access to files.  No Windows users associated with the same UNIX UID will be able to access files through Server for NFS. Try removing the duplicate UNIX UID entries.

Event 4012 indicates that the configured identity mapping store contains multiple user accounts that have an identical value for attribute uidNumber (the value is given in the event message text).

 Run the following PowerShell command to find out the user accounts having identical value for attribute uidNumber.

 Get-NfsMappedIdentity –AccountType user –Uid <UIdNumber>

 Then correct the value of the uidNumber attribute of the user accounts using the following PowerShell command.

 Set-NfsMappedIdentity –UserName <sAMAccountName> -UId <UidNumber>

4013

Error

Active Directory Domain Services(R) contains multiple groups which match attribute <AttributeName>.  Only one Windows(R) group should be associated with each UNIX GID. With multiple Windows groups associated with one UNIX GID, Server for NFS cannot determine which Windows group to use to grant access to files.

 Try removing the duplicate UNIX GID entries.

Event 4013 indicates that the configured identity mapping store contains multiple group accounts that have an identical value for attribute gidNumber (the value is given in the event message text).

 Run following PowerShell command to find out the group accounts having identical value for attribute gidNumber.

 Get-NfsMappedIdentity –AccountType group –Gid <GIdNumber>

 Use following PowerShell command to correct the value of the gidNumber attribute of the group account.

 Set-NfsMappedIdentity –GroupName <sAMAccountName> -GId <GidNumber>

4014

Error

Active Directory Domain Services(R) contains multiple users which match attribute <AttributeName>.  Only one Windows(R) user should be associated with each sAMAccountName. With multiple Windows users associated with one sAMAccountName, Server for NFS cannot determine which Windows user is requesting access to files.  No Windows users associated with the same sAMAccountName will be able to access files through Server for NFS. Try removing the duplicate sAMAccountName entries.

Event 4014 indicates that the configured identity mapping store contains multiple users that have an identical value for attribute sAMAccountName (the value is given in the event message text).

 Try removing the duplicate user accounts having identical sAMAccountName.

4015

Error

Active Directory Domain Services(R) contains multiple groups which match attribute <AttributeName>.  Only one Windows(R) group should be associated with each sAMAccountName. With multiple Windows groups associated with one sAMAccountName, Server for NFS cannot determine which Windows group to use to grant access to files. Try removing the duplicate sAMAccountName entries.

Event 4015 indicates that the configured identity mapping store contains multiple groups that have an identical value for attribute sAMAccountName (the value is given in the event message text).

 Try removing the duplicate group accounts having identical sAMAccountName.

4016

Error

Server for NFS could not connect to the Lightweight Directory Access Protocol (LDAP) server for domain <DomainName>. Without a connection to the LDAP server, Server for NFS cannot query for Windows-to-UNIX user account mappings and cannot grant file access to any user. Verify that Server for NFS is configured to use the appropriate LDAP server using the Nfsadmin command-line tool.

Event 4016 indicates that Server for NFS is not configured to use either Active Directory Domain Services (AD DS) or any other LDAP store or User Name Mapping as a Windows-UNIX identity mapping source.

 Use Set-NfsMappingStore PowerShell cmdlet to set the identity mapping store for the Server for NFS.

4017

Error

Server for NFS could not find any Lightweight Directory Access Protocol (LDAP) accounts which match attribute <AttributeName>. Without attribute <AttributeName>, Server for NFS does not know the corresponding Windows user account for the Unix user and cannot grant file access to the UNIX user.%n%n Verify that the LDAP server is configured with the appropriate attributes.

Event 4017 indicates that Server for NFS could not find any Lightweight Directory Access Protocol (LDAP) accounts that match the attribute specified in the event message text.

 Add the necessary account information to the LDAP store by using New-NfsMappedIdentity or set-NfsMappedIdentity cmdlet. Then use Resolve-NfsMappedIdentity cmdlet to verify that Server for NFS is able to resolve the user account using the attribute specified in the event text.

 

The following are some of the critical events when the identity mapping store User Name Mapping (UNM) server.

Event ID

Level

Message

Resolution

1005

Error

Server for NFS could not obtain mapping information from User Name Mapping.  Server for NFS will make another attempt after <Duration> minutes. Without any mapping information, Server for NFS will not be able grant file access to users. Verify the User Name Mapping service is started on the User Name Mapping server, and User Name Mapping ports are open on firewalls.

Event 1005 indicates that Server for NFS cannot obtain mapping information from User Name Mapping (UNM) server. Incorrect settings in User Name Mapping source could cause this. Use Set-NfsMappingStore PowerShell cmdlet to configure User Name Mapping server. Get-NfsMappingStore cmdlet can be used to retrieve the current configuration. Use Resolve-NfsMappedIdentity cmdlet to verify that Server for NFS can obtain the mapping information from UNM server.

1006

Error

Server for NFS is not configured for either Active Directory Lookup or User Name Mapping. Without either Active Directory Lookup or User Name Mapping configured for the server, or Unmapped UNIX User Access configured on all shares, Server for NFS cannot grant file access to users. Configure Server for NFS for either Active Directory Lookup or User Name Mapping using the Nfsadmin command-line tool, or Unmapped UNIX User Access using the Nfsshare command-line tool.

Event 1006 indicates that Server for NFS is not configured for either Active Directory Lookup or User Name Mapping.

If you have configured shares on the Server for NFS to use ‘Unmapped UNIX User Access’ mode, you may ignore this event.  Otherwise, to solve this problem, configure Server for NFS to use an identity mapping source using Set-NfsMappingStore PowerShell cmdlet. To verify that the mapping store is configured correctly, use Get-NfsMappingStore cmdlet.

1056

Error

Server for NFS could not obtain updated mapping information from User Name Mapping.  Server for NFS will continue to use the mapping information it has and make another attempt after <Duration> minutes. If this problem persists, Server for NFS mapping information may become significantly out-of-date and may not be able grant file access to users. Verify that the User Name Mapping service is started either locally or on the remote server, and that User Name Mapping ports are open on firewalls.

Event 1056 indicates that Server for NFS cannot obtain mapping information from User Name Mapping (UNM) server. Incorrect settings in User Name Mapping source could cause this. Use Set-NfsMappingStore PowerShell cmdlet to configure User Name Mapping server. Get-NfsMappingStore cmdlet can be used to retrieve the current mapping store configuration. Use Resolve-NfsMappedIdentity cmdlet to verify that Server for NFS can obtain the mapping information from UNM server.

 

Admin Channel Events

The Server for NFS logs critical events that need admin’s intervention into Admin channel. Following are some of the critical events and recommended resolution steps.

Event ID

Level

Message

Resolution

1059

Error

Server for NFS could not register with RPC Port Mapper on all requested port/protocol combinations.  Server for NFS will attempt to continue but some NFS clients may not function properly. Network File System (NFS) clients discover NFS servers by querying the port mapper for a remote server (also known as Portmap and Rpcbind).  NFS clients may not be able to discover and communicate with Server for NFS on this computer.

These events indicate that other programs might be using some of the TCP/IP ports that are required by Server for.

 Determine if Server for NFS has registered all protocols

 To determine the ports and transports that Server for NFS uses, at an elevated command prompt on the affected server, type rpcinfo.

 Server for NFS registers on port 2049 for udp, tcp, udp6, tcp6

 Make this TCP/IP port available and restart Server for NFS.

 To make TCP/IP port 2049 available and restart Server for NFS, use the following procedure:

 1. At an elevated command prompt, type “netstat -a -b –o” to display all connections with their associated executables and processes.

 2. Resolve port allocations conflicting with the NFS ports identified in Step 1 by stopping conflicting services, or programs.

 3. Type “nfsadmin server stop”.

 4. Type “nfsadmin server start”.

 

1060

Error

Server for NFS could not register the Network File System (NFS) protocol on the specified port (%5). Status: %6.  Server for NFS is will attempt to continue.  At least one successful NFS port registration is required to start Server for NFS but some NFS clients may not function properly without this specific port registration. Verify that no other programs have registered with RPC Port Mapper with the following parameters. Program Name:%1 Program Number%2 Version:%3 Protocol: %4 Port:%5

1064

Warning

Server for NFS cannot initialize the volume with drive letter %1 for sharing. Network File System (NFS) shared resources on the volume will not be available to NFS clients. Windows(R) may be low on system resources.  Try increasing available system resources by closing programs, then restart Server for NFS manually.

Event 1064 indicates that Server for NFS cannot provision the volume for sharing; therefore, shared resources on the volume will not be available to NFS clients. The likely cause is that the computer is short of resources.

 To resolve this issue increase available system resources using the following procedure:

 1. Close all programs and stop unnecessary services on the affected server.

 2. At an elevated PowerShell prompt , type “nfsadmin server stop”.

 3. Type “nfsadmin server start”.

 To verify Server for NFS is sharing files, use the following procedure:

 1. On the affected server, type Get-NfsShare.

 2. Verify that the list of shared resources is correct.

1069

Error

Server for NFS could not establish a connection with configured NIS server

Event 1069 indicates that Server for NFS is unable to access the Network Information Service (NIS) store where the netgroup configuration is stored. The most likely causes are:

 • NFS server is not configured appropriately to access NIS based netgroups.

 • There is a network connectivity issue between the Server for NFS and the NIS server.

 If Server for NFS is unable to access the netgroup store, determine if the location of the NIS NetGroup Source is accurate by using the following procedure:

 1. At PowerShell prompt on the affected server, type Get-NfsNetgroupStore.

 2. Verify that the NISDomain, and NISServer  are configured correctly.

 3. Verify that network connectivity exists between the Server for NFS and the NIS server where netgroups are configured as follows:

 • Use the rpcinfo.exe tool to verify that the NIS server is accessible over the network. To check if the source computer is accessible and the NIS service is registered on the source computer, type the following command, where <computername> is the name of the NIS server: rpcinfo <computername>. 

• The NIS service should appear in the output of this command as RPC program number 100004 and protocol version 2.

 Verify Server for NFS is configured appropriately to access NIS server

 Verify that Server for NFS is correctly configured to access the NIS server as follows:

 1. In PowerShell window, run Get-NfsServerConfiguration cmdlet.

 2. Verify that Protocol for NIS is UDP, TCP, or both, and is compatible with the protocol allowed at the NIS source computer as determined from the output of the command rpcinfo.exe <computername>.

 To verify that issue is resolved, use Get-NfsNetGroup cmdlet. You should be able to retrieve the netgroups from the netgroup store.

1071

Warning

Server for NFS was unable to obtain security information for the configured UnmappedUnixUserUsername user account %1. Check that the user account %1 is valid and meets all configured security policies. There may be additional information in the Windows Security event log. Server for NFS will attempt to revert to the default anonymous account. MSV Status: %2, SubStatus: %3S4U Status: %4, SubStatus: %5

Event 1071 indicates that Server for NFS was unable to obtain a logon token for the account used to process anonymous logons or for UNIX UIDs that do not have an explicit mapping. The event message details the account that led to the problem report. Ensure that the account is valid and can be used to perform a successful logon.

 

 

1072

Warning

Server for NFS was unable to obtain security information for the GSS user account %1. Check that the user account %1 is valid and meets all configured security policies. There may be additional information in the Windows Security event log. MSV Status: %2, SubStatus: %3 S4U Status: %4, SubStatus: %5

Event 1072 indicates Server for NFS was unable to obtain a logon token for the account used to access the NFS server when using an RPCSEC_GSS based identity. The event message details the account that led to the problem report. Ensure that the account is valid and can be used to perform a successful logon.

1073

Warning

Server for NFS was unable to obtain or refresh security information for the user account %1. Check that the user account %1 is valid and meets all configured security policies. There may be additional information in the Windows Security event log.%n%nMSV Status: %2, SubStatus: %3%nS4U Status: %4, SubStatus: %5

Event 1073 indicates Server for NFS was unable to refresh an access token. The event message details the account that led to the problem report. Ensure that the account is valid and can be used to perform a successful logon.

 

4021

Error

The Server for NFS was unable to begin monitoring of NFS related cluster events (%1). The Server for NFS will continue in a non-clustered mode.

These events indicate that either the Cluster Service is not running or the computer is low on resources.

 Determine if the Cluster Service is running as follows:

 1. At command prompt on the affected server, type services.msc.

 2. Check if Cluster Service is running.

 

4022

Error

The Server for NFS thread monitoring NFS related cluster events ended unexpectedly (%1). The Server for NFS will continue in a non-clustered mode.

4023

Warning

Server for NFS encountered an error condition when checking for the presence of Failover Clustering (%1) and will continue to operate but in a non-clustered configuration only. To re-detect Failover Clustering and allow Server for NFS to operate in a clustered configuration, Server for NFS should be restarted using either the Services for Network File System (NFS) administrative tool or nfsadmin server stop and nfsadmin server start

 

Performance Counters

 Server for NFS-NFSv4 Statistics

This performance counter set includes performance counters related to compound requests processed by Server for NFS. It also includes a performance counter indicating the count of virtual servers hosted by Server for NFS.

Name

Description

Total Compound Requests

Total number of compound requests processed by Server for NFS since startup

Successful Compound Responses

Total number of compound requests succeeded since Server for NFS started

Failed Compound Responses

Total number of compound requests failed since Server for NFS started

Total Virtual Servers

Current count of virtual servers hosted by the Server for NFS. This counter is incremented when virtual server is successfully started and decremented on virtual server stop. This counter will be set to one in non-cluster case. In case of cluster, there will be one instance of virtual server per Server for NFS resource.

 

Server for NFS-Netgroup

Name

Description

Failures Communicating With NIS

Number of time the Server for NFS failed to communicate with the NIS server.

 

Server for NFS-User Mapping

LDAP refers to Active Directory, Active Directory Lightweight Directory Services or any other RFC 2307-based LDAP Store. UNM Server refers to User Name Mapping server.

Name

Description

Total LDAP Requests

Number of LDAP query requests made by the Server for NFS since startup.

Total LDAP successes

Count of LDAP lookup requests which resulted in successful UID/GID to account name or account name to UID/GID lookup.

Total LDAP Failures

Count of LDAP lookup requests which failed to retrieve the identity mapping information from LDAP store.

Total LDAP Requests Per Second

Number of LDAP lookup requests performed  per second by the Server for NFS.

Total UNMP Requests

Number of user name mapping lookup requests performed by the Server for NFS since startup.

Total UNMP Failures

Count of user name mapping lookup request issued by the Server for NFS which resulted in failure. The failure reason could be anything like mapping does not exist or communication failure with the UNM server.

Total UNMP Successes

Count of mapping lookup request made against UNM Server which resulted in successful mapping information.

Total UNMP Requests Per Second

Count of UNMP mapping lookup requests issued by the Server for NFS per second.

Average LDAP Lookup Latency

Average amount of time taken by Server for NFS to resolve UID/GID to account name from the LDAP mapping store and vice versa. It is the total time spent doing the lookup in the LDAP mapping store divided by the total number of mapping lookup requests made to the LDAP mapping store.

Maximum LDAP Lookup Latency

Maximum amount of time taken by Server for NFS to resolve the identity mapping in the LDAP mapping store.

Average UNMP Lookup Latency

Average amount of time taken by Server for NFS to resolve the UID/GID to account name from UNMP mapping store and vice versa. It is basically the total time spent doing the lookup in the UNMP mapping store divided by the total number of mapping lookup requests made by the server to the UNMP mapping store.

Maximum UNMP Lookup Latency

Maximum amount of time spent by Server for NFS to resolve the identity mapping from the UNMP mapping store.

 

Server for NFS-NFSv4 Read Write Statistics

Name

Description

Total cached MDL Reads

Number of times the read operation is performed using a memory descriptor list (MDL) from the system cache manager.

Total Fast IO Reads

Number of times the read operation is performed using buffered IO from the system cache manager.

Total Unstable Writes

Count of NFS unstable writes performed by Server for NFS.

Average Fast IO Read Latency

Average time taken by Server for NFS to perform read operation using buffered IO from the system cache manager. It is the total time taken by the server performing all buffered IO reads divided by the number of buffered IO reads performed so far.

Average Non Fast IO Read Latency

Average amount of time taken by Server for NFS to perform read operations using IRP based IO.

 

Server for NFS-NFSv4 Request/Response Sizes

Name

Description

Maximum Size of NTFS Reads

Maximum size in bytes of the read request performed by Server for NFS.

Minimum Size of NTFS Reads

Minimum size in bytes of the read request performed by Server for NFS.

Maximum Size of NTFS Writes

Maximum size in bytes of the write request performed by Server for NFS.

Minimum Size of NTFS Writes

Minimum size in bytes of the write request performed by Server for NFS.

Maximum Compound Request Size

Maximum size in bytes of the NFS compound request.

Average Compound Request Size

Average size in bytes of the NFS compound request.

Maximum Compound Reply Size

Maximum size in bytes of the NFS compound reply.

Average Compound Reply Size

Average size in bytes of the NFS compound reply.

Maximum Compound Operations in Request

Maximum number of operations in a single NFS compound request.

Average Compound Operations in Request

Average number of operations in NFS compound request.

 

Server for NFS-NFSv4 Throughput

Name

Description

NFS Compounds Processed/Sec

Number of NFS compound requests processed per second.

 

Server for NFS- NFSv4 Operation Statistics

Server for NFS- NFSv4 Operation Statistics performance counter set is reported for each compound operation. There is one instance of the following of performance counters for each compound operation in the Server for NFS-NFS v4 implementation.

Name

Description

Count Of Operations Processed

Count of this NFS4 compound operations processed by the Server for NFS so far.

% Operations At Dispatch

This counter is not used in the current release.

Average Number of Times Operation requeued

Average Number of times this compound operation was re-queued for processing by the worker thread.

Reply Packet Not Cached Count

Number of times reply packet was not cached when requested by the client.

Average latency

Average amount of time taken by the server to execute this compound operation. This includes time taken to decoding the request and executing the operation.

 

Server for NFS - Session and Connection Statistics

Name

Description

Active Sessions Count

Number of active sessions with Server for NFS.

Active Connections Count

Number of active connections with Server for NFS.

Total Bad Session Requests

Number of session requests (OP_CREATE_SESSION) received by the Server for NFS so far with invalid arguments to operation.

KRB5 RPCSEC_GSS Requests Count

Number of requests received by Server for NFS with krb5 RPCSEC_GSS authentication.

KRB5I RPCSEC_GSS Requests Count

Number of requests received by Server for NFS with krb5i RPCSEC_GSS authentication.

KRB5P RPCSEC_GSS Requests Count

Number of requests received by Server for NFS with krb5p RPCSEC_GSS authentication.

AUTH_NONE Requests Count

Number of requests received by Server for NFS with AUTH_NONE authentication.

AUTH_UNIX Requests Count

Number of requests received by Server for NFS with AUTH_UNIX authentication.

Client With Sessions

Current count of clients that have session established with Server for NFS.

Total Client With Sessions

Number of clients that have created a session to Server for NFS since startup.

Number of times admin forcefully closed a session

Number of sessions force-closed by an administrator (Disconnect-NfsSession cmdlet).

Number Of Times Admin Revoked State

Number of open/lock states force-closed by an administrator (Revoke-NfsOpenFile / Revoke-NfsClientLock cmdlets).

Lease Expiry Revoke State Count

Number of open/lock states revoked by Server for NFs due  to the lease expiry.

Client Sessions Using Back Channel

Number of client sessions using back channel.

Clients Requesting SP4_MACH State Protection

Number of clients requesting SP4_MACH state protection.

Clients Requesting SP4_NONE State Protection

Number of clients requesting SP4_NONE state protection.

Clients Requesting SP4_SSV State Protection

Number of clients requesting SP4_SSV state protection.

Clients Requesting Bind Principal To State

Number of clients requesting bind principal to state.

Clients Requesting Persistent Session

Number of clients requesting a persistent session.

Number of clients requesting READW_LT

Count of requests for READW_LT.

Number of clients requesting WRITEW_LT

Count of requests for WRITEW_LT.

Special Anonymous State ID Use Count

Count of requests for special anonymous state ID.

Special Read Bypass State ID Use Count

Count of requests for special read bypass state ID.

Special Current State ID Use Count

Count of requests for special current state ID.

 

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment
  • very useful resource, thanks for share.

  • Is there any tracing or similar that can be turned on to see what's happening during cluster failover between the logs indicating volume arrival/creation and discovery completing? I'm having an issue with a Server 2012 cluster where NFS on one file server takes 10-12 minutes to resume after failover between nodes - more details and logs on the forums at social.technet.microsoft.com/.../2cd8a6e7-0fcb-4e6c-9672-bdfa19de00de...