System Center Mobile Device Manager Team Blog : Device Management

Troubleshooting device connection to the Device Management server

Jarrett Renshaw — Wed, 20 Aug 2008 00:25:00 GMT

You are enrolled. The device resets and shortly afterward you see the checked "V" icon, indicating you have a successful VPN tunnel to the MDM GAteway. Yes!

Now you wait for that PIN policy to come down, or that software package... and nothing happens.

Hmmm... are we REALLY connecting to the VPN server? The VPNDiag tool confirms that indeed we are.

At this point, the device has a VPN tunnel to the MDM Gateway, but something is wrong with the rest of the path to or from the Device Management server. (If VPN is not working, check out the Enrolled device cannot connect to Gateway topic the Technet troubleshooting documentation.)

If VPN is working but the client cannot connect to the DM, the client will not receive policy, software or Wipe Now requests. Until the device can contact the device management server and become managed, it remains in the Pending Enrollments node.

Troubleshooting

Test the DM server URL from the client and the DM server. You can find the URL in a number of ways to verify it is correct:

a. Device Status Viewer tool in the Client Resource kit. The URL is displayed on the home screen of the tool

b. \Deviceupdate.log on the device will show this if logging is enabled (see troubleshooting step 2 below)

c. The URL is also stored in the SCMDM2008DeviceManagement Active Directory SCP under ‘keywords’ > “url=”

d. You can easily enter the URL on the device or a test computer using the format:

https://DMServer.Mydomain.com:8443/MDM/TEE/handler.ashx

Enter this into Pocket IE on the device. You should get a “Choose a certificate” warning.

If the device cannot connect to the URL, you can also test this URL locally on the DM to narrow down the issue to connectivity or a problem with the server itself.

We’ll assume for now that the DM can connect to the URL using the localhost address so we know IIS and the server-side MDM services are working. Only devices cannot reach the DM. In this case, the issue is with the network, firewall, web certificate, or DNS.

The device might fail to contact the device management server for the following reasons:

1. It is necessary for the device to be able to resolve the FQDN of the DM server in DNS. Make sure there is a correct A record on the DNS server for the DM server. Depending on the network topology, DNS may be configured in different ways. If using the DNS server on the internal network, DNS traffic must be allowed on the internal firewall so the device can query the DNS server. If using a DNS server on the perimeter network, DNS must be configured to resolve the DM FQDN to the internal IP address of the DM.

2. The company firewall is blocking port 8443. For a list of ports required by SCMDM on the firewall see the planning guide.

3. The company proxy is blocking TCP port 8443 to the device management server. Allow tunneling on this port to allow the device to contact the device management server. This will only occur if the device connected successfully once before and received proxy policy. For details and resolution steps see Error 2147467259 When Synchronizing Policy in the Technet troubleshooting documentation.

4. A persistent route is needed on the MDM gateway to the corporate network through the back-end firewall, and another route is needed on the back-end firewall server to the VPN client address pool subnet through the MDM gateway. One simple method that may or may not satisfy your topology requirements is to configure the routes locally on the MDM Gateway and backend firewall servers as shown in this example:

Route #1 (on the GW): “route –p add <corporate subnet> mask <subnet mask> <Firewall IP>” which adds a route to the corporate network through the Back-end Firewall.

Route #2 (On the back-end firewall): “route –p add <Client pool subnet> mask <subnet mask> < Gateway IP>” which adds a route to the SCMDM 2008 client network through the SCMDM GW.

Note: If a Redirection Default Gateway is configured, MDM Gateway server will still prioritize any local static routes configured to particular destinations. If there is no such static route for the destination that you are trying to reach, then the packet will be forwarded to the Redirection Default Gateway.

5. It is also necessary to make sure the internal DNS server can access the Client VPN address pool on the MDM Gateway in order to forward resolved queries back to the devices. If the default gateway is, for example, the internal IP address of the back-end firewall, and the DNS server default gateway is not the internal IP address of the back-end firewall, a persistent (static) route may need to be configured as follows:

“route –p add <Client pool subnet> mask <subnet mask> < Firewall IP>” which adds a route to the SCMDM 2008 client network through the back-end firewall.

6. The DM’s web certificate subject name must match the FQDN of the DM server and chain to a valid root CA. Use the Best Practices Analyzer in the resource kit to check this.

These scenarios can vary, but the principle is the same

- Devices must have access to a DNS server to resolve the FQDN of the DM server

- There must be a route from the client VPN address pool on the MDM Gateway to the internal network

- There must be a route from the internal network to the Client VPN address pool on the MDM Gateway

- The internal DNS server must have access to the VPN client address pool subnet in order to respond.

- The DM certificate must be configured correctly and chain to the same root CA as the other servers – SCMDM only supports one root CA per instance.

Logging

Enable OMADM logging on the device using the Connect Now tool from the Client Tools in the MDM resource kit download. Issues contacting the DM server are logged in \deviceupdate.log.

The error codes in the log are Wininet errors, so you can look them up. Some that I have run across are:

Fail (-2147012889)

This error translates to ERROR_WINHTTP_NAME_NOT_RESOLVED and the symptoms are that we cannot connect to the internal network or the internet.

Check all the items above and make sure routing and DNS are configured properly. Also make sure port 8443 and Protocol 50 (IPSec ESP) are allowed both ways on the external firewall and Gateway Server’s Windows Firewall if it is enabled. This protocol allows browsing of internal and external web sites.

Failed sending an HTTP request to the server (0x80072f7d).

This can be caused by a problem with the certificate on the DM web site. Usually the log will show a successful connection to the server followed by the failure:

2008-07-17 15:58:38 omadmclient.exe: Establishing connection to https://dm.mdm.com:8443/MDM/TEE/Handler.ashx

2008-07-17 15:58:39 omadmclient.exe: [PID = 0x9e0b15d2] + Attempting to establish connection

2008-07-17 15:58:39 omadmclient.exe: [PID = 0x9e0b15d2] - Establishing connection

2008-07-17 15:58:39 omadmclient.exe: [PID = 0x9e0b15d2] + Transmitting package data

2008-07-17 15:58:41 omadmclient.exe: Failed sending an HTTP request to the server (0x80072f7d).

2008-07-17 15:58:41 omadmclient.exe: [PID = 0x9e0b15d2] - Transmitting package data FAILED (hr = 0x80072f7d)

Check the DM's web certificate and make sure the subject name matches the DM FQDN, the url= value in the SCMDM2008DeviceManagement SCP, and that the certificate is valid. Certutil is invaluable for verifying that all the certs are correctly deployed. A good resource for how to use certutil is http://blogs.technet.com/pki/archive/2006/11/30/basic-crl-checking-with-certutil.aspx

You can replace the certificate manually with a new one by using the MDM Certificate tool in the MDM Server resource kit or by following the intructions in the MDM product documentation at http://technet.microsoft.com/en-us/library/cc135742.aspx.

Remote Wipe Now and MDM Alerter troubleshooting - Part 2

Dave Madison — Mon, 28 Jul 2008 04:21:00 GMT

In my last post I discussed the basics of Remote Wipe, and some steps to take to ensure if it is working or not. In this second part, we will get more detailed on how to troublehoot Device Wipe.

1. Troubleshooting with the Device Log

Once you have confirmed that the conditions in 1-3(described in the previous post) are not the case, you can turn on device logging in order to narrow down the problem. The easiest way to turn on device logging is by using the SCMDM 2008 Resource Kit client tools, located here: http://technet.microsoft.com/en-us/scmdm/cc304591.aspx

1. Enable Alerter client logging

a. Download the client tools from the above location

b. Unzip to a location on your computer.

c. Once you have unzipped this file, connect your device to the computer via a cable

d. Transfer the MDMConnectNow.CAB (located in the MDMResourceKit\MDMClientTools\MDMConnectNow folder) to the device.

e. Disconnect your device from the computer.

f. On the device, navigate to the location where you transferred the MDMConnectNow.CAB. Click on the MDMConnectNow.CAB and install it.

g. Once installed, go to the Start Menu and then All Programs. Run the MDMConnectNow tool.

h. Choose Menu >> Logging

i. Under logging, check the “Enable Alerter Log” check box

j. Once you check that box, below in the dropdown box, you must also select Alerter (without this it will not work).

k. Now Alerter logging is enabled and it will write a log to ./deviceupdate.log on the device.

2. Once you have Alerter logging enabled, issue a wipe from the Mobile Device Manager Console, the MDM shell, or the Self Service Portal. Mark the time that the wipe was issued.

3. Look for the presence of 5506 or 5507 event IDs on the Gateway Server (as in the section “Check the Event Log on the MDM Gateway” above. Mark the time for the 5507(s) or 5506(s).

4. Once you see a 5506 or 5507 event on the Gateway for the device, transfer the deviceupdate.log to a computer from the device. You must do this before the wipe happens or you will lose all the tracing information.

5. Open the client logging file on the computer in a text reading program like Notepad.

6. Search for the following terms in the document:

a. “Rejecting packet” – See below for Rejected packets

b. “Received push with timestamp” and “Accepting packet” -- See below for Successful packets section

c. No “Received push with timestamp” in the log

a. Rejected Packets

If you see the text “Rejecting packet” in the Alerter log, it means that the Alerter was rejecting the packet because the source address did not match the expected address.

Seeing this in the logs is actually a good sign, in that the Alerts are getting to the device, but they are being rejected.

In the log it will give the expected source IP (The IP address that the Mobile VPN client is communicating with) and the actual IP (The IP address that the packet came from). A mismatch here generally indicates one of two issues:

· There is a routing problem on the VPN Gateway

· There is a network element between the Gateway and the device performing Network Address Translation for the Gateway external IP address

· Check Again for a NAT in front of the GW

Check the actual IP address in the log? Is this address known to you? Does it look like something is performing a NAT on the Gateway IP?

Remove the NAT and try the wipe again.

· Check the routing on the Gateway

There may be an issue with the routing on the Gateway. Perform the following steps to diagnose the problem.

Determine the Address Pool

1. On a machine with the Administrator Tools installed on the corporate network, open the MDM Shell.

2. Run Get-MDMGatewayServer

3. For the Gateway Server(s) where devices are connecting, obtain the address pool which should be in the form of something like: {10.10.10.0/255.255.255.0}. It may contain multiple values.

Determine the routes on the Gateway Server

4. Log into the Gateway Server machine(s)

5. Open a Command Prompt on the Gateways and run the following command

>route print

6. The above command will print out a list of routes like the following:

>route print

IPv4 Route Table

===========================================================================

Interface List

……

===========================================================================

Active Routes:

Network Destination Netmask Gateway Interface Metric

0.0.0.0 0.0.0.0 aaa.107.247.aaa aaa.107.247.aaa 10

3.4.0.0 255.255.0.0 aaa.54.44.1 aaa.54.44.33 1

10.0.0.0 255.0.0.0 aaa.54.44.1 aaa.54.44.33 1

10.10.10.0 255.255.255.0 aaa.54.44.32 aaa.54.44.33 1

………continues……

===========================================================================

Persistent Routes:

Network Address Netmask Gateway Address Metric

aaa.23.64.227 255.255.255.255 aaa.54.44.1 1

aaa.27.52.13 255.255.255.255 aaa.54.44.1 1

aaa.27.52.33 255.255.255.255 aaa.54.44.1 1

aaa.29.28.102 255.255.255.255 aaa.54.44.1 1

aaa.29.28.136 255.255.255.255 aaa.54.44.1 1

………continues……

7. Look in the Active routes for routes with a Network Destination that matches the address pool(s) obtained in Step 3 above.

8. There should be only 1 route to each of the device address pool(s) configured on your Gateway(s). That route should go through the external interface on the Gateway.

For example if a Gateway has the following interfaces:

o aaa.54.44.33 – external interface

o bbb.192.0.1 – internal interface

The below is a correct route because it goes through the external aaa.54.44.33 interface:

Active Routes:

Network Destination Netmask Gateway Interface Metric

10.10.10.0 255.0.0.0 aaa.54.44.1 aaa.54.44.33 1

The below is an incorrect route because it goes through the internal bbb.192.0.1 interface:

Active Routes:

Network Destination Netmask Gateway Interface Metric

10.10.10.0 255.0.0.0 aaa.54.44.1 bbb.192.0.1 1

The below is incorrect as well. Although there is a route correctly through the external interface, there is also one through the internal interface.

Active Routes:

Network Destination Netmask Gateway Interface Metric

10.10.10.0 255.0.0.0 aaa.54.44.1 aaa.54.44.33 1

10.10.10.0 255.0.0.0 aaa.54.44.1 bbb.192.0.1 1

9. You can reboot the Gateway server and recheck the route. If the incorrect route still remains, you can correct the route using the “route” command. Run “route” at the command prompt for more information.

10. Also check for incorrect routes in the Persistent Routes section.

11. Once the routing is correct, try the troubleshooting steps again.

b. Successful Push Packet

If you see successfully received push packets to the Alerter it means that Alerts are getting to the device and they are being processed.

If you are still having issues with the wipe request and the time it is taking, you can check the Device Management server to ensure that it is functioning as expected.

c. No Successful Push packet in the log

If you do not see any successful push packets in the log, it means that the device is not receiving any alert requests. Check the routing on the Gateway by following the “Check the routing on the Gateway” steps above.

Check the networking between the Gateway server and the device and correct any networking issues.

Ideally, Remote Wipe Now should work smoothly. In the event there are problems, these steps should help you determine the cause.

Marc McClure

Program Manager

Mobile Information Worker

Remote Wipe Now and MDM Alerter troubleshooting

Dave Madison — Tue, 22 Jul 2008 03:07:00 GMT

Remote Wipe Now and MDM Alerter troubleshooting

A key security feature of SCMDM is the ability to wipe a device remotely. Often time is of the essence, so it is important to know if a wipe was successful or not. Here we will discuss how remote wipe works and how to troubleshoot it,

Introduction

Remote wiping of a device is a security feature that allows administrators to remotely send a command to an MDM managed device that causes it to erase all data and return to factory defaults. This is useful if the device has been lost or stolen. In a remote wipe, the flash and storage card data are overwritten, leaving only the base OS on the device and no user data. Administrators can also deploy the Self Service Portal which allows users to wipe their own devices.

For wipe operations, time is critical - that's why we tend to call the feature "Wipe Now" internally. Without the Now you just have a Wipe Soon operation. Wiping in 8 hours doesn't work great when your device and all its data are lost or stolen. Remote wipe commands in MDM depend heavily on networking between the server and the device and this is where problems can often occur.

In pilot environments administrators often run MDM Remote Wipe tests because it is easy to see it working. However, this is one area in deployment where it is easy to make mistakes and cause the Wipe to take longer than expected. Below are troubleshooting steps to help to determine the cause of a Wipe Now command taking longer than the expected ~~2-6 minutes~~. Failed to catch this before posting...expected time is 3-15 minutes.

How Remote Wipe Works

Before going into troubleshooting, here's a brief overview of how Wipe Now works.

An administrator or user submits a wipe request through the console, MDM Shell, or Self Service Portal.
The wipe request is stored in the DM Engine database for the device to pick up at its next scheduled OMA session

If we were to stop at this point it would be a "Wipe Soon" and the device would connect at its regular interval (4 hours, for example) and pick up the wipe. Wipes submitted are always submitted as a "Wipe Now" now command, and thus we have to go a step further

In parallel to adding the wipe request for retrieval, the wipe driver also calls the Alerter component to inform the device of a pending wipe request.
The Alerter sends an alert to the device over the Mobile VPN. The alert can only be sent through the VPN tunnel and thus Wipe Now requires VPN connectivity
The Alerter client on the device receives this Alert and immediately starts a management session with the Device Management server.
The device picks up its wipe request from the Device Management server, sends back an Acknowledgement that started the wipe, and starts the wipe process

Troubleshooting Wipes that are Taking Longer than Expected

From the moment a wipe request is submitted, a wipe should take approximately ~~2 - 6 minutes~~ 3-15 minutes] depending on the network and other factors. If it is taking longer than that, below are some troubleshooting steps that you can perform to determine the cause of the latent wipe request.

1. Verify that Management Sessions are operating as expected

Ensure that typical device management sessions, not related to wipe are working as expected. You can download the MDM Connect Now tool located in the SCMDM 2008 Resource Kit Client Tools package here: http://technet.microsoft.com/en-us/scmdm/cc304591.aspx Using this tool and associated documentation, you can verify that management sessions for devices are successful.

If management sessions are not working, you will need to fix this problem before devices can successfully receive the remote wipe command.

2. Verify the device is connected to the VPN

As discussed in the above "How Remote Wipe Works" section, remote wipe relies on the VPN being connected in order to send alerts to the device. The device may not be connected to the Mobile VPN for many reasons, but some of the most common are:

The device is switched off
The Mobile VPN is disabled (if users have the ability to disable it)
The user is roaming and VPN is off
The data connection is improperly configured
The user is temporarily out of service coverage area

In all the above cases, the device will not receive the Alert message as it relies on the VPN tunnel being up. When the device reconnects to the VPN, it will receive an Alert message or will start a management session immediately if it has missed its regularly scheduled session.

3. Verify the Gateway Server is not behind a Network Address Translator (NAT)

Once you have verified that management sessions are operating and the VPN was up on the device you were attempting to wipe, you need to check that the Gateway is not behind a Network Address Translator. MDM does not support locating a Gateway Server behind a NAT. There are several reasons for this requirement; one of the reasons has to do with the Alerter.

The reason the Alerter does not work with Gateway Servers behind a NAT is for security purposes. We'll talk about this some more, but for added security, the Alerter checks to ensure that the Alert it received is really from the MDM Gateway Server and not from a potential attacker.

The Gateway Server must have a public IP address and must not sit behind a NAT or the Alerter cannot verify the alert that the alert is valid. The Alerter discards invalid alerts. Ensure that your Gateway is not behind a NAT.

4. Check the Event Log on the MDM Gateway

On the MDM Gateway server, open up the "VPN Policy Engine" Event Log. Search for events 5507 and events 5506 in the log

Event 5506 indicates that the Alerter received a response from the device.

Event 5507 indicates that the Alerter sent a number of retries,but never heard from the device.

Further debugging information is needed for the following scenarios:

5507 events for every wipe issued

5507 event for a wipe issued in a controlled environment, where you know the device connected, online and available

Inconsistent 5507 and 5506 events

A mixture of 5507 and 5506 events are expected for normal operations as some devices may be offline, out of service range, or not connected for another reason. This is generally normal.

In our next post we will look at how to use the some client tools available in the resource kit, and how to use the Device Log to further narrow down any issues.

Marc McClure

Program Manager

Mobile Information Worker

Getting Started with SCMDM

Dave Madison — Mon, 07 Jul 2008 07:39:00 GMT

At the risk of stating the obvious, System Center Mobile Device Manager (SCMDM) is a Version 1 product. There are many opportunities that come with shipping a V1 product. The most important opportunity is for the team firs to identify the need for the product, it was clear to us that mobile devices were becoming an essential tool for information workers. In addition, the upcoming workforce will expect and demand that the majority of their information be accessible from their mobile device. In short, your mobile device is just as important as your laptop. That situation also presents opportunities for a company's IT departments. How do they manage those devices in the same manner they manage laptops and desktops? This, then, was the challenge we set out to solve with SCMDM. We worked closely with key enterprise customers, to identify what they needed--the ability to enforce policies, wipe devices, access data inside the firewall securely, and make Line of Business applications available on the mobile device. As we built the product and went through our milestones, we worked with our Technology Adoption Program customers to get specific technical feedback on the product in "the real world". These customers told us we had met their primary requirements and the product was ready to go. So, we launched SCMDM this past April 2008.

Three months have since passed, with many of our TAP customers continuing with their testing and pilot programs. But the momentum has yet to really get started. Because SCMDM supports Windows Mobile v. 6.1 and above, the real benefit of the product will start to take off over the next weeks and months as the 6.1 devices start to become commercially available. You will also be hearing more about SCMDM from your Microsoft account teams.

But don't wait for them; you can get started testing and piloting SCMDM. There is a 180-day Evaluation version available at http://technet.microsoft.com/en-us/evalcenter/cc339027.aspx. The perfect place to get the technical content you need to get started is The SCMDM Tech Center.

Many customers that are testing and piloting SCMDM have asked how to best integrate it with existing server technologies they may have. We've heard them and responded. We recently published a series of technical guides that explain how SCMDM can integrate with your existing solutions:

· Configuring External and Internal Firewalls for MDM

· Integrating MDM with Microsoft Exchange Server

· Deploying Mobile Device Manager in a Global Enterprise Environment

· Integrating Mobile Device Manager with Existing Web Sites or SharePoint Server

· Integrating Mobile Device Manager with Office Communications Server

We hope this gives you a good start in understanding and planning for SCMDM. While pointing you to all this great content is important in building your foundation, over the next weeks, months, and beyond you will start seeing posts from the product team that will discuss specific SCMDM technologies in depth. Your comments and feedback will be essential in ensuring we continue to deliver the best Mobile Device Management technology.