System Center Mobile Device Manager Team Blog : wipe

Remote Wipe Now and MDM Alerter troubleshooting - Part 2

Dave Madison — Mon, 28 Jul 2008 04:21:00 GMT

In my last post I discussed the basics of Remote Wipe, and some steps to take to ensure if it is working or not. In this second part, we will get more detailed on how to troublehoot Device Wipe.

1. Troubleshooting with the Device Log

Once you have confirmed that the conditions in 1-3(described in the previous post) are not the case, you can turn on device logging in order to narrow down the problem. The easiest way to turn on device logging is by using the SCMDM 2008 Resource Kit client tools, located here: http://technet.microsoft.com/en-us/scmdm/cc304591.aspx

1. Enable Alerter client logging

a. Download the client tools from the above location

b. Unzip to a location on your computer.

c. Once you have unzipped this file, connect your device to the computer via a cable

d. Transfer the MDMConnectNow.CAB (located in the MDMResourceKit\MDMClientTools\MDMConnectNow folder) to the device.

e. Disconnect your device from the computer.

f. On the device, navigate to the location where you transferred the MDMConnectNow.CAB. Click on the MDMConnectNow.CAB and install it.

g. Once installed, go to the Start Menu and then All Programs. Run the MDMConnectNow tool.

h. Choose Menu >> Logging

i. Under logging, check the “Enable Alerter Log” check box

j. Once you check that box, below in the dropdown box, you must also select Alerter (without this it will not work).

k. Now Alerter logging is enabled and it will write a log to ./deviceupdate.log on the device.

2. Once you have Alerter logging enabled, issue a wipe from the Mobile Device Manager Console, the MDM shell, or the Self Service Portal. Mark the time that the wipe was issued.

3. Look for the presence of 5506 or 5507 event IDs on the Gateway Server (as in the section “Check the Event Log on the MDM Gateway” above. Mark the time for the 5507(s) or 5506(s).

4. Once you see a 5506 or 5507 event on the Gateway for the device, transfer the deviceupdate.log to a computer from the device. You must do this before the wipe happens or you will lose all the tracing information.

5. Open the client logging file on the computer in a text reading program like Notepad.

6. Search for the following terms in the document:

a. “Rejecting packet” – See below for Rejected packets

b. “Received push with timestamp” and “Accepting packet” -- See below for Successful packets section

c. No “Received push with timestamp” in the log

a. Rejected Packets

If you see the text “Rejecting packet” in the Alerter log, it means that the Alerter was rejecting the packet because the source address did not match the expected address.

Seeing this in the logs is actually a good sign, in that the Alerts are getting to the device, but they are being rejected.

In the log it will give the expected source IP (The IP address that the Mobile VPN client is communicating with) and the actual IP (The IP address that the packet came from). A mismatch here generally indicates one of two issues:

· There is a routing problem on the VPN Gateway

· There is a network element between the Gateway and the device performing Network Address Translation for the Gateway external IP address

· Check Again for a NAT in front of the GW

Check the actual IP address in the log? Is this address known to you? Does it look like something is performing a NAT on the Gateway IP?

Remove the NAT and try the wipe again.

· Check the routing on the Gateway

There may be an issue with the routing on the Gateway. Perform the following steps to diagnose the problem.

Determine the Address Pool

1. On a machine with the Administrator Tools installed on the corporate network, open the MDM Shell.

2. Run Get-MDMGatewayServer

3. For the Gateway Server(s) where devices are connecting, obtain the address pool which should be in the form of something like: {10.10.10.0/255.255.255.0}. It may contain multiple values.

Determine the routes on the Gateway Server

4. Log into the Gateway Server machine(s)

5. Open a Command Prompt on the Gateways and run the following command

>route print

6. The above command will print out a list of routes like the following:

>route print

IPv4 Route Table

===========================================================================

Interface List

……

===========================================================================

Active Routes:

Network Destination Netmask Gateway Interface Metric

0.0.0.0 0.0.0.0 aaa.107.247.aaa aaa.107.247.aaa 10

3.4.0.0 255.255.0.0 aaa.54.44.1 aaa.54.44.33 1

10.0.0.0 255.0.0.0 aaa.54.44.1 aaa.54.44.33 1

10.10.10.0 255.255.255.0 aaa.54.44.32 aaa.54.44.33 1

………continues……

===========================================================================

Persistent Routes:

Network Address Netmask Gateway Address Metric

aaa.23.64.227 255.255.255.255 aaa.54.44.1 1

aaa.27.52.13 255.255.255.255 aaa.54.44.1 1

aaa.27.52.33 255.255.255.255 aaa.54.44.1 1

aaa.29.28.102 255.255.255.255 aaa.54.44.1 1

aaa.29.28.136 255.255.255.255 aaa.54.44.1 1

………continues……

7. Look in the Active routes for routes with a Network Destination that matches the address pool(s) obtained in Step 3 above.

8. There should be only 1 route to each of the device address pool(s) configured on your Gateway(s). That route should go through the external interface on the Gateway.

For example if a Gateway has the following interfaces:

o aaa.54.44.33 – external interface

o bbb.192.0.1 – internal interface

The below is a correct route because it goes through the external aaa.54.44.33 interface:

Active Routes:

Network Destination Netmask Gateway Interface Metric

10.10.10.0 255.0.0.0 aaa.54.44.1 aaa.54.44.33 1

The below is an incorrect route because it goes through the internal bbb.192.0.1 interface:

Active Routes:

Network Destination Netmask Gateway Interface Metric

10.10.10.0 255.0.0.0 aaa.54.44.1 bbb.192.0.1 1

The below is incorrect as well. Although there is a route correctly through the external interface, there is also one through the internal interface.

Active Routes:

Network Destination Netmask Gateway Interface Metric

10.10.10.0 255.0.0.0 aaa.54.44.1 aaa.54.44.33 1

10.10.10.0 255.0.0.0 aaa.54.44.1 bbb.192.0.1 1

9. You can reboot the Gateway server and recheck the route. If the incorrect route still remains, you can correct the route using the “route” command. Run “route” at the command prompt for more information.

10. Also check for incorrect routes in the Persistent Routes section.

11. Once the routing is correct, try the troubleshooting steps again.

b. Successful Push Packet

If you see successfully received push packets to the Alerter it means that Alerts are getting to the device and they are being processed.

If you are still having issues with the wipe request and the time it is taking, you can check the Device Management server to ensure that it is functioning as expected.

c. No Successful Push packet in the log

If you do not see any successful push packets in the log, it means that the device is not receiving any alert requests. Check the routing on the Gateway by following the “Check the routing on the Gateway” steps above.

Check the networking between the Gateway server and the device and correct any networking issues.

Ideally, Remote Wipe Now should work smoothly. In the event there are problems, these steps should help you determine the cause.

Marc McClure

Program Manager

Mobile Information Worker

Remote Wipe Now and MDM Alerter troubleshooting

Dave Madison — Tue, 22 Jul 2008 03:07:00 GMT

Remote Wipe Now and MDM Alerter troubleshooting

A key security feature of SCMDM is the ability to wipe a device remotely. Often time is of the essence, so it is important to know if a wipe was successful or not. Here we will discuss how remote wipe works and how to troubleshoot it,

Introduction

Remote wiping of a device is a security feature that allows administrators to remotely send a command to an MDM managed device that causes it to erase all data and return to factory defaults. This is useful if the device has been lost or stolen. In a remote wipe, the flash and storage card data are overwritten, leaving only the base OS on the device and no user data. Administrators can also deploy the Self Service Portal which allows users to wipe their own devices.

For wipe operations, time is critical - that's why we tend to call the feature "Wipe Now" internally. Without the Now you just have a Wipe Soon operation. Wiping in 8 hours doesn't work great when your device and all its data are lost or stolen. Remote wipe commands in MDM depend heavily on networking between the server and the device and this is where problems can often occur.

In pilot environments administrators often run MDM Remote Wipe tests because it is easy to see it working. However, this is one area in deployment where it is easy to make mistakes and cause the Wipe to take longer than expected. Below are troubleshooting steps to help to determine the cause of a Wipe Now command taking longer than the expected ~~2-6 minutes~~. Failed to catch this before posting...expected time is 3-15 minutes.

How Remote Wipe Works

Before going into troubleshooting, here's a brief overview of how Wipe Now works.

An administrator or user submits a wipe request through the console, MDM Shell, or Self Service Portal.
The wipe request is stored in the DM Engine database for the device to pick up at its next scheduled OMA session

If we were to stop at this point it would be a "Wipe Soon" and the device would connect at its regular interval (4 hours, for example) and pick up the wipe. Wipes submitted are always submitted as a "Wipe Now" now command, and thus we have to go a step further

In parallel to adding the wipe request for retrieval, the wipe driver also calls the Alerter component to inform the device of a pending wipe request.
The Alerter sends an alert to the device over the Mobile VPN. The alert can only be sent through the VPN tunnel and thus Wipe Now requires VPN connectivity
The Alerter client on the device receives this Alert and immediately starts a management session with the Device Management server.
The device picks up its wipe request from the Device Management server, sends back an Acknowledgement that started the wipe, and starts the wipe process

Troubleshooting Wipes that are Taking Longer than Expected

From the moment a wipe request is submitted, a wipe should take approximately ~~2 - 6 minutes~~ 3-15 minutes] depending on the network and other factors. If it is taking longer than that, below are some troubleshooting steps that you can perform to determine the cause of the latent wipe request.

1. Verify that Management Sessions are operating as expected

Ensure that typical device management sessions, not related to wipe are working as expected. You can download the MDM Connect Now tool located in the SCMDM 2008 Resource Kit Client Tools package here: http://technet.microsoft.com/en-us/scmdm/cc304591.aspx Using this tool and associated documentation, you can verify that management sessions for devices are successful.

If management sessions are not working, you will need to fix this problem before devices can successfully receive the remote wipe command.

2. Verify the device is connected to the VPN

As discussed in the above "How Remote Wipe Works" section, remote wipe relies on the VPN being connected in order to send alerts to the device. The device may not be connected to the Mobile VPN for many reasons, but some of the most common are:

The device is switched off
The Mobile VPN is disabled (if users have the ability to disable it)
The user is roaming and VPN is off
The data connection is improperly configured
The user is temporarily out of service coverage area

In all the above cases, the device will not receive the Alert message as it relies on the VPN tunnel being up. When the device reconnects to the VPN, it will receive an Alert message or will start a management session immediately if it has missed its regularly scheduled session.

3. Verify the Gateway Server is not behind a Network Address Translator (NAT)

Once you have verified that management sessions are operating and the VPN was up on the device you were attempting to wipe, you need to check that the Gateway is not behind a Network Address Translator. MDM does not support locating a Gateway Server behind a NAT. There are several reasons for this requirement; one of the reasons has to do with the Alerter.

The reason the Alerter does not work with Gateway Servers behind a NAT is for security purposes. We'll talk about this some more, but for added security, the Alerter checks to ensure that the Alert it received is really from the MDM Gateway Server and not from a potential attacker.

The Gateway Server must have a public IP address and must not sit behind a NAT or the Alerter cannot verify the alert that the alert is valid. The Alerter discards invalid alerts. Ensure that your Gateway is not behind a NAT.

4. Check the Event Log on the MDM Gateway

On the MDM Gateway server, open up the "VPN Policy Engine" Event Log. Search for events 5507 and events 5506 in the log

Event 5506 indicates that the Alerter received a response from the device.

Event 5507 indicates that the Alerter sent a number of retries,but never heard from the device.

Further debugging information is needed for the following scenarios:

5507 events for every wipe issued

5507 event for a wipe issued in a controlled environment, where you know the device connected, online and available

Inconsistent 5507 and 5506 events

A mixture of 5507 and 5506 events are expected for normal operations as some devices may be offline, out of service range, or not connected for another reason. This is generally normal.

In our next post we will look at how to use the some client tools available in the resource kit, and how to use the Device Log to further narrow down any issues.

Marc McClure

Program Manager

Mobile Information Worker