The below article outlines the steps that should be taken if a Task Sequence is deployed to an incorrect set of computers. Since mistargeting a Task Sequence could result in total data loss on the computers that were mistargeted, time is of essense to remedy the problem. It is also important to know not only what steps to take to remedy the problem, but also what steps not to take. Some steps which at first may seem as good steps to take, such as deleting the Advertisement for the Task Sequence, end up actually hindering efforts to remediate the issue.
For these reasons it may be a good idea to review this article and become familiar with the overall process at a high level. In this way if the issue ever occurs, you will be able to quickly and correctly respond to the issue.
If one of the following actions occurs:
then an OSD Task Sequence can accidentally start running on unintended computers. This can possibly cause total data loss on the computers that the Task Sequence runs on.
In order to remediate the situation there are several actions that need to be taken and also several actions should also NOT be taken. The actions not to be taken are equally as important as those that should be taken, so it is important to understand both sets of actions and follow the actions accordingly. This article outlines and details those actions.
IMPORTANT! For time sensitive remediations where immediate preventative actions need to be taken, take ONLY the following two actions:
For detailed instructions on the above two actions, follow the two corresponding sections:
As soon as the above actions have been taken, make sure to review the section "Steps NOT To Take To Remediate An Accidental OSD Task Sequence Deployment" to make sure certain actions are NOT taken that may make the situation worse.
Steps NOT To Take To Remediate An Accidental OSD Task Sequence Deployment
Before taking steps to remediate an accidental OSD Task Sequence deployment, the following actions should NOT be taken to try and remediate the situation:
The above actions will not always stop the Task Sequence on computers that have already received the policy to run the Task Sequence. More importantly, the first four actions will cause some or all of the history of the Task Sequence advertisement to be lost. This will result in loss of reporting information including what computers received the Task Sequence policy and what computers ran the Task Sequence. This information is needed to help remediate computers.
The last action, deleting the Boot Image from the ConfigMgr 2007 admin console, should not be taken so that there is a record of how the Boot Image is configured and what Distribution Points the Boot Image is on. This information is also important and used during remediating tasks.
If any of the above actions have already been taken, continue following the below sections. The majority of the actions will still apply. The main limitation caused by taking any of the above actions is that computers that have received policy for the Task Sequence may not be able to be identified for possible remediation.
Stop A Task Sequence From Running On Computers That Have Not Received Policy For The Task Sequence
The first steps to take to stop a Task Sequence from running on additional computers are DISABLE (not delete) the Task Sequence and change the properties of the Advertisement for the Task Sequence
Steps 1-4 are the equivalent as disabling the Task Sequence by checking the option "Disable this task sequence on computers where it is advertised" under the "Advanced" tab in the properties of the Task Sequence. Steps 5-9 are not absolutely necessary once the Task Sequence has been disabled but added as an extra precaution.As the message in Step 2 above indicates, the above actions will prevent the Task Sequence from running on computers that have not yet received the policy to run the Task Sequence or that receive updated policy not to run the Task Sequence. However it does not prevent the Task Sequence from running on any computers that have already received the policy to run the Task Sequence but which have not received updated policy. To prevent these computers from being wiped by the Task Sequence, additional actions are necessary.
Prevent A Task Sequence From Running Successfully On Computers That Have Received Policy For The Task Sequence
Unfortunately for computers that have already received the policy to run the Task Sequence there may not be a reliable way to stop the Task Sequence from running. The computer may receive updated policy that disables the Task Sequence, but there are no guarantees that the computer will receive updated policy before the Task Sequence actually runs.
As a workaround, instead of relying on the policy on the computer to be updated, the Task Sequence can be set to fail purposely on the computer before it is wiped. This will cause the computer not to be wiped.
In order to cause the Task Sequence to fail on these computers, the Boot Image associated with the Task Sequence can be removed from all Distribution Points (DPs). As long as the Boot Image has not yet downloaded on the computer, either as part of the current Task Sequence OR another past Task Sequence, deleting the Boot Image from the Distribution Point will cause the Task Sequence to fail on any computer that tries to run the Task Sequence .
The wipe of the computer's hard drive occurs either at the "Format and Partition Disk" task or the "Apply Operating System Image" task, both which take place when the computer is in WinPE. The computer does not reboot into WinPE until the Boot Image has been downloaded and staged locally on the hard drive. Downloading and staging of the Boot Image takes place at the "Restart Computer" task (sometimes labeled as "Restart in Windows PE"). If the Boot Image associated with the Task Sequence is deleted from the Distribution Points, when the "Restart Computer" ("Restart in Windows PE") task runs, it will fail to download the Boot Image, causing the "Restart Computer" task to fail, and subsequently, the Task Sequence to fail.
To delete the Boot Image associated with the Task Sequence from all the DPs:
Once the Boot Image is deleted from all Distribution Points, the Task Sequence should no longer run successfully on any computer as long as the computer has not already downloaded the Boot Image.
IMPORTANT! DO NOT put the Boot Image back on ANY Distribution Point again at any point, even in the future. For further information, see the section "Setting Up The Task Sequence Again".
As an extra precaution, the steps in the next section "How To Manually Delete And Verify Deletion Of The Boot Image From The Distribution Points" may want to be followed to ensure that the Boot Image has been completely deleted from the Distribution Points.
How To Manually Delete And Verify Deletion Of The Boot Image From The Distribution Points
Since time is of essence to prevent any additional computers from running the Task Sequence successfully, if the Boot Image is taking a long time to delete from the Distribution Points, or if the Distribution Points are at sites with slow links, then it may be best to delete the Boot Image manually from the Distribution Points. The longer the Boot Image stays on the Distribution Points, the greater the risk that additional computers may run the Task Sequence and possibly cause data loss on the computers.
Manual deletion of the Boot Image from the Distribution Points may also be needed under certain scenarios where the ConfigMgr 2007 site server is one of the computers that has been wiped by the Task Sequence. At this point it may not be possible to delete the Boot Image from Distribution Points via the ConfigMgr 2007 admin console, so a manual deletion of the Boot Image will be necessary.
To manually delete and verify deletion of the Boot Image from the Distribution Points:
Identify Computers That May Need Additional Remediation
Although disabling the Task Sequence and deleting the Boot Image from the Distribution Points may stop the majority of computers from being wiped, it may not stop 100% of the computers in the environment from being wiped. For example, if the computer has already downloaded the Boot Image before the Boot Image was deleted off the Distribution Point, the Task Sequence may still run on the computer. The following are scenarios where this could happen:
To help identify computers that may possibly be in this state, the report "Status summary of a specific task sequence advertisement" (Report ID 145) can be run. Please note that if the Task Sequence, Advertisement for the Task Sequence, Collection, or Computer Objects have been deleted, this report may no longer be available or may not be accurate. This is the reason why it is important NOT to delete these items.
To run the report "Status summary of a specific task sequence advertisement" (Report ID 145):
A report window will come up showing the current status of the Task Sequence deployment.
IMPORTANT NOTE! Although ConfigMgr reports are a good resource to help identify computers that are in a state that need remediation, it is NOT a real time system. The report is only as good as the last status message sent by a computer. Please keep this in mind when identifying computers that may need remediation.
The following are statuses that will be seen in the report, what they mean, and whether the computer will need further remediation:
Please keep in mind that all of the above statuses are based on the last time that the computer reported up its status. It should NOT be considered real time. The report should be refreshed regularly to ensure up to date reporting. Please also remember that computers that were shut off or taken off the network may have taken a particular action (i.e. started the Task Sequence) but have not had the chance to report up their current status. Since the computers are currently off or disconnected from the network, they will not report up their status until they are turned back on or plugged back into the network.
Remediating Computers Where The Boot Image Has Already Downloaded But The Task Sequence Has Not Wiped The Computer
Although there is a very short window of opportunity, there may be scenarios where a computer had downloaded the Boot Image, but not yet wiped the contents on the hard drive. In these scenarios the computer has already run the "Restart Computer" ("Restart in Windows PE") task, but not the "Format and Partition Disk" task or the "Apply Operating System Image" task. Such a scenario could have occurred if the computer was manually shut off at the correct moment to prevent the computer from continuing the Task Sequence. In these scenarios there is not a way to resolve the issue centrally from a site server, but the issue can be resolved locally at the computer through manual intervention.
To resolve the issue, a file can be deleted off of the hard drive of the computer that will cause the Task Sequence to fail before the computer is wiped. Once the Task Sequence fails, it will clean up after itself, including restoring the boot manager from WinPE back to the preexisting Windows OS.
If the Boot Image is in the local ConfigMgr client cache, the Task Sequence could possibly start up again even after initially causing the Task Sequence to fail. For this reason, in addition to causing the Task Sequence to fail, the contents of the local ConfigMgr client cache directory should also be deleted. This scenario is common if the "Distribution Points" setting in the properties of the Advertisement for the Task Sequence was set to "Download all contents locally before start task sequence". It could also happen if the Boot Image was downloaded to the local ConfigMgr client cache as part of a previous Task Sequence that ran on the computer.
Before running the below steps, make sure that the below two sections have already been followed:
To cause the Task Sequence to fail and to delete the contents of the local ConfigMgr client cache, follow the below steps:
When the computer restarts, it will boot into WinPE from the hard drive. Once it finishes booting WinPE, the Task Sequence will continue but will immediately fail due to the missing TSEnv.dat file.
Once the Task Sequence fails it will take the following actions:
These actions should only take a few seconds to occur and there may not be any visual indicators of these actions taking place.
Remediating Computers When the Advertisement Is Set To "Download all contents locally before start task sequence"
If the "Distribution Points" setting in the properties of the Advertisement for the Task Sequence was originally set to "Download all contents locally before start task sequence", then additional actions may be necessary.
In these scenarios, if the computer has both received the policy to run the Task Sequence and downloaded the content for the Task Sequence, primarily the Boot Image, then disabling the Task Sequence on the site server and deleting the Boot Image from the Distribution Points may not prevent the computer from running the Task Sequence.
Unfortunately in these scenarios there is nothing that can be done from the server side that guarantees stopping the Task Sequence from running on these computers. The only way to ensure the computers are remediated is by resolving the issue physically at the local computer and deleting the local ConfigMgr client cache directory.
Before taking the below steps, please make a note of the following two items:
To remediate computers that are still in the full Windows OS but have downloaded the Boot Image into the local ConfigMgr client cache:
Once the Boot Image has been deleted from the computer's local ConfigMgr client cache and from the Distribution Points, the Task Sequence will not have access to the Boot Image and will fail if it attempts to run.
Setting Up The Task Sequence Again Once Remediation Steps Are Completed
Once all of the remediation tasks have taken place, if the Task Sequence is still needed, it is recommended NOT to reuse the following items:
This is to prevent the possibility of the Task Sequence accidentally running on additional computers. For example, if a computer that was shut down or taken off the network after receiving the policy to run the Task Sequence, it could possibly still run the Task Sequence months later when the computer is turned back on or placed back on the network if the resources for that Task Sequence were once again available. If the resources for that Task Sequence are never made available again, even if months later, then it is still assured that the Task Sequence would fail and the computer would not be reimaged.
Instead of reusing the different components of the Task Sequence, the components can be recreated by taking the following actions:
Before taking the below steps, make sure that any of the actions that caused the issue to initially happen have been resolved. For example:
Create A New Boot Image Package
Recreate The Task Sequence
Please note: If the original Boot Image was also used in other Task Sequences, make sure to go into the properties of those Task Sequences and select the newly created Boot Image instead.
Create a new Advertisement for the new Task Sequence
To create a new Advertisement for the newly created duplicated Task Sequence, follow the standard procedures to create a new advertisement for a Task Sequence:
Frank RojasSupport Escalation Engineer
This is great article Frank, thanks for taking the time to put this together.
Wonderful work done !!!
Great stuff! One needs a cool head when things go wrong, and this article is the cool head. Need to create a shortcut to this :D
Hi there Frank.
Great article and fab work yet again. Of course it's rather important that folks read this BEFORE they inadvertently deploy the Task Sequence to the All Systems collection.
I find that there are a couple of preventative measures that one can take in order to minimise the impact of an incorrectly deployed Task Sequence:
1. You can create a "one time" maintenance window in the past on the All Systems Collection and make this apply to just OSD/Task Sequence events. Now if someone inadvertently deploys a task sequence we'll see lots of status messages saying the the task sequence could not be executed but at least we don't see the issue. Of course, in order to deploy the OS you now need to create a maint-window for the near future.
2. You can create a collection which has a Task Sequence Variable associated with it. Let's call it "OS installation allowed" and set it to "No". The first step of the task sequence checks for the value of "OS installation allowed" and throws an error if it is wrong.
3. If you are in a PXE boot environment then you can take advantage of the MAC ignore list and put in here a list of the sensitive MAC addresses or servers etc.
Nice suggestions! I actually consided adding a section to the article on how to prevent the problem from happening again, but the article was already long enough so I'm saving it for a future article.