Michael Niehaus' Windows and Office deployment ramblings
I mentioned a while back that I wanted to do a blog post talking about how to troubleshoot the new MDT 2012 monitoring feature for Lite Touch deployments, but first I had to actually describe it. If you haven’t reviewed that post, you might want to check it out first at http://blogs.technet.com/b/mniehaus/archive/2012/03/09/mdt-2012-new-feature-monitoring.aspx.
So now let’s talk about troubleshooting. First, let’s look at the server side. You have to enable monitoring on a computer that has MDT 2012 installed. When you use Deployment Workbench on that computer and check the box to enable monitoring, Workbench will first check to see if the specified monitoring host name is local:
It doesn’t really matter if you specify an IP address, a short host name, or a fully-qualified host name, as long as the clients can resolve whatever you specify. If you specify a name that Workbench doesn’t think is local (because Workbench itself can’t resolve the name back to an IP address assigned to the current machine), it won’t try to install the monitoring component; instead, it will try to contact that server to see if monitoring is running on that computer. If it is, great; if it isn’t, you’ll see an error message:
If you look closely at the error “tip” at the end of the “Monitoring host” line, you’ll see a message like “Unable to connect to the specified server and port”. If you think you specified the local computer name and got that error, then Workbench couldn’t figure out that it was the local computer name (something that’s harder to do than you might think). If you are specifying a different server and see this error, then it’s having problems communicating with that other server.
Tip #1: Make sure the name you specify in Workbench can be resolved to the IP address of the current machine.
You’ve checked the checkbox and can’t see that anything happened. So what was actually done? Two things:
Tip #2: Make sure the “Microsoft Deployment Toolkit Monitor Service” is installed and running. If it’s not installed and it should be, you can uncheck the box, click apply, then check the box again and click apply to reinstall it. If it’s installed but not running, try to start it.
Tip #3: Make sure the entry was added to CustomSettings.ini by looking at the Rules tab. Because of a peculiarity with the way Workbench works, if you make any changes to the Rules tab after you’ve clicked the “Enable monitoring” checkbox but before you’ve clicked OK, it’s possible that the changes made on the Rules tab overlay the EventService entry in CustomSettings.ini, but it’s easy enough to put it back manually.
The service has two dependencies:
So there’s no dependency on IIS or SQL Server. The service uses .NET to host a web server as part of the service process, and it uses a SQL Compact database (basically a set of DLLs, which ship with MDT, that run in the service process) to store the monitoring information. It’s designed to be easy to install and run.
Tip #4: If you try to start the service and it won’t start, that most likely means the ports you chose were already in use. (If you want to know what’s using the ports, use a tool like TCPView, available from http://technet.microsoft.com/en-us/sysinternals/bb897437.) Pick different ports.
While it’s always possible that there could be some other reason the service fails, I haven’t seen any other causes. But if you know the ports are not in use and the service still won’t start, capture a trace using DebugView (http://technet.microsoft.com/en-us/sysinternals/bb896647) to see if it provides any further clues. If not, contact Microsoft Support for assistance.
The monitoring service listens on the two ports that you specified. The first of these ports (9800) is used by computers being deployed to send progress events. The second (9801) is used by Workbench itself to query information about deployments being monitored. To make sure these ports are accessible, we can manually connect to each one using Internet Explorer.
To verify the “event port” from the monitor server itself, you can use a URL such as:
http://localhost:9800/MDTMonitorEvent/
If that works, you should see a response like:
That’s a proper response in this case – the web service doesn’t expect to be called in this way (an HTTP GET request instead of an HTTP POST request), so it’s telling you the proper way to call the service.
To verify the “data port” from the monitor server itself, you can use a URL such as:
http://localhost:9801/MDTMonitorData/
This response (which is an ODATA feed in case you are curious) confirms that the data feed is working as expected.
But those are the easy queries – they are using “localhost”, which is almost never subject to firewall restrictions. Next, you need to try these queries remotely, using the appropriate “remote” URLs:
http://mdt-server:9800/MDTMonitorEvent/
http://mdt-server:9801/MDTMonitorData/
If those work, great. If they don’t, then you need to make sure that whatever firewall is running on the monitoring server allows the ports you specified (e.g. 9800 and 9801) to be accessed from remote hosts.
Tip #5: Make sure you can access the monitor service ports both locally and remotely. Adjust the firewall rules as necessary.
Note that there are other networking “challenges” that can get in the way, e.g. IPSec domain isolation. In this configuration, computers that aren’t domain-joined, e.g. running from Windows PE, can’t talk to domain-joined computers because they aren’t using encrypted IPsec communication. This type of configuration will never work – you would need to set up the monitoring service on a “boundary server” that has been configured to allow non-IPsec traffic on the configured ports. So don’t assume that if a “remote” URL works from a domain-joined machine, it will also work from a workgroup machine (or Windows PE) – know how your network is configured.
When the EventService task sequence variable is set (via the processing of CustomSettings.ini), each MDT script executed in the task sequence will send an event to the monitor service on the “event port” URL. When this succeeds, you’ll see a message like this:
If a script is unable to send an event, you’ll see something different:
That’s a clear sign that something isn’t right. Make sure the service is running, that the firewall ports are open, etc. – the same challenges we already reviewed.
Tip #6: Check the client logs to make sure the clients are able to talk to the monitoring service.
Another way you might notice an issue: If the monitor service isn’t running, the clients will still try to connect to it, eventually timing out. This timeout process will cause a delay at the end of each step in the task sequence, so if you are watching the task sequence progress dialog, you’ll see steps that you never noticed before (because they usually run so fast) now taking a long time.
When you try to look at the monitoring data from Workbench, it calls a PowerShell cmdlet (Get-MDTMonitorData), then that PowerShell cmdlet makes the “data port” query to retrieve the details from the monitoring service. If the service is working as expected, you can see the list of monitored machines in Workbench. If the service isn’t working, you’ll see something like this instead:
Good advice, make sure the service is running
Still having issues? Post them as comments here, or send me an e-mail at mniehaus@microsoft.com and we’ll try to figure out what’s going on.
Now that Configuration Manager 2012 RTM is available, people will certainly start installing it, both in their labs and in production – and will start finding out that the SQL version check in the ConfigMgr installer is looking for some very specific SQL maintenance levels. (I’ve pulled out plenty of my own hair doing this recently, just like I did with the prerelease versions.) The supported versions can be found at http://technet.microsoft.com/library/gg682077.aspx#BKMK_SupConfigSQLDBconfig. The summary (always subject to change):
If you are running SQL 2008 SP2, install CU9 from http://support.microsoft.com/kb/2673382/. If you are running SQL 2008 SP3, install CU4 from http://support.microsoft.com/kb/2673383/. If you are running SQL Server 2008 R2 SP1, use CU6 from http://support.microsoft.com/kb/2679367/. If you aren’t yet on the needed service pack levels, upgrade now, then install the cumulative updates :-)
I’m still looking for a statement on ConfigMgr 2012 RTM and SQL Server 2012. (ConfigMgr 2007 now supports it, see http://blogs.technet.com/b/configmgrteam/archive/2012/04/17/configuration-manager-support-announcements-for-april-2012.aspx.) There’s already a CU1 for SQL Server 2012, http://support.microsoft.com/kb/2679368/.
I’ve done a few sessions now where I demonstrated how to put some hooks into a task sequence so that you can single step through the process and “poke around” to see exactly what is going on. Since I’ve promised to make those scripts available, and will get lots of e-mails from people if I don’t do that, I’ll provide those at the bottom of this blog posting.
But the scripts by themselves require some explanation, some instructions, and some caveats.
First, let’s talk about the basic setup. If you look at a ConfigMgr task sequence, it contains a number of steps and groups that define the sequential process that is going to be executed. This is actually converted into XML, where each step and group becomes a series of XML entries that define:
Before the client computer begins to execute the task sequence, it receives (as part of the task sequence policy) the entire XML blob. That XML blob is then processed by the Task Sequence Manager, TSManager.exe, which is what actually starts the commands for each step (at least those where it has decided the conditions have been met).
So that’s a key part of what we want to do: We want to intercept each of the commands that are being executed by the TSManager.exe process before they actually start. That way, we can look at the task sequence variables currently defined, the command line details, the step’s XML, etc. How do you do this intercepting? That’s where a debugger comes in. A debugger can attach the TSManager.exe process, then set a breakpoint on the Windows API call, CreateProcessW, that is used by TSManager to run the command for each step. Each time TSManager.exe then tries to run a new command, the debugger will stop the process until you manually say continue.
That’s where the scripts come in: The provided “InsideTS” scripts take care of setting up the debugger (attaching to the TSManager.exe process, setting the breakpoint), watching for breakpoints (checking the log for “breakpoint hit” messages), and showing the current state (displaying the step details, task sequence variables, logs, etc.). The pieces involved:
In addition to the files listed above, you need to gather these additional files:
Once you have gathered all of these files into one folder, they need to be added into the ConfigMgr boot image that you will be using and into the new OS image that you are deploying. Create an “InsideTS” folder at the root of both of these. (The simplest way to do this: mount both of the WIMs, XCOPY the folder to X:\InsideTS, commit the changes, then update the DPs.) Why do I include these files in the images? Well, it was easier to do it that way. (More on that later.)
Now that the files have been added to the boot and OS images, you then need to add commands to the task sequence to run InsideTS_Attach.wsf. How many steps you need to insert into this depends on how many reboots you expect the task sequence to perform; you need one after each reboot. At minimum, you will need two, one to run in Windows PE for preinstallation steps and a second to run in the new OS. The step should look like this:
Assuming you have set everything up right, you should be able to start the task sequence and see something like this show up:
This shows the step after the “InsideTS Attach” as being the next one to execute. You can then click on the “Variables” button and see that all of the variables are highlighted in green because they are all considered “new” (as in “never seen before”) at this point:
Once you’ve poked around enough, click the “Finish” button to close the wizard. At that point, you’ll see a strange sight: the NTSD debugger. In a perfect world, the InsideTS_Monitor.wsf script would feed the commands to continue with the step execution once you’ve clicked “Finish” on the wizard, but after messing around with that for a couple of hours, I gave up on that and went with a lower-tech approach: Type in the needed commands yourself. In case you aren’t familiar with NTSD command line options, here’s a quick reference to the ones you’ll need to know:
That’s really all you need to know. (I used one other command, “du rdx”, to dump out the CreateProcessW command line parameter during the session, but you can see the same value at the end of the SMSTS.LOG, which you can open by clicking the “Log” button in the wizard.)
Some people know the story behind this little tool: I originally put these scripts together for a BDNA “SCCM Guru” webcast back in November. The webcast was scheduled months in advance, so I assumed I had plenty of time to prepare. For a variety of reasons (including procrastination) I didn’t prepare for this nearly as soon as I had hoped. In fact, I didn’t start most of it until about 30 hours before the presentation was due to start. And for about 18 of those hours, I was travelling back to Seattle from Moscow – in coach of course. So all the preparation work was done on the airplane, with the scripting done on the Moscow to New York flight, when there was no internet connection, and the testing (and bug fixing) happening on the flight from New York to Seattle (VPN connection from the airplane to my office Hyper-V server, thank you http://www.gogoair.com). The next morning, I presented the session, completely jetlagged…
So there’s my excuse for the following caveats:
This script code is provided as is with no guarantee or warranty concerning the usability or impact on systems and may be used, distributed, and modified in any way provided the parties agree and acknowledge the Microsoft or Microsoft Partners have neither accountability or responsibility for results produced by use of this script. Microsoft will not provide any support through any means.
If you manage to set this up successfully, or if you run into any issues, please e-mail me at mniehaus@microsoft.com to let me know. I think the instructions and scripts should work, but I offer no guarantees
System Center 2012 Configure Manager includes an application catalog web site, which allows end users to select the applications that they want to install. As part of this, they provide the ability to specify that approval is required before the software can be installed, a required feature in most enterprises. If you aren’t familiar with the application catalog, the blog posting at http://blogs.technet.com/b/ptsblog/archive/2011/12/20/configuration-manager-2012-rc-configure-software-catalogue-portal-and-publish-applications.aspx goes through the flow (although it’s slightly out of date, it gets the point across).
But out of the box, approvals come from exactly one source: the ConfigMgr administrator. What if you want others to do the approvals (mainly because you have better things to do)? That’s where the newest free solution accelerator comes in. The Application Approval Workflow solution accelerator was announced on the Service Manager blog at the beginning of this week:
http://blogs.technet.com/b/servicemanager/archive/2012/03/26/application-approval-workflow-aaw-solution-accelerator-beta-now-available.aspx
The solution works by leveraging three different System Center 2012 products:
The beta ends soon, so download quickly. Expect to see more about this at MMS 2012 too.
I was going to do a blog posting talking about how to troubleshoot issues with the new monitoring feature available in MDT 2012 for Lite Touch deployments, but then I realized I’ve not yet done an initial post talking about the feature (although I did mention it in a previous blog posting talking about DaRT integration). So I guess I need to start with more of an overview.
Over the years, there have been requests for a way to see what deployments are presently in progress. Way back when, we had a MOM management pack that tried to do this, but there were numerous challenges with that so we ended up removing it – we needed something much simpler. So with MDT 2012, we implemented something simpler to monitor Lite Touch deployments. To enable this, just check one box in the deployment share properties:
When you do this, two things happen:
That’s all there is to it. Once you do that, you should be able to track all subsequent deployments via the “Monitoring” node in Deployment Workbench:
If you look at the properties of any of the computers being monitored, you can see the details:
You’ll notice that the display automatically updates every 10 seconds, so you can watch the computer progress. Also, there are three possible buttons (two of which are shown below):
A few other details:
That’s the quick overview.
For those of you who have already installed MDT 2012 RC1, you might notice that there is a new behavior in Deployment Workbench: It will keep the list of items (applications, drivers, OS packages, task sequences) in alphabetical order.
Now when you first install MDT 2012 RC1, you might notice that the items aren’t sorted right away – you have to change something in the folder first. As soon as you do (e.g. add an item, rename an item), the items in that folder will be sorted.
Keeping the list of folders sorted is a much bigger challenge, so at this point they will still show up in the order that they were added. That’s something we will have to look at again in a future version.
Now that Configuration Manager 2012 RC2 is available, people will certainly start updating their labs – and finding out that the SQL version check in the ConfigMgr installer is looking for some very specific SQL maintenance levels. (I’ve pulled out plenty of my own hair doing this recently.) The release notes cover the supported versions:
Supported versions of SQL Server 2008 for RC2: SQL Server 2008 SP2 Standard and Enterprise CU7 SQL Server 2008 R2 SP1 and CU4 SQL Server Express 2008 R2 and CU4
Supported versions of SQL Server 2008 for RC2:
If you are running SQL 2008 SP2, install CU7 from http://support.microsoft.com/kb/2617148. If you are running SQL Server 2008 R2 SP1, use CU4 from http://support.microsoft.com/kb/2633146. If you aren’t yet on the needed service pack levels, upgrade now, then install the cumulative updates :-)
Many of you have customized the MDT 2010 wizards, and I expect that will be fairly common with MDT 2012 as well – after all, one of the design goals with having HTA-based wizards is to enable customization by creative IT pros. But the process is going to be slightly different with MDT 2012.
First, let’s review the basic process most of you would go through with MDT 2010:
So it’s quite reasonable for you to try the same thing with MDT 2012. But you’ll quickly discover that you can’t get very far. You can open the file, but as soon as you click on one of the wizard panes in the left-hand column, you get this exception:
So what causes this? Well, if you look at the files in the deployment share, you’ll notice that there are quite a few more that start with “DeployWiz”. That’s because the entire wizard has been restructured. Instead of having one set of large files (DeployWiz_Definition_ENU.xml, DeployWiz_Initialization.vbs, DeployWiz_Validation.vbs), there are now separate files for each wizard pane (e.g. DeployWiz_ComputerName.xml and DeployWiz_ComputerName.vbs). The original files (DeployWiz_Definition_ENU.xml, etc.) are still around, but are much smaller – the bulk of the content has been separated out.
Why was this done? Mainly because it makes the wizard code much easier to maintain. It’s now much more obvious what script pieces are used for which wizard panes. Additionally, it makes the wizard easier to test, as you can work on a single pane at a time without worrying about breaking another unrelated pane.
The individual panes are then tied together by the DeployWiz_Definiton_ENU.xml file. If you look at this file, you’ll see that it’s pretty short, with entries like this:
<Pane id="SelectTaskSequence" reference="DeployWiz_SelectTS.xml"> <Condition><![CDATA[UCASE(Property("SkipTaskSequence"))<>"YES" ]]></Condition> </Pane>
All that is there is a reference (or link) to the separate files for each wizard pane, along with the conditions for when each wizard pane should be displayed. (This helps with the testing as well: By keeping the conditions in the DeployWiz_Definition_ENU.xml file instead of in the individual wizard files, those conditions don’t get in the way of “offline testing”, e.g. running just that one single wizard pane without going through a full deployment.)
So why does that cause problems with the Wizard Editor? Simple: It doesn’t understand that the wizard pane body is in a separate file. It can’t follow the reference link (highlighted in yellow above) to the separate wizard pieces. So does that mean the Wizard Editor is no longer useful? Not at all – you can still use it to work on (and test) individual pages like DeployWiz_ComputerName.xml:
But you’ll need to edit the DeployWiz_Definition_ENU.xml file by hand.
I’m working on a new version of the Wizard Editor in my “free time” so that it learns how to follow these links (with other improvements added in too), but it might take a while before I can complete that work.
So what does the recommended workflow look like if you wanted to add a new wizard pane to MDT 2012? Here are the basics:
This new flow has another advantage too: It makes it much easier to integrate a new wizard pane into MDT. You don’t have to worry about reintegrating your changes into the “big” XML file each time you upgrade MDT – your separate files will continue to exist, untouched, so all you will need to do is add the “link” into DeployWiz_Definition_ENU.xml again and you’re good to go.
One of the steps in the Configuration Manager installation process is to manually create the “System Management” container in Active Directory, then give the ConfigMgr computer account the ability to create objects in it. Yes, even with Configuration Manager 2012, this is still something that needs to be done manually.
So that was this evening’s challenge: Automating that seemingly simple task. As with all automation tasks, you always hope that someone has already solved the problem. But even with searching multiple search engines (something that always pains me), I didn’t really find what I was looking for. (No executables, no third-party tools, no ugly ADSI code, and ideally no VBScript – PowerShell is the future.) So I created a new PowerShell script, incorporating bits and pieces from several other scripts. The basic steps:
Sounds simple enough, and except for the ACL part, it is. The complete script:
#Requires -version 2.0 # *************************************************************************** # # File: SystemManagement.ps1 # # Version: 1.0 # # Author: Michael Niehaus # # Purpose: Create the AD "System Management" container needed for # ConfigMgr 2007 and 2012, and grant access to the current # computer account. # # This requires PowerShell 2.0 and Windows Server 2008 R2. # # Usage: Run this script as a domain administrator, from the ConfigMgr # server. No parameters are required. # # ------------- DISCLAIMER ------------------------------------------------- # This script code is provided as is with no guarantee or waranty concerning # the usability or impact on systems and may be used, distributed, and # modified in any way provided the parties agree and acknowledge the # Microsoft or Microsoft Partners have neither accountabilty or # responsibility for results produced by use of this script. # # Microsoft will not provide any support through any means. # ------------- DISCLAIMER ------------------------------------------------- # # *************************************************************************** # Load the AD module Import-Module ActiveDirectory # Figure out our domain $root = (Get-ADRootDSE).defaultNamingContext # Get or create the System Management container $ou = $null try { $ou = Get-ADObject "CN=System Management,CN=System,$root" } catch { Write-Verbose "System Management container does not currently exist." } if ($ou -eq $null) { $ou = New-ADObject -Type Container -name "System Management" -Path "CN=System,$root" -Passthru } # Get the current ACL for the OU $acl = get-acl "ad:CN=System Management,CN=System,$root" # Get the computer's SID $computer = get-adcomputer $env:ComputerName $sid = [System.Security.Principal.SecurityIdentifier] $computer.SID # Create a new access control entry to allow access to the OU $ace = new-object System.DirectoryServices.ActiveDirectoryAccessRule $sid, "GenericAll", "Allow", "All" # Add the ACE to the ACL, then set the ACL to save the changes $acl.AddAccessRule($ace) Set-acl -aclobject $acl "ad:CN=System Management,CN=System,$root"
#Requires -version 2.0
# *************************************************************************** # # File: SystemManagement.ps1 # # Version: 1.0 # # Author: Michael Niehaus # # Purpose: Create the AD "System Management" container needed for # ConfigMgr 2007 and 2012, and grant access to the current # computer account. # # This requires PowerShell 2.0 and Windows Server 2008 R2. # # Usage: Run this script as a domain administrator, from the ConfigMgr # server. No parameters are required. # # ------------- DISCLAIMER ------------------------------------------------- # This script code is provided as is with no guarantee or waranty concerning # the usability or impact on systems and may be used, distributed, and # modified in any way provided the parties agree and acknowledge the # Microsoft or Microsoft Partners have neither accountabilty or # responsibility for results produced by use of this script. # # Microsoft will not provide any support through any means. # ------------- DISCLAIMER ------------------------------------------------- # # ***************************************************************************
# Load the AD module
Import-Module ActiveDirectory
# Figure out our domain
$root = (Get-ADRootDSE).defaultNamingContext
# Get or create the System Management container
$ou = $null try { $ou = Get-ADObject "CN=System Management,CN=System,$root" } catch { Write-Verbose "System Management container does not currently exist." }
if ($ou -eq $null) { $ou = New-ADObject -Type Container -name "System Management" -Path "CN=System,$root" -Passthru }
# Get the current ACL for the OU
$acl = get-acl "ad:CN=System Management,CN=System,$root"
# Get the computer's SID
$computer = get-adcomputer $env:ComputerName $sid = [System.Security.Principal.SecurityIdentifier] $computer.SID
# Create a new access control entry to allow access to the OU
$ace = new-object System.DirectoryServices.ActiveDirectoryAccessRule $sid, "GenericAll", "Allow", "All"
# Add the ACE to the ACL, then set the ACL to save the changes
$acl.AddAccessRule($ace) Set-acl -aclobject $acl "ad:CN=System Management,CN=System,$root"
The same script is attached.
Sometimes it’s the little things that most people fail to notice that need some attention. In this case, we’re talking about new task sequence variables automatically set by the MDT “Gather” process (specifically, the ZTIGather.wsf script). There are a few new ones in MDT 2012:
We also made a few other related changes: