Knock, knock! Who's there?

FIM 2010 Self-Service Password Reset ActiveX Misconfiguration Issues

2012-04-11T09:01:10Z

This post is about some of the errors you may find when setting up SSPR with Forefront Identity Manager 2010. The main resource to setup SSPR is the guide found at http://technet.microsoft.com/en-us/library/ee534892(v=WS.10).aspx.

Although the guide is very complete and detailed step-by-step, there are a few issues you may find if you misconfigure the Add-Ins and Extensions during its setup.

When installing the FIM Add-Ins and Extensions package, it will prompt you for “FIM Service Server Address”. Here you have to type the FQDN of the FIM Service server, without protocol or port. You can check "C:\Program Files\Microsoft Forefront Identity Manager\2010\Service\Microsoft.ResourceManagement.Service.exe.config" file and find externalHostName in there.

Then it will prompt for FIM Portal sites. This is important because having it misconfigured leads to the errors described later. In a single-box setup, you will have both FIM Service and FIM Portal in the same FQDN. However, in more complex architectures you will be separating FIM Portal and FIM Service in separate URLs. So it is important that you get the right servers in the right configuration dialogs.

If you have your externalHostName as fimservice.corp.contoso.com, but your FIM Portal is running at https://fimportal, you have to enter in this second dialog both the FQDN and the NetBIOS name of your FIM Portal server, as end users may be accessing it, separated by semicolon.

If you mistake like this when configuring with the FIM Service site instead of FIM Portal site (when different URLs), like the following, you will have problems initiating the ActiveX controls.

FIM Portal site is OK, but the FIM Service address is not.

In this case, the ActiveX component will initialize properly, but remain with buttons grayed out, cannot close the window, and the dialog gets hanged, so you will have to kill iexplore.exe process.

The attempt to access the Q&A gate through logon screen link is however more explicit, throwing the error “A service proxy exception was encountered while running the Password Reset Application. Error Text: An unexpected error has ocurred. Please contact helpdesk or your administrator. Error Code: 40007”.

FIM Service address is OK but FIM Portal site is not

In this case, SSPR functionality will work OK if you initiate registration using c:\windows\system32\MsPwdRegistration.exe program, and reset password through logon screen link. However, all attempts to register or reset through FIM Portal will fail, no matter what you do with your browser settings. The errors you will find if FIM Portal addresses get misconfigured include the ones described below.

“Registration is not possible at this time. The FIM Password and Authentication Extensions experienced an error when trying to launch registration for password reset. Please reinstall or contact your system administrator”.

When user click on “Register for password reset” link, this error window pops up, and a browser information bar shows “Internet Explorer blocked an ActiveX control, so this page might not display correctly”.

“Authentication operation failed”

Going directly to the registration workflow page at https://fimportal/identitymanagement/aspx/authn/AuthNWFUserRegistration.aspx, the yellow banner about blocked ActiveX appears and disappears, and then above message is shown in red when user click on “Register” button.

“The FIM Password and Authentication Extensions experienced an error when trying to reset a password. Please reinstall or contact your system administrator”.

Through https://fimportal/PasswordPortal, you will get the error above in red text once type a username and hit “Reset Password” button.

On the other hand, even when Add-Ins and Extensions are properly configured, a weird issue occurs when using the https://fimportal/PasswordPortal and users do not enter domain\accountname as the picture above shows. When not including the domain, the FIM Password Reset shows up, but hangs and “ghosts” when moving the window around

Hope this helps!

Enabling Safe Mode in Windows 8 Developer Preview

2011-09-16T19:53:49Z

Just a quick post to describe how to enable Safe Mode in Windows 8 Developer Preview.
If you are reading this post, probably you have already tried to boot in Safe Mode for whatever reason using classic F8 key during WIndows boot, to discover that Windows Recovery Environment (with Metro style ) is what you get. This is how you can get to boot into Safe Mode and leave it there in case you would need to.

1.- At home screen, type “cmd” in home screen to access it. To run it elevated, press Ctrl+Shift and then click on cmd icon. Accept User Account Control prompt.

2.- At command prompt, use bcdedit /enum /v to list the boot entries. The first entry will have “Windows Boot Loader” as description. The second will have “Windows Developer Preview”. Copy the identifier GUID, {23fc82d2-….} in the picture below.

3.- Duplicate the entry by using the following command (use your entry’s GUID ). You can use whatever description you want:

bcdedit /copy {23fc82d2-….} /d “Windows Developer Preview (Safe Mode)”

4.- From command prompt, invoke msconfig.exe. In there, select the newly created boot item and enable Safe Mode for it by selecting the checkbox shown in the picture below. Also mark the checkbox “Make all boot settings permanent”. Then hit OK, and Yes again in warning message. You will be prompted then to exit msconfig with or without restart. Up to you . You may also change timeout so it takes just a few seconds before normal start.

5.,- When you reboot you will get the following:

6.- If you click in “choose other options” text (not much intuitive in Developer Preview ) you will access other special boot options:

7.- If you select the boot option you created before, you will access Safe Mode, after warm boot.

Hope you find this useful.

Enjoy!

Office 365 Directory Synchronization Tidbits, Part 2: More about Filtering

2011-07-11T16:58:59Z

In this post I will comment a little bit more on object filtering. As you probably know, Microsoft does not support any modifications in the out of the box configuration of Office 365 Synchronization Tool. There is a good reason for that, including that incorrect changes may trash your online services, including data loss. And nevertheless, the only pain for unwanted synched users is that they appear in the administration tools, but they are not showed in the GAL, so there is no big impact on service to end users anyway.

In this post you will find that “Customers that have previously configured custom filters in the directory synchronization tool, either via the filter file or directly via the Identity Lifecycle Manager (ILM) UI, should find alternate means for preventing Active Directory objects from synchronizing to Microsoft Office 365.”

There is really not much margin for “alternate means”, as customers will be limited as of today to update their users, groups and contacts to match the built-in filters, such as sAMAccountName to contain “}”, mailNickName start with “CAS_”, etc., not really convenient…. . On the other hand, the “filter file” mentioned refers to DirSyncFilters.xml file placed in Sync tool installation path (as referred in “HKLM\Software\Microsoft\MSOLCoexistence\InstallPath” registry value), and described quite well here is not supported anymore as it used to be in BPOS.

However, while *UNSUPPORTED*, this both techniques are still functional, so you may use them at least in your proof of concept and non-productive environments to get your hands on Sync Tool and find gotchas for your specific environment before going live.

Enjoy!

Update:

Ivan pointed in his comment (thanks Ivan!) that the DirSyncFilters.xml is not functional. It is partially true (does not work as in BPOS), and partially false (you can make it work ). Nevertheless, if you decide to use filtering in test environment, is more practical to filter objects directly in the Management Agent that through the DirSyncFilters.xml file anyways.

Here are some evidences of DirSyncFilters.xml usage, again, *UNSUPPORTED*:

With Process Monitor you can see miiserver.exe accesing the file …

If you intentionally malform the XML file content, you will get errors in the event viewer (in the example, closed <ExcludeDN> with <ExcludedDNs>) …

If you duplicate an entry, it will also give you event log errors …

If an object is filtered, no event is logged. You will have to enable .NET tracing to get such information. You can do that by adding the following to %ProgramFiles%\Microsoft Online Directory Sync\SYNCBUS\Bin\miiserver.exe.config file:

</configuration>
    <system.diagnostics>
        <trace autoflush="true" indentsize="4">
            <listeners>
                <add name="TextListener"
                    type="System.Diagnostics.TextWriterTraceListener" initializeData="trace.log" />
            <remove name="Default" />
            </listeners>
        </trace>
    </system.diagnostics>
</configuration>

Doing so will generate %ProgramFiles%\Microsoft Online Directory Sync\SYNCBUS\Bin\trace.log file with content similar to:

miiserver.exe Information: 0 : Initializing the object DN filter list.
miiserver.exe Information: 0 : The DN filter list loaded 1 filters.
miiserver.exe Information: 0 : \nIgnoring filtered object CN={795662656868316173557571543434337275635038513D3D}\n

If you enable tracing but DirSyncFilters.xml is not present, it will also trace so:

miiserver.exe Information: 0 : Could not find file 'C:\Program Files\Microsoft Online Directory Sync\DirSyncFilters.xml'.

What it is true is that the BPOS technique does not work in the same way now. Instead of specifying the objectGUID in this form:

<?xml version=”1.0” encoding=”utf-8”?>
< DirectorySyncFilters>
< ExcludedDN>CN={xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx}</ExcludedDN>
< /DirectorySyncFilters>

You have now to use the object DN that sync engine assigns to the object when it is not filtered out:

<?xml version=”1.0” encoding=”utf-8”?>
< DirectorySyncFilters>
< ExcludedDN>CN={795662656868316173557571543434337275635038513D3D}</ExcludedDN>
< /DirectorySyncFilters>

Note that the DirSyncFilters.xml file is processed during object provisioning. So if you modify it you also need to make sync engine discard last file and reload it. The simplest way is copy and delete a file into the \Extensions folder.

Enjoy!

Office 365 Directory Synchronization Tidbits, Part 1

2011-07-05T22:51:20Z

In this post I will try to cover some details that may help to understand how Microsoft Online Services Directory Synchronization Tool (MOSDST) works when it synchronizes on-premises Active Directory with Office 365. MOSDST is nothing else than a custom solution built on top of Microsoft Identity Lifecycle Manager 2007, but for non-ILM people it may seem a black box that synchronizes objects from AD to O365 without control. Knowing what it does may help understanding what it does. I name it “Part 1”, as I expect to hopefully add some more posts around this topic .

SourceAD Management Agent

The SourceAD Management Agent created by MOSDST is simply an Active Directory MA, with the following configuration details:

Forest: The ADMA connects to your on-premises Active Directory forest. As MOSDST uses a single ADMA, it only supports one forest.
Credentials: During setup, MOSDST creates MSOL_AD_Sync user account, with unknown password. You can reset its password to a known one if you wish, providing that you update the ADMA configuration. You may also rename the account, but notice that MOSDST filters out “MSOL_AD_Sync” value in sAMAccountName, so you will find your SourceAD MA user account in Office 365 users list if it does not match other filters (see below). If you accidentally delete the account, you can recreate it with a password and update the MA configuration.
Containers: The ADMA does not filter out any container, so it can potentially load every users, groups and contacts (providing that are not filtered out by the MA code) to Office 365.
Password Synchronization: The ADMA is not enabled as password synchronization source, so your Active Directory passwords, even if you have PCNS installed, will not flow to Office 365. This is expected, as your users will typically be using ADFS to sign-in into Office 365 using corporate credentials.

Active Directory Objects In Scope Of Synchronization

Not all your objects in AD will end up in Office 365. While it is said that MOSDST only synchronizes users, groups and contacts, it is not exactly like that. What it loads into Office 365 depends on:

Mapped Objects: Along with “group” and “contact” object in AD, “user” and “inetOrgPerson” objects are also subject to synchronization to Office 365. inetOrgPerson are treated somewhat similar to “user” objects, if they do not fall into a connector filter match.
Connector Filters: There are several connector filters defined that filter out objects, so you may expect certain users, groups and contacts to be synched to Office 365 but they will not if they match a filter. So knowing those filter conditions are important to know what to expect.

What is Filtered Out

The ADMA filters out certain “user”, “inetOrgPerson”, “contact” and “group” objects if they match certain conditions. Here are those. If you expect an object to be synchronized to Office 365 but it is not, check if it is matching one of these filters.

“inetOrgPerson” objects are filtered out if:
- sAMAccountName attribute is not present.
- isCriticalSystemObject attribute set to true.
“User” objects are filtered out if:
- sAMAccountName attribute is not present.
- isCriticalSystemObject attribute set to true.
- sAMAccountName attribute is “SUPPORT_388945a0”. More info about this account here.
- sAMAccountName attribute is “MSOL_AD_Sync”.
- mailNickname attribute starts with “SystemMailbox{“.
- sAMAccountName or mailNickname attributes start with “CAS_”.
- sAMAccountName or mailNickname attributes contain “{“ or “}”.
- msExchRecipientTypeDetails attribute is 0x1000, 0x2000, 0x4000, 0x400000, 0x800000, 0x1000000 or 0x20000000.
“contact” objects are filtered out if:
- displayName attribute contains “(MSOL)” and msExchHideFromAddressLists attribute is set to true.
- mailNickname attribute starts with “CAS_” and contains “{“.
- The object is not mail-enabled.
- The displayName attribute is not set.
“group” objects are filtered out if:
- Contains more than the maximum number of allowed members. By default, it is set to 15,000 members, but can be adjusted by the “GroupMembershipSizeFilter” REG_DWORD value under “HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSOLCoExistence” registry key. This limit is based on absoult members list count, not based on those members that are actually synchronized to Office 365.
- Is a security-enabled group and isCriticalSystemObject attribute is set to true.
- Is a security-enabled group and is mail-enabled but displayName attribute is not set.
- The object is not mail-enabled.
- The displayName attribute is not set.

“mail-enabled” in this context means that either has a primary SMTP address in proxyAddresses attribute (i.e. “SMTP:johndoe@contoso.com”), or mail attribute has an SMTP address (i.e. “johndoe@contoso.com”).

Update: Somewhat similar info can be found here and here. A great PDF is also available here that contains lots of inner details about Sync Tool too.

Enjoy!

Forefront Identity Manager 2010 R2 CEP

2011-05-11T07:37:00Z

In june 2011 the Forefront Identity Manager 2010 R2 CEP program is launched.

CEP stands for Community Evaluation Program at Microsoft Connect.

As explained on the landing page: “The Community Evaluation Program from the Management and Security team at Microsoft provides IT professionals a structured approach to evaluating System Center and Forefront products before their final release. Members of this program are able to evaluate early versions of products with guidance from the product team and by sharing of experiences and best practices among a community of peers.

This site is a starting point to get involved with the Community Evaluation Program. Once you have applied and been accepted to a program, you will be invited to a product specific site on Connect. Please check this page often for updates on current and upcoming programs.“

To apply for the Forefront Identity Manager 2010 R2 Community Evaluation Program, see theForefront Identity Manager 2010 Community Evaluation Program Application.

Additionally you can Download the Datasheet.

Analyzing a Crash Dump, aka BSOD

2011-03-20T22:29:26Z

Today I have to face a weird behavior in my laptop since a few of days ago. It only happens that when the computer goes to sleep, that is, stand-by, either by my request or due to lack of battery, the screen goes blank but seemed to never finish the sleep process.

I say “seemed” because the last times this happened, I did not have the patience to wait and long-pressed the power button to force a shutdown. However, last night, I went to bed and left it in the sleeping attempt, and this morning I found a “nice” crash dump, also known as Blue Screen Of Death, complaining about DRIVER_POWER_STATE_FAILURE issue:

Sometime ago I had other BSOD that pointed me to a networking drivers issue (ndis.sys), and updating those available seemed to work and did not get BSOD anymore:

This time, while “DRIVER_POWER_STATE_FAILURE” points initially to a device driver that is preventing the kernel to suspend the computer, but the BSOD does not show which driver could it be, so it could be tedious to go and try to update and test every single driver. So lets get into memory dump analysis to see if we can find the faulty driver.

To do so, the first thing we need is a kernel memory dump. To get so, we need to have the system configured to do so in “Startup and Recovery” dialog. By default, Windows 7 does not show BSOD, but restarts the computer after system crash, so if you want to see the BSOD message, you need to uncheck the “Automatically restart” checkbox.

To analyze the memory dump generated after a BSOD, we can use WinDbg, included with Windows Debugging Tools. There is great intro and download links at at http://msdn.microsoft.com/en-us/windows/hardware/gg462988.aspx.

Once we launch WinDbg, the first thing to do is configure the symbols path. We do so from “File/Symbol File Path”, and specify “SRV*c:\SymbolsCache*=http://msdl.microsoft.com/download/symbols” as path (without quotes). This will download symbols from Microsoft to c:\SymbolsCache as needed by WinDbg.

Then we open the crash dump from “File/Open Crash Dump”

In my case, I opened “DRIVER_POWER_STATE_FAILURE.dmp”, as I renamed the dump file to prevent other dumps to overwrite it.

At the end of the initial output, there is a candidate driver for the BSOD, netw5s64.sys. Executing “!analyze –v” confirms this fact:

We can see that the error is caused by a device driver blocking an IRP (IoCompleteRequest) for too long, and we can see that the IRP address is fffffa80101fac10.

If we run !IRP fffffa80101fac10, we can see the drivers involved:

We can again see netw5s64, and this time also Virtual WiFi bus (vwifibus), available in Windows 7 for WiFi hosted networks feature. So here is my supossedly faulty driver:

The point is that it has being working without issues for a long time, so it may not necessary be a bug in the driver itself, but a hardware failure or a driver configuration. The first thing we can try though is look for driver updates. We can do this directly from Device Manager. Unfortunately, I had no luck this time:

Another thing we can try is go to the computer’s integrator website, or even directly to the driver’s manufacturer website, Intel Corporation in this case, to see if there is a new driver. This operation can be more or less lengthy, depending on the manufacturer/integrator's sites organization and complexity. In my case, my laptop vendor was not providing a good driver revision (even older than the one I had installed), but Intel did: there was an updated driver not available in Windows Update:

While I download the updated driver, I take a look at “Power Management” tab of the device which driver was causing the BSOD in Device Manager, and found the most possible reason for the failure: The interface is configured to not allow the computer to turn of this device to save power.

Before updating the driver, I checked this checkbox and try to suspend the computer. Guess what, this time the computer suspended just fine.
So there is no reason to update the driver…. but …. well…. we all like to be up-to-date … so I’ll install it anyways

Enjoy!

Forefront Identity Manager 2010 Ramp Up

2010-12-14T19:00:07Z

At my job role, I have being asked several times about what resources I would recommend to technical consultants that want to start with Identity Management solutions. Fortunately we at Microsoft have a recommended training roadmap created for internal staff that - again fortunately - uses lots of public resources. While I cannot share here internal resources, I think it is good for building ILM/FIM technical community to share what training resources our experts recommend. So here it goes, plus some bonus links .

Happy learning!!

Level	Description	Resource	Duration
Basic	After completing the solutions recommended for this level, you should be able to articulate the Microsoft strategy for the relevant area, the value proposition Microsoft brings to customers and the main competitors in this space. You should also be able to identify the different components of this product and the relationships between them.	FIM 2010: Technical Overview and Deployment	1h
Intermediate	After completing the solutions recommended for this level, you should be able to describe the product architecture, have a detailed understanding of the product’s capabilities and limitations, and understand its deployment scenarios and requirements.	Microsoft Identity Integration Server 2003 Scenarios	7 h
		FIM Introduction Technical Library Documents: FIM 2010 Technical Overview Understanding Custom Resource and Attribute Management Introduction to Custom Resource and Attribute Management Understanding Configuring and Customizing the FIM Portal Introduction to Configuring and Customizing the FIM Portal Introduction to Resource Control Display Configurations Resource Control Display Configuration XML Reference Introduction to Management Policy Rules Introduction to Inbound Synchronization Introduction to Outbound Synchronization Introduction to Publishing Active Directory From Two Authoritative Data Sources Introduction to User and Group Management Introduction to Distribution Group Management Introduction to Security Group Management Common Configuration for Getting Started Guides Introduction to Request Management Password Reset Deployment Guide FIM Installation Guide FIM 2010 Capacity Planning Guide Configuration Migration Deployment Guide FIM Backup and Restore Guide	19 h
		CLM Technical Library Documents: CLM Technical Library Documents Certificate Mangement Deployment FIM CM Backup and Restore Guide	4.25 h
		Course 50382A: Implementing Forefront Identity Manager 2010	32 h
Advanced	After completing the items recommended for this level, you should be able to build, deploy and troubleshoot solutions based on the product for enterprise customers. At this point, a consultant should be to lead customer engagements involving the product.	Exchange Provisioning using ILM 2007 and FIM 2010	1.5 h
		FIM CM: Getting Started	4 h
		Extensible MA Development: Developing a File-Based Import Management Agent Developing a File-Based Export Management Agent Developing a Call-Based Export Management Agent	4 h
		Modeling Business Policy Rules with FIM	1.5 h
		Course 50383A: Upgrading ILM 2007 to FIM 2010	32 h
Expert	After completing the items recommended for this level, you should be able to build solutions for complex usage scenarios (large scale, mission critical, integration, migration).	Generate Specialized Test Data with a Custom Data Generator	3 h

Community Resources:

- FIM Technet Group
- FIM Experts Corner
- FIM Scriptbox

Other:
- MVPs
- displayName – The Silent Identity Management Killer
- Identity Management Project Phase One: Joins and Data Matching

Extended Resource Library:

- Understanding FIM 2010
- Xpath Reference
- Workflow Foundation Overview
- Introduction to Sequential Workflows
- Custom Activities & Workflows
- Develop Custom Activities & WF
- Create Custom Activity Library
- Rendering Custom Activities in the FIM Portal
- Enterprise PKI Concepts
- Technical Overview of CLM 2007
- CLM 2007 Operations
- ILM 2007 Resources
- FIM 2010 Webcasts and videos

Product Team and Microsoft Blogs:
- Identity and Access Management Blog
- Identity Management Extensibility
- Shawn’s MIIS/ILM/FIM Code Experiment

MVPs and Experts Blogs (alphabetical by givenName ):
- Almero Steyn’s Puttyq
- Blain Checkley's Identity Minded
- Brad Turner’s 1dent1ty cHa0s
- Carol Wapshere’s Missmiis
- Craig Martin’s Identity Trench
- David Lundell’s ILM Best Practices
- Dmitry Kazantsev’s Lost and Found Identity's blog
- Henrik Nilsson’s Identity Management Crisis
- Jeremy Palenchar’s Identity Notes
- Joe Stepongzi’s Microsoft IDM.
- Joe Zamora’s CShark
- Jorge de Almeida Pinto’s Jorge 's Quest For Knowledge!
- Kim Cameron's Identity Weblog.
- Laura E. Hunter’s Shut Up, Laura
- Marc Mac Donell’s Assurances in Identity
- Naohiro Fujie’s IdMlab
- Paolo Tedesco’s Identity Management at CERN.
- Paul Loonen’s be-Id.
- Peter Geelen’s Identity Underground

Playing with Windows Phone 7 as USB Storage

2010-12-10T10:30:11Z

Disclaimer: Procedures here are fully unsupported. Use at your own risk.

Today I will post about my experiences trying to use Windows Phone 7 as a USB storage device. These “hacks” are already covered at many places and forums dedicated to Zune and Zune HD devices, but not for WP7. As long as I experienced some quirks playing with , I want to share them here.

By default, WP7 is not visible in Windows 7 as a portable device, so it is not possible to transfer files to it except by uploading to SkyDrive or them as attachments to yourself, for example. It is not shown up in “Computer” applet, and in “Devices and Printers” applet it is shown as a portable MP3 player.

Enabling device storage visibility in Windows shell

First attempt was modify the following registry key: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\USB\VID_045E&PID_04EC&MI_00\6&53be20e&0&0000\Device Parameters]
“PortableDeviceNameSpaceExcludeFromShell”=dword:00000000

Doing so has the following effects (Zune software must be closed for most of the screenshots to show up like as here):

WP7 appears (twice) as Portable Media Player in Computer applet.

The following additional registry hack helps on removing the duplicate:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\USB\VID_045E&PID_04EC&MI_00\6&53be20e&0&0000\Device Parameters\ZuneDriver]
“UseWpdPrivateInterface”=dword:00000000

You can inspect some device details.Battery charge is wrong.

You can see how much space you have free.

You may format the disk (WARNING: I did not have enough courage to try this, so you are alone here if you do so!)

You can see some disk details.

You get “Import pictures and videos” option from device’s context menu (does fail though ).

In “Devices and Printers” applet the context menu shows file system browsing option.

As covered in other blogs and forums, there are some other registry changes that are in theory needed for content access to work for Zune HD devices, such as:

EnableLegacySupport = dword:00000001
ShowInShell = dword:00000001

However, for me it worked with just the specified setting for WP7 device. I was even able to create a “Documents” folder and add content to it:

Unfortunately, documents are not visible to applications on the phone and it only serves as a carry-on device. That is, you can open files for viewing (pictures need to use Open from context menu). When opening files, they are first copied to user’s %Temp% folder, and opened from there. Files in tempo folder are read-only, and save attempts do not update the device copy and trying to save directly to device also fails. The only option is save somewhere else and then copy the updated file back to the device using Windows Explorer.

A look at “Pictures” folder

When opening the WP7 device storage, you will only see a “Pictures” folder with your pictures and videos. Let’s take a quick look at it.

Interesting folder structures:

Named folders. The ones you see with normal ones are just synched from your “Pictures” folder, with the folder structure you have in that folder. For example, if you inspect your pictures in your computer with Zune software and drag “%userprofile%\Pictures\Holidays\2010\On The Beach\DSC001.jpg” to the phone icon, you will get “Holidays\2010\On The Beach” folder hierarchy in WP7 storage, and then your JPG file inside. The good thing is that once copied to the WP7 device, you can freely reorganize the folder structure if you do not like having same folder depth or naming as on PC, for example.
“ {9ae241c6-e6cc-4080-a2ba-245e0f7c47c5}”. This folder serves as location to save images and videos from WP7 camera.
“ {1e544589-04c4-492f-87ca-294a52149279}”. This folder serves as location to save images downloaded from the Internet.
“ {4915925E-FB2A-11DE-AE1C-DD6355D89593}”. This folder holds sample pictures that come with the device.
“ {9ae241c6-e6cc-4080-a2ba-245e0f7c47c6}”. No idea what this empty folder is for yet, it looks empty in my device….

I don’t know if these GUIDs are global or device-based, as I only plugged my own WP7 device.
Those spaces on the left are important though. If you rename those GUID’ed folders, you will experience that and it seems impossible to rename them back to the original names with those spaces at the beginning of the folder name. First time I rename one of those, I had to call my “ {4915925E-FB2A-11DE-AE1C-DD6355D89593}” to “Sample Pictures” . However, here is how you can get your names back:

Create a folder with the desired GUID in a hard disk, i.e. c:\tmp\{4915925E-FB2A-11DE-AE1C-DD6355D89593}”.
Using command prompt, rename the folder to add those spaces:

Drag the folder on the device.

Trying to disable exclusive access… just fails ..

If you launch Zune software with storage opened, you will see the following message:

On the same hand, when Zune software is accessing the WP7 device, device storage looks empty.

If this is the case, check your favorite task manager for Zune.exe process. Sometimes Zune does not exit in a timely manner and remains opened for a while when you close its window. When Zune.exe is running, the device content cannot be accesed. Just kill existing instances (you may have two sometimes) to get access to storage content again.

Sometimes also even when Zune.exe is not running, storage still looks emtpy and Zune software complains about “Portable Devices Namespace is using your device”. If this happens, you can try kill all Zune.exe instances, unplug the device, launch Zune so it promptf

I tried to enable both Explorer shell and Zune software to access the device simultaneously by modifying the following registry key: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\USB\VID_045E&PID_04EC&MI_00\6&53be20e&0&0000\Device Parameters\WUDF]
“Exclusive”=dword:00000000

However, this value did not get the desired effect. Unplug/replug the device did not help neither. Nevertheless, I would prefer to leave exclusive access as it is by default, and go kill Zune.exe process when needed, just to avoid chances of data corruption .

Different device types

There is an interesting registry value there… “PortableDeviceType”. It says “2” by default. If set to “0” it says “Portable Device”. “1” as “Digital Camera”. “3” as “Mobile Phone”. “4” and up as “Portable Device” again. None of these changes did not help with direct-editing attempts, but Zune software stops seeing it as a WP7 device.

Messing with drivers, Part 1

I have an old MP3 player that shows as portable device by default, and behaves just like WP7 in terms of Windows Explorer usage. I wanted to copy music to my MP3 using robocopy tool. Do do so, it worked to force the device driver from “Portable Device” to “USB Mass Storage Device” and it did the trick perfectly, opening the device storage as a regular USB memory stick. So I wanted to give a try to this with WP7 … unfortunately “USB Mass Storage Device” was not an option, so I tried a few of what showed up as available …

First attempt: “Windows Mobile-based device”. The driver does not load properly, and unplug/replug the device just updates to WP7 drivers and fixes it. The same happened with “WPD FileSystem Volume Driver”.

However, “Zune” drivers did load properly. However, Zune software does not recognize the phone properly, and doing the reghacks does not make the device appear in Explorer shell .

With all these driver hacks, a few interesting registry values appeared: AutoPlaySourceOnly, OptimalTransferSize, ConvertSerialNumberToANSI, DeviceSelectiveSuspended or PortableDeviceIsMassStorage.

With all this messing, Zune software and Explorer stop viewing the WP7 device .
To get back to work, I had to open regedit as SYSTEM account (used psexec.exe –s –d –i cmd.exe) and delete existing messed “Device Parameters” subkey. Then unplug/replug the device, choose “Update driver” and pick “Windows Phone 7” in Device Manager to reload the right driver and “Device Parameters” subkey values.

Once restored original functionality, I gave a try to “PortableDeviceIsMassStorage” value. Though promising, setting to “1” hided the device in Windows shell, and 0 or deleting the value got back to good-old known visibility effects described above.

Messing with drivers, Part 2

As messing with drivers from Windows side did not took me to any results, I tried from the WP7 device side. By using MFG tool I changed my WP7 device from “MS COMPOSITE” USB mode (default) to “MS DEFAULT”. For more information about MFG tool, search here .

When plugging the phone back, I got the following:

No luck going further… Neither Windows Mobile Device Center nor Zune software can see the device …

If you know other/better USB storage tricks, please share them!
Thanks!

Update: There is now a tool that ease the process. You can find its (translated) descriptioin here. Download link here.

Internet Explorer Protected Mode Elevation Policy and Administrative Templates

2010-10-29T08:25:00Z

Overview:
You probably have run through the following popup when using Internet Explorer add-ons and extensions:

In this example, I clicked a link in an Internet web page that pointed to an XPS file.
This prompt is due to the fact that Internet Explorer is running in Protected Mode, and tries to open an application or extension outside of Protected Mode.

The IE behavior regarding this prompt is governed by the following registry keys:

HKLM\SOFTWARE\Microsoft\Internet Explorer\Low Rights\ElevationPolicy. Some software installation programs register themselves under this key (each with its own subkey GUID) to manage Protected Mode behavior. On the same hand, if Microsoft determines that an application has a vulnerability and presents a danger to end users, Microsoft reserves the right to remove that application at any time from the elevation policy.
HKCU\SOFTWARE\Microsoft\Internet Explorer\Low Rights\ElevationPolicy. When the user checks the “Do not show me the warning for this program again” in the prompt dialog, Internet Explorer writes a subkey GUID and associated values under this key to “register” Protected Mode behavior for a given application for that particular user.

You can check more details at http://msdn.microsoft.com/en-us/library/bb250462(VS.85).aspx and http://blogs.msdn.com/b/ieinternals/archive/2009/12/01/understanding-internet-explorer-security-protected-mode-elevation-dialog.aspx

If an administrator wants to deploy specific Protected Mode prompts for specific applications, can do two things:

Directly change HKLM\HKCU keys. This can be implemented through logon scripts (HKCU part) or PC imaging/software deployment (HKLM part). The caveat is that the setting is not enforced (users can override it) and leaves traces on the system that need to be tracked back.
Apply changes in Policies branches. This overrides the actual HKM\HKCU keys, enforces behaviors does not alter system configuration and cleanly disappear when the Group Policy settings are removed, so this is the recommended approach.

Managing Elevation Policy through Group Policy:
Using Group Policy infrastructure, you can actually change the Protected Mode behavior, to for example, always prompt for to allow an extension to run. Notice the greyed-out checkbox. This prevents the user to check it and override this prompt.

In the same manner, you can use Group Policy to hide prompts for trusted applications automagically, so if you are deploying an application you can transparently run it through Protected Mode (i.e. Microsoft Office Live Meeting) whenever it gets deployed to the managed desktop computer, without bothering the user with the security warning.

The problem here is that neither Windows Vista nor Windows 7 (or their server counterparts) include an Administrative template to manage these settings out of the box, so http://support.microsoft.com/kb/918239 provides some guidance and templates to manage these configuration. However, the provided ADM/ADMX are not perfect. This is how it looks in GPEditor:

The problems of this ADMX include:

Hardcoding. The provided ADMX hardcodes registry values for the applications you want to manage in the ADMX itself, not through policy options. It actually behaves as an “configuration set” that applies elevation policies to applications as specified within the ADMX.
No customization point for administrators. Administrators only can set the policy to Enabled or Disabled, and does the registry changes included in the ADMX. If the ADMX includes three applications with specific elevation policies, administrators cannot configure just one of them or change the elevation policy from what is hardcoded in the ADMX. If there is need for delegated administrators to manage different policies or application sets, they need to generate separate ADMX files.

So I decided to give ADMX creation a try, and end up with the following:

The advantages of this custom ADMX include:

Information for administrators. ADML file includes details for administrators about what the policy does and how to configure it.
Allows to customize up to 15 applications (by default). Administrators can specify executable names, paths, CLSIDs and policy levels. Actually, the ADMX file can easily be extended to support whatever number is needed beyond 15.
Provide a common GUID space for policies. ADMX file uses registry subkeys under Software\Policies\Microsoft\Internet Explorer\Low Rights\ElevationPolicy in the form of GUIDs like {DEADBEEF-CAFE-DEAD-BEEF-000000000001}, {DEADBEEF-CAFE-DEAD-BEEF-000000000002}, {DEADBEEF-CAFE-DEAD-BEEF-000000000003}, up to {DEADBEEF-CAFE-DEAD-BEEF-000000000015}. For lower OU levels GPOs to override higher level OU GPOs, you just need to customize a few and set the rest to “Disable”.
Support for variables in extension path. In the extension path textbox, administrators can specify %ProgramFiles%, %SystemRoot% or other variables to avoid hardcoding paths.

Here are the ADMX files. Just copy ADMX to “%SystemRoot%\PolicyDefinitions” and ADML to “%SystemRoot%\PolicyDefinitions\en-US”, or place them in Central Store.

Dual Boot Hyper-V VHD

2010-10-26T06:02:08Z

This is a note to myself and others about how I added a Windows Server 2008 Hyper-V VHD as dual boot over an existing Windows 7 installation.

Before start, generate a new partition at the end of the Windows 7 partition, using diskmgmt.msc console. You can do this by shrinking the existing Windows 7 volume and add a new partition in the free space you get afterwards.

If you are using Bitlocker in your Windows 7 partition, note that your VHD needs to boot from an unencrypted partition, so if you want to protect your Hyper-V VMs with Bitlocker, the way to go is running Hyper-V VM from unencrypted partition and mount your Windows 7 partition with your Bitlocker keys and load your protected VMs from there.

Important: Ensure that your Hyper-V dedicated partition has enough space to host both your Hyper-V VM and a pagefile.sys of the size of your physical RAM. In my case, my Hyper-V VM was 8,5 GB, and had to leave another 8 GB for paging file. It is also recommended to leave some more space for Windows Updates to the Hyper-V VM itself.

Once you have your fresh new partition with your “Hyper-V.vhd” file on it and room for paging file, do the following:

1.- Run BCEDIT without params to show boot entries and copy "Windows 7" entry GUID to clipboard

2.- Copy Windows 7 boot entry as Hyper-V entry

bcdedit /copy {Win7-GUID} /d "Hyper-V"

3.- List boot entries again (BCDEDIT) and copy "Hyper-V" entry GUID to clipboard

4.- Change Hyper-V entry device and osdevice options to VHD

bcdedit /set {Hyper-V GUID} device vhd=[H:]\Hyper-V.vhd
bcdedit /set {Hyper-V GUID} osdevice vhd=[H:]\Hyper-V.vhd

5.- You can also specify [LOCATE] as drive letter, so boot loader will seek existing
volumes for your VHD:

bcdedit /set {Hyper-V GUID} device vhd=[LOCATE]\Hyper-V.vhd
bcdedit /set {Hyper-V GUID} osdevice vhd=[LOCATE]\Hyper-V.vhd

6.- You may need to enable Hypervisor with bcdedit if you are unable to enable Hyper-V role
    even with VT extensions and DEP enabled in system BIOS, and you get the following warning
    in the Hyper-V role event log:
    Event 5: Hyper-V launch aborted due to auto-launch being disabled in the registry.

bcdedit /set {Hyper-V GUID} hypervisorlaunchtype auto

Now you can restart and see your “Windows 7” and your “Hyper-V” entries in your boot menu.

Enjoy!

SQL Script: Finding duplicates on supposedly unique values

2010-08-12T12:18:36Z

Unsupported: Use at your own risk

Sometimes you load data into metaverse that is supposed to be unique and you do not want to use FindMVEntries technique due to performance impact.

This SQL scripts allow you to find out directly from DB if you have unexpected duplicates so you can talk to data store owners to go and fix their data.

The following example looks for repeated mailNickName attribute values in metaverse. Just replace “mailNickName” with “SAMAccountName”, “uid” or whatever attribute you have that should be unique:

SELECT mailNickName, COUNT(mailNickName) AS NumOccurrences
FROM FIMSynchronizationService.dbo.mms_metaverse with (nolock)
where object_type = 'person'
GROUP BY mailNickName
HAVING ( COUNT(mailNickName) > 1 )
order by NumOccurrences desc

You could also use this technique to find the opposite, that is, records with a particular value that occur exactly once:

SELECT email
FROM FIMSynchronizationService.dbo.mms_metaverse with (nolock)
GROUP BY email
HAVING ( COUNT(email) = 1 )

However, the previous queries give just details about the number of records that you have duplicates or unique. Finding out more details about the metaverse records require more elaborated SQL query.

In the following example, we first find duplicated sAMAccountName values and store in temporary “tblDups” table, that then we use to join with full metaverse (“MV”) and show “displayName” and “object_id” attributes.

select tblDups.sAMAccountName, MV.displayName, MV.object_id
FROM
(
SELECT sAMAccountName, COUNT(sAMAccountName) AS NumOccurrences
FROM FIMSynchronizationService.dbo.mms_metaverse with (nolock)
where object_type = 'person'
GROUP BY sAMAccountName
HAVING ( COUNT(sAMAccountName) > 1 )
) as tblDups
INNER JOIN 
FIMSynchronizationService.dbo.mms_metaverse AS MV WITH (nolock) 
ON
tblDups.sAMAccountName = MV.sAMAccountName
 
 

You can also add WHERE sentences at the end of the query to find specific values, such as those sAMAccountNames duplicates that belong to HR department:

select tblDups.sAMAccountName, MV.displayName, MV.object_id

FROM

(

SELECT sAMAccountName, COUNT(sAMAccountName) AS NumOccurrences

FROM FIMSynchronizationService.dbo.mms_metaverse with (nolock)

where object_type = 'person'

GROUP BY sAMAccountName

HAVING ( COUNT(sAMAccountName) > 1 )

) as tblDups

INNER JOIN

FIMSynchronizationService.dbo.mms_metaverse AS MV WITH (nolock)

tblDups.sAMAccountName = MV.sAMAccountName

WHERE MV.department = 'HR' 

SQL Script: Delete all CS and MV data

2010-08-12T12:01:13Z

Unsupported: Use at your own risk

This script will delete all the records from FIM Sync database tables without affecting the MAs. It is useful when you have lots of MAs and export srvconfig/restore empty db/import srvconfig means painful enter of login data against all data sources.

TRUNCATE TABLE FIMSynchronizationService.dbo.mms_connectorspace;
TRUNCATE TABLE FIMSynchronizationService.dbo.mms_cs_link;
TRUNCATE TABLE FIMSynchronizationService.dbo.mms_csmv_link;
TRUNCATE TABLE FIMSynchronizationService.dbo.mms_joiner_log;
TRUNCATE TABLE FIMSynchronizationService.dbo.mms_metaverse;
TRUNCATE TABLE FIMSynchronizationService.dbo.mms_metaverse_lineagedate;
TRUNCATE TABLE FIMSynchronizationService.dbo.mms_metaverse_lineageguid;
TRUNCATE TABLE FIMSynchronizationService.dbo.mms_metaverse_multivalue;
TRUNCATE TABLE FIMSynchronizationService.dbo.mms_mv_link;
TRUNCATE TABLE FIMSynchronizationService.dbo.mms_run_history;
TRUNCATE TABLE FIMSynchronizationService.dbo.mms_step_history;
TRUNCATE TABLE FIMSynchronizationService.dbo.mms_step_object_details;
TRUNCATE TABLE FIMSynchronizationService.dbo.mms_tracking_entries;
TRUNCATE TABLE FIMSynchronizationService.dbo.mms_tracking_entries_history;
 

SQL Script: Delete Run History

2010-08-12T11:52:27Z

Unsupported: Use at your own risk

These three simple lines allow you to clear the Run History.
You can use this when admin client does not allow you to clear the history through the UI. I've seen this a couple of times when History is huge.

TRUNCATE TABLE FIMSynchronizationService.dbo.mms_run_history;
TRUNCATE TABLE FIMSynchronizationService.dbo.mms_step_history;
TRUNCATE TABLE FIMSynchronizationService.dbo.mms_step_object_details;

Standalone Network Emulator Tool

2010-03-05T16:05:10Z

All credits go for Lonny Kruger and Ivan Neganov.

NEWT (Network Emulator for Windows Toolkit) is a software-based emulator which can emulate the behavior of both wired and wireless networks using a reliable physical link, such as an Ethernet. A variety of network attributes are incorporated into the NEWT emulation model, including round-trip time across the network (latency), the amount of available bandwidth, queuing behavior, packet loss, reordering of packets, and error propagations. NEWT also provides flexibility in filtering network packets based on IP addresses or protocols such as TCP, UDP, and ICMP.

NEWT (Network Emulation for Windows Toolkit) is a fabulous network emulator tool that allows you to simulate different network conditions, and include ports, protocols and interface filters. This tool is now part of Games for Windows LIVE SDK and XBOX XDK.

Unfortunately for lots of IT professionals out there, these SDKs are not generally available, so they have to engage with networking guys, their knowledge and their devices and networks to perform tests related to how well or bad their infrastructures work under different network conditions.

The good news is that driver portion of NEWT was transfered to Visual Studio 2010 Team to support the new load testing features, so then it became possible to develop NEWT-like tool. Lonny Kruger posted about how to build a sample standalone network emulator source code for Visual Studio 2010 Beta 1 in those days. Later, Ivan Neganov posted a similar solution, already compiled for everyone to use, this time for Visual Studio 2010 Beta 2.

The issue I find myself is that both of posts mean that you have to install/use Visual Studio 2010 to be able to run network emulation. This is how I do it know with this tool, just for those ITPro non-developer guys out there:

1.- Download Visual Studio 2010 RC and extract files from vs_setup.msi. To do so you can use your favorite MSI extraction tool, or just install it in a VM or such a thing. You will need just a few files out of it.

2.- Download and install Microsoft.NET Framework 4.0 RC from (not sure if client profile is enough).

3.- Grab and install Visual C++ 2010 RC runtime installers needed by network emulation driver for your platform from VS 2010 bits, located at %ProgramFiles%\Microsoft SDKs\Windows\v7.0A\Bootstrapper\Packages\vcredist_x86\ and %ProgramFiles%\Microsoft SDKs\Windows\v7.0A\Bootstrapper\Packages\vcredist_x64\.

4.- Grab and install network emulation driver from VS2010 bits. You can find them both for x86 and x64 at %ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\NetworkEmulation\x86\ and %ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\NetworkEmulation\x64\. Install network emulator driver (NDIS6 for Vista/2008/W7/2008R2, NDIS5 for XP/2003) for your platform. Just in case you do not remember how, here are the screenshots:

5.- Get Ivan Negalov's Stand-Alone Network Emulator bits from http://neganov.blogspot.com/2010/01/stand-alone-network-emulator-for-vs2010.html

6.- Grab the updated DLLs from VS2010 RC bits and overwrite the ones included in Ivan Negalov’s package (Beta 2 files):

%ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.NetworkEmulation.dll
%ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\DataCollectors\x86\Microsoft.VisualStudio.QualityTools.NetworkEmulationAPI.dll or %ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\DataCollectors\x64\Microsoft.VisualStudio.QualityTools.NetworkEmulationAPI.dll, depending on your platform.

Now you have your own neat and simple Network Emulation Tool!

Best regards.

Hacking Application Compatibility Toolkit 5.5 Logs

2010-01-22T08:23:54Z

Disclaimer: The information in this article is for educational purposes only, provided "AS IS" with no warranties, and confers no rights. If you use it, you are actually hacking your ACT implementation, and therefore will be completely on your own, in an unsupported state. USE AT YOUR OWN RISK.

After writing the needed disclaimer, today I would like to post about my findings trying to “trick” ACT 5.5 data in order to feed applications into the DB with actual DCP deployment.

One of the features some customers request would like to see in ACT would be the ability to import their already existing software inventory data into ACT database to check for compatibility, instead of having to deploy DCP (Data Collection Packages) around to gather the “same” information they already have in the inventory.

I intentionally quoted “same”. DCP inventory actually gathers information (“indicators” as it calls in XML logs it generates) about applications from many places that traditional inventories may not look at, such as AppPaths, path environment variable or file extension handlers, to name a few.

Inside AppIDs

The problem with “injecting” inventory data into ACT is the application IDs used to check gathered information against ACT web service. As covered by Chris Jackson at http://technet.microsoft.com/en-us/magazine/2009.06.act.aspx and http://blogs.msdn.com/cjacks/archive/2010/01/06/windows-7-vender-compatibility-data-now-available-in-act-5-5.aspx, applications are tied to compatibility data with a unique Application ID, generated using the name, version, vendor, and language (NVVL) of the application.

Looking at the XML files consumed by the “ACT Log Processing Service”, they already contain those kind-of-magic AppIDs that allow ACM (Application Compatibility Manager) check for data, for example:

These hashes are generated by bucketizer.exe, and are key to application compatibility checks against the Web service. Bucketizer program takes the XML file generated by collector.exe, looks at NVVL data and generates the hashes.

So if I just managed to generate a properly formatted XML file similar to what Collector.exe creates, Bucketizer.exe may not notice the difference and generate the very same AppIDs in the XML that finally can be dropped into ACT logs folder to be processed by ACT Log Processing Service.

Your own Collector XML Template

So… next step in the hack will be generating our own collector XML file.
I managed to generate some sort of template for the collector XML that then we can distinguish in ACM as “imported” data, rather than “collected” data.

<?xml version="1.0" encoding="utf-16" ?>
<Collector>
   <SystemList>
      <ChassisInfo Vendor="Microsoft" AssetTag="" SerialNumber=""/>
      <OsInfo BuildNumber="0" CsdVersion="" MajorVersion="6" MinorVersion="1" PlatformId="2" ProductType="0" ServicePackMajor="0" ServicePackMinor="0" Suite="0" WindowsDirectory="" SystemDirectory=""/>
      <NetworkInfo Domain="" MachineName="_IMPORT_" UserName="" SmsGuid="" SmsHwId="">
         <NIC MacAddress="00-00-00-00-00-00"/>
      </NetworkInfo>
      <HardwareInfo>
         <MemoryInfo PageFile="0" Ram="0" Virtual="0"/>
         <ProcessorInfo Name="" Mhz="0" Architecture="x64" Family="0" Type="0" Level2CacheSize="0" Level="0" Vendor="" Caption="" ProcessorId=""/>
      </HardwareInfo>
      <CustomValues>
      </CustomValues>
   </SystemList>
   <Collection>
[ YOUR APPS INFO WILL BE HERE ]
        </Collection>
</Collector>

This will look like this in ACM, so then you can filter it in/out in case you need it:

Adding Application Data to your Collector XML file

The next step will be adding application data to your custom Collector XML file template. To do so, we need to be respectful with the expected tags and values as much as we can to avoid errors during log processing, so again I built a couple of templates for this. Here are for examples:

Looking at this XML data, you can notice that all the information is just fake data for Bucketizer.exe and Log Processing Service not to complain about it, but actually only NVVL data needs to be accurate.

You also may notice that there are UniqueId values. However, they do not have to be unique for the purpose of the hack. What it has to is –just for MSI blocks– the ProductId value. Originally this is a GUID, but as you see, it doesn’t even have to be a former GUID, but some sort of different string across the applications represented in the XML.

Another interesting thing, again for the MSI blocks, is that you need to add additional evidence data for it to work. In this case I added a completely fake info about “file.exe”. As you see, this file evidence is exactly the same for the three MSI applications I added, as it does not seem to matter much to the AppID generation we are looking for.

From NVVL data to AppID

For the exactly same AppID to be generated in your log file, we need to provide the very same application name, version, vendor and language information as a real DCP will gather in the data collection process.

For example, if DCP puts “Microsoft Corp.” in the logs, and you write just “Microsoft” in your custom XML, the AppID will be completely different.

Something I have notice though is that building a custom “MSI” or “AddRemoveProgram” block type makes a different AppID, so not only NVVL data is important, but also the evidence type you are trying to build. For example, if a normal DCP gathers an application as MSI, and you put the very same NVVL data in your XML, you will get a different AppID.

Building your Collector Log

So now we know what we have to include in the collector-like log to be processed properly by bucketizer.exe and get proper AppIDs to check with the ACT Web Service. So now.. how can I easily build that XML data?

Well, that is completely up to you. My best friend in these type of cases is Excel. If you can have your software inventory to export a CSV or similar Excel-friendly data with application name, vendor, version and language, you are ready to rock!

Just use the CONCATENATE() function in Excel to concatenate the proper strings. For example, if you have appName, appVersion and appPublisher, you can build “XML MSI” column and “XML AddRemove” column to generate data to paste into your collector XML file template.

The formula for MSI blocks:

=CONCATENATE("<StaticProperty Type=",CHAR(34),"Msi",CHAR(34)," ProductName=",CHAR(34),A20963,CHAR(34)," CompanyName=",CHAR(34),C20963,CHAR(34)," ProductVersion=",CHAR(34),B20963,CHAR(34)," InstallDate=",CHAR(34),"01/01/1900 00:00:00",CHAR(34)," Path=",CHAR(34),"",CHAR(34)," FileName=",CHAR(34),"",CHAR(34)," Language=",CHAR(34),"0",CHAR(34)," ProductId=",CHAR(34),"{",RANDBETWEEN(10000000,99999999),"-",RANDBETWEEN(1000,9999),"-",RANDBETWEEN(1000,9999),"-",RANDBETWEEN(1000,9999),"-",RANDBETWEEN(100000000000,999999999999),"}",CHAR(34)," PackageId=",CHAR(34),"",CHAR(34)," GUID=",CHAR(34),"",CHAR(34)," RNP=",CHAR(34),"0",CHAR(34)," DNP=",CHAR(34),"0",CHAR(34)," UniqueId=",CHAR(34),"0",CHAR(34),">
<StaticProperty Type=",CHAR(34),"File",CHAR(34)," Name=",CHAR(34),"file.exe",CHAR(34)," Path=",CHAR(34),"C:\Program Files",CHAR(34)," Size=",CHAR(34),"0",CHAR(34)," PeChecksum=",CHAR(34),"0",CHAR(34)," Checksum=",CHAR(34),"0",CHAR(34)," LegalCopyright=",CHAR(34),"",CHAR(34)," OriginalFilename=",CHAR(34),"",CHAR(34)," InternalName=",CHAR(34),"",CHAR(34)," ProductName=",CHAR(34),"",CHAR(34)," CompanyName=",CHAR(34),"",CHAR(34)," ProductVersion=",CHAR(34),"",CHAR(34)," FileVersion=",CHAR(34),"",CHAR(34)," BinProductVersion=",CHAR(34),"",CHAR(34)," BinFileVersion=",CHAR(34),"",CHAR(34)," VerLanguage=",CHAR(34),"",CHAR(34)," FileDescription=",CHAR(34),"",CHAR(34)," LinkDate=",CHAR(34),"01/01/1900 00:00:00",CHAR(34)," Created=",CHAR(34),"01/01/1900 00:00:00",CHAR(34)," Modified=",CHAR(34),"01/01/1900 00:00:00",CHAR(34)," BinaryType=",CHAR(34),"32BIT",CHAR(34)," RNP=",CHAR(34),"0",CHAR(34)," DNP=",CHAR(34),"0",CHAR(34)," UniqueId=",CHAR(34),"0",CHAR(34)," LowerCaseLongPath=",CHAR(34),"",CHAR(34),"/>
</StaticProperty>")

Important: You can see appName, appVendor and appVersion cells refered in the formula. However, in my test inventory data I had no language data, so I just used literal version data in the formula (bolded “0”), what causes lots of AppIDs not to be properly generated though. You can also notice that to generate unique GUIDs I used RANDBETWEEN() formula, to generate look-a-like numeric GUIDs.

The formula for AddRemoveProgram blocks (again fixed language data):

=CONCATENATE("<StaticProperty Type=",CHAR(34),"AddRemoveProgram",CHAR(34)," DisplayName=",CHAR(34),A45892,CHAR(34)," CompanyName=",CHAR(34),C45892,CHAR(34)," Language=",CHAR(34),"1033",CHAR(34)," ProductVersion=",CHAR(34),B45892,CHAR(34)," Path=",CHAR(34),"",CHAR(34)," RegistrySubKey=",CHAR(34),"",CHAR(34)," RegistryPath=",CHAR(34),"HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Uninstall\MyApp",CHAR(34)," UninstallString=",CHAR(34),"",CHAR(34)," GUID=",CHAR(34),"",CHAR(34)," RNP=",CHAR(34),"0",CHAR(34)," DNP=",CHAR(34),"0",CHAR(34)," UniqueId=",CHAR(34),"0",CHAR(34),"/>")

Bucketizing your Collector Log

Once you have your custom Collector log, you need to use Bucketizer.exe to generate the XML with the right AppIDs, that you then will manually drop in your ACT Logs folder to be consumed by Log Processing Service.

bucketizer.exe /c "<custom collector log folder>" /o "<bucketizer log destination folder>" /b "<bucketizer log destination folder>" /d "<bucketizer log destination folder>" /s "_IMPORTED_"

where:
        /c    Collector log directory path.
        /o    Output directory path.
        /b    Principle (bucketized) log directory path.
        /d    Dynamic agent log directory path.

So you put your custom collector XML in a folder and bucketizer.exe processes it and dumps a bucketized XML in another folder.

An interesting Bucketizer.exe switch is /m. Running bucketizer.exe without parameters it says that it is used for “Custom Md5 attributes xml file path. This suppresses built-in Md5
attributes.”. I have not being able to get further information about this switch though.

Caveats

There are some caveats when using this hack that may cause your custom collector log unusable or without value, though, including:

NVVL data has to be exactly the same as the one used to generate hashes used as AppIDs in ACT web service. Any minimal difference in any NVVL field will generate a completely different AppID that then will not find a match in ACT web service. This applies also to standard ACT usage. If you deploy a vendor application using a customized MSI package and you change how they look like in the system, DCP will collect customized information and therefore generate a different AppID that ACM will not be able to match against the web service.
MSI vs. AddRemoveProgram block. AppID generated by bucketizer.exe is different if you use the very same NVVL data either in an MSI or an AddRemoveProgram block. If you don’t know wheather a particular app is installed through an MSI or not, you may need to generate both blocks and therefore have more noise in ACM.
Expect log processing errors. Using this technique, some data generated in the custom collector log may not get properly processed. During my tests, I built an XML file with 1000 application blocks, and while bucketizer.exe did not complain, the generated log failed to get into ACM. However, it did work using just the first 500 application blocks, so you may need to find offending records.

The Case Of… Windows 7 Network Connections empty folder

2010-01-11T13:10:49Z

Well... this is not an Identity Management specific post, but I would like to make my blog more content wide and cover other areas of work and experience, so here it goes the first non-Identity post :).

For some unknown reason, it happened to me that my “Network Connections” folder in Windows 7 x64 got empty., Network connections all still work, and I could even go and change some settings through netsh, but I was commited to fix the issue. Hitting refresh button updated the status line of the window with “0 connections” message.

After some “Bingging” ;-), I found similar issues and fixes for XP both in Microsoft Support Knowlege Base and some forums that were not applicable.

System reboots, updating network drivers, removing and adding interfaces in Device Manager, removing network services related stuff such as network virtualization bridges, re-registering netshell.dll, netcfgx.dll and netman.dll, adding Network Service account to Local Administrators group (this one fixed another “Network and Sharing Center” empty screen issue in the past) were not helpful neither.

Normally system restore points will have being really helpful here, but due to lack of disk space, I had not enough restore points to get back to a healthful state, specially after last three Windows Update packages came through.

So after several hours and system reboots I was almost ready to give up when I decided to give Process Monitor and myself a last chance before rebuilding my system. I added svchost.exe and svchost.exe’s instance that holds “Network Connections” service (NetMan) in ProcMon filters, like this:

I just got the needed svchost.exe instance PID by looking for Netman service in Process Explorer tooltip, like this:

Then I looked at registry calls when hitting refresh button in ncpa.cpl window (that is how you can launch “Network Connections” dialog quickly) and comparing it with the same activity in a healthy system.

While the healthy system registry activity queried for NIC information in the registry, my faulty system was giving a BUFFER OVERFLOW result when quering for HKLM\SYSTEM\CurrentControlSet\Control\Network\Connections\ClassManagers value. A simple look to this value seems fine, but when I edited it I just found that for some reason the faulty system had all 4 GUIDs ({B4C8DF59-D16F-4042-80B7-3557A254B7C5},{BA126AD3-2166-11D1-B1D0-00805FC1270E},{BA126AD5-2166-11D1-B1D0-00805FC1270E} and {BA126ADD-2166-11D1-B1D0-00805FC1270E}) in a single line, instead of in separate lines of this REG_MULTI_SZ value as it should be.

So just fixing the REG_MULTI_SZ value to a proper format and hitting refresh in “Network Connections” dialog made network connection icons come back again!

Hope this helps.

Things to Consider when Provisioning to AD / ADAM

2009-05-12T19:56:24Z

Active Directory / ADAM provisioning can be tricky (and same may apply to other LDAP directories) when you attempt to provision objects using potentially non-unique things like names, surnames or object descriptions (such as “CN=John Smith”).

There are a few things you need to consider for best results and less errors during bulk provisioning:

RDNs have to be unique within the same container. If you have two “CN=John Smith” to get provisioned to the same container, you need to generate some sort of tie braker to rename one of them. In ILM Developer Reference there is a sample code snippet that shows how to handle this situation, by building an initial DN for the object, attempt to commit the connector, and if you get ObjectAlreadyExistsException then try to build an alternative DN using RDN with suffixes.
RDN is limited to 64 characters. You can extend this length (using ADSIEdit MMC to inspect the schema and changing the rangeUpper attribute value), but I would rather not change this particular length as other Windows components may have problems if you do so. Therefore, when building your RDN for provisioning, you will have check its length and shorten it if needed. If you apply the suffixing technique for RDNs, consider that when a collision occurs and you add a suffix, you may be hitting the 64 limit just by the suffix length, so another length to check :).
Accented RDNs generate collisions. If you provision “CN=John Müller” in a given container, trying to provision “CN=John Muller” works in your provisioning code (no ObjectAlreadyExistsException raises), but you will get export errors because the object already exists. So you may need some normalization.
RDN renames. If, like I do sometimes, your provisioning logic uses some kind of OU assignment (ie. based on countries, departments, business units and such), and your “CN=John Smith – johnd” (because you decided to use uid as suffix for example) moves away from the same OU where “CN=John Smith” lives, now both objects could be named “CN=John Smith” without collisions, so you may apply the same RDN building logic both on Connectors.Count==0 and Connectors.Count==1 (to apply RDN renames). Also same checks are needed.
Provisioining in containers. When the provisioning logic has to “calculate” the AD OU, I normally assume that the OU has being previously created in AD/ADAM from a given data source (i.e. a Departments MA). However, you may find that the “Departments Table” does not contain your “DeptXYZ”, while your “Users Table” says that “John Smith” is in “DeptXYZ”. In this case, your provisioning logic will attempt to create the object in a non-existing OU, so it will raise a MissingParentObjectException. So to handle this, your provisioning logic will need to catch both ObjectAlreadyExistsException (to change the RDN) and MissingParentObjectException (to change the “calculated” container with a default provisoining container). And again, the same logic needs to be included both with 0 or 1 connectors, to apply a proper "OU move” whenever the OU comes into your AD MA.
sAMAccountName. This is another fighting element. This attribute has to be unique across a given domain, and has characters and length restrictions. If you build this attribute based on names and surnames, you will have to apply normalization (again, jmüller and jmuller are considered the same user), length checks (max. is 20 characters) and characters validations so your provisioning logic works fine. I was not able to find detailed technical information, but these are some invalid characters and conditions:

It accepts unicode characters, but accented characters generate collisions. So you can provision jmüller today and if tomorrow you try to provision jmuller, you will get a collision. So better normalize your sAMAccountNames.
The same sAMAccountName is valid across multiple domains (while better to avoid migration/consolidation issues later). This point leads to handling sAMAccountName uniqueness, but that is another story.
These characters are invalid: \/:*?\"<>|=;
Blanks (space) are allowed.
The sAMAccountName cannot end with . (dot, full stop, period).

mailNickname. If you happen to provision Exchange mailboxes too, mailNickname is another interesting attribute. Again, if you build this attribute based on names and surnames, you will have to apply normalization, length and characters validations so your provisioning logic works fine. This attribute has to conform the following:

Ensure contains only format and lengh (up to 64)
May use any of these ASCII characters:

Uppercase and lowercase English letters (a-z, A-Z)
Digits 0 through 9
Characters ! # $ % & ' * + - / = ? ^ _ ` { | } ~
Character . (dot, period, full stop), provided that it is not the first or last character, and provided also that it does not appear two or more times consecutively.

mailNickname uniqueness is important too. While it does not have to be unique by itself as sAMAccountName, it is used by Exchange to build the primary SMTP address. So it is very likely that you get SMTP address collisions if you have repeated mailNickname values. Again, uniqueness story here :)

Dealing with OpenLDAP XMA 1.1: Avoiding duplicate-object errors and bad csentry object type

2009-04-30T18:39:12Z

When you first try to import from an LDAP using this XMA, you may get duplicated objects. This is due to the way the XMA builds the LDAP queries that creates to perform object discovery: it will find the same object several times due to matching several object classes.

You can fix it by tweaking the "Map Object Types" dialog in your XMA and also adjusting the undocumented excludedTypes parameter.

The XMA builds a search filters hashtable with object types (not object classes you have in your LDAP) specified in "Map Object Types" dialog as the objectclass to search for. It excludes there those object types specified in the excludedTypes custom parameter.

In the default configuration, object types are “organizationalUnit”, “top”, “inetOrgPerson” and “organizationalRole”, and "top" is in ExcludedTypes.

Therefore, the XMA performs one search per non-excluded object type, per naming context, such as: "(|(objectclass=inetOrgPerson))" "(|(objectclass=organizationalRole))" "(|(objectclass=organizationalUnit))".

When defining the object class to object type mapping, you have to specify all possible sequences (or begening part of sequences) as they appear in the import file, and map to the desired object type. This makes the csentry object type to be mapped properly to the object type you want to (so your rules can work based on object types), otherwise you end up seeing users as "top" object types, and such. Also carefully exclude in excludedTypes those object classes that render a single object to be found more than once as per search filters.

For example, if your users objects can be like three sets of object classes:

<attr name="objectClass">
  <value>top</value>
  <value>person</value>
  <value>organizationalPerson</value>
  <value>CustomPerson</value>
  <value>mdsCVLink</value>
  <value>wdmperson</value>
  <value>mdsMVLink</value>
  <value>inetOrgPerson</value>
  <value>mdsSelfCareUser</value>
  <value>CpPSMPrefs</value>
  <value>CpEMIPrefs</value>
  <value>CpUSRUserRecordAbs</value>
  <value>CpUSRUserRecordAux</value>
</attr>
 
<attr name="objectClass">
  <value>top</value>
  <value>person</value>
  <value>organizationalPerson</value>
  <value>CustomPerson</value>
  <value>mdsCVLink</value>
  <value>inetOrgPerson</value>
  <value>mdsSelfCareUser</value>
  <value>CpPSMPrefs</value>
  <value>CpEMIPrefs</value>
  <value>CpUSRUserRecordAbs</value>
  <value>CpUSRUserRecordAux</value>
</attr>
 
<attr name="objectClass">
  <value>top</value>
  <value>person</value>
  <value>organizationalPerson</value>
  <value>CustomPerson</value>
  <value>mdsCVLink</value>
  <value>wdmperson</value>
  <value>mdsMVLink</value>
  <value>CpPSMPrefs</value>
  <value>CpEMIPrefs</value>
  <value>CpUSRUserRecordAbs</value>
</attr>

You can specify as mapping "top,person,organizationalPerson,CustomPerson" –> “CustomPerson”. This will map all three "supersets" of object classes to CustomPerson, providing that the mapping matches the begining values of objectClass tag.

However, if an object for some reason comes in the DSML like:

<value>person</value>

<value>CustomPerson</value>

<value>organizationalPerson</value>

<value>mdsCVLink</value>

<value>wdmperson</value>

<value>mdsMVLink</value>

<value>CpPSMPrefs</value>

<value>CpEMIPrefs</value>

<value>CpUSRUserRecordAbs</value>

</attr>

You will need to specify an additional mapping like "top,person,CustomPerson,organizationalPerson" –> “CustomPerson”, or you will see your customPerson objects to be mapped to an unwanted object type. You can check if this is occurring by performing a “Search Connector Space” and sort by object type.

Dealing with OpenLDAP XMA 1.1: Undocumented Parameters

2009-04-30T18:15:00Z

Along with the parameters available in the Configuration Guide, there are a few that are not documented but can be kind of useful:

excludedTypes. Object types separated by comma. Provides a way to filter out certain object types in LDAP searches. This typically helps solving duplicated objects in DSML import file.
storedChangeNumberOverride. You can put a number that matches the lastChangeNumber you are interested in. Makes the XMA try to use an override for 'lastChangeNumber' when using changelog. This can be used for troubleshooting to roll through changes again.
deltaAttributeFormat. The default format for the deltaAttribute parameter is a string that represents a timestamp in "yyyyMMddHHmmss" format. The deltaAttributeFormat custom parameter allows to override this format when the XMA uses System.DateTime.TryParseExact() to build watermarks.
ocFilter-<object type>. Allows to define fine grain filters for the object types you get in your import file. Multiple filters can be defined for a single object type by separating LDAP filters with "~". For example:
- If you want only inetOrgPerson objects wich shoeSize attribute is between 35 and 40, and those that have 42, you can define the following custom parameter:
  ocFilter-inetOrgPerson = (&(shoeSize>=35)(shoeSize<=40))~(shoeSize=42)
  
  This parameter will produce two searches indeed:
  (&(objectclass=inetOrgPerson)(&(shoeSize>=35)(shoeSize<=40))) and
  (&(objectclass=inetOrgPerson)(shoeSize=42))
  
  This is the code snippet that builds the searchFilters when processing the ocFilter-* parameters (put into m_SearchFilters[] array):

Dealing with OpenLDAP XMA 1.1: Working with Deltas

2009-04-30T18:01:35Z

OpenLDAP XMA can run delta imports using a timestamp attribute you define or the changelog.

The MA uses the attribute specified in “deltaAttribute” custom parameter in order to implement watermarks and detect object changes. You may experience a couple of issues using a delta attribute:

If the attribute does not exist for all objects you import, you will need to specify a common timestamp attribute, or use separate MAs if your timestamp attribute differs among the objects you want to import.
If the timestamp attribute is in string format different to "yyyyMMddHHmmss", you will have to set the undocumented “deltaAttributeFormat” custom parameter so the XMA is able to parse it when using System.DateTime.TryParseExact() to calculate watermarks.
If the timestamp attribute is in an format that System.DateTime.TryParseExact() cannot parse, you will have to modify the XMA code to convert the attribute into a valid datetime value.

If you find like me that the LDAP implementation you are connecting to does not use a timestamp attribute across all objects, your other option to use deltas is using changelog. In this case, what you have to do is leave “deltaAttribute” custom parameter blank, so the XMA will use changelog. If the account you are using in the XMA to access the LDAP is not administrative, you may need to adjust the permissions in cn=changelog to allow the XMA to work, adjusting the ACI to allow your MA account to read, compare and search all attributes for objects under cn=changelog.

Dealing with OpenLDAP XMA 1.1: Inspecting Records in Inport File

2009-04-30T17:41:28Z

There is a great tool in MIIS 2003 Resource Kit 2.0 called FileViewer.exe. This tool provides a way to display or dump an entry or a range of entries from a file-based MA. This is specially useful when a particular object is generating some import/sync problems and you want to inspect the input file record but it is too big to be managed in your favorite text editor.

FileViewer.exe uses a File-based MA XML export to in order to parse the import file properly. However, for XMAs such as OpenLDAP XMA, it does not recognizes the XML as a usable MA, so you have to do a little trick:

Create a “File-Based” MA for the file format your XMA generates. In this case, DSML 2.0.
Input the XMA-generated file as template for your new File MA.
Expor the File MA and use it as MA definition for FileViewer.exe.

Dealing with OpenLDAP XMA 1.1: Template Bugs and Quick Fix

2009-04-15T13:32:49Z

I was not able to find much guidance out there about the issues I found when attempting to use this great XMA, so I decided to write this post in case others find it useful.

After installing OpenLDAP XMA (http://openldap-xma.sourceforge.net) MSI package, you will find a new MA available in ILM list of MAs, called “OpenLDAP XMA”. When creating the MA to connect to your LDAP directory, you have to input the connection information: server, port, user and password. After that, you will find the “Configure Additional Parameters” dialog and the first issues with it.

The XMA template (%ProgramFiles%\Microsoft Identity Integration Server\UIShell\XMLs\PackagedMAs\OLXMAPackage.xml) contains a few bugs that make the MA fail with “stopped-extensible-extension-error” errors:

SASL. The source code does not treat this parameter at all, so having it makes the MA fail to run. This parameter is not mentioned in the Configuration Guide. As this is a parameter built into the XMA template, the “Remove” button is grayed-out (See notes about SASL support at http://blogs.msdn.com/adamw/archive/2007/06/27/openldap-milestone-1-release-scheduled-for-6-29-07.aspx).
namingContext. This parameter should be called namingContexts. This parameter is properly documented in the Configuration Guide.
pagedSearch. It says “true” as default value. However, treating this parameter as boolean is commented out in the code, so if you use “true” or anything other than “NONE”, “PAGING” or “TRAWLING”, will default to “NONE”, so pagedSearch=true ends up not using paged searches at all. If the directory requires using paged searches, the MA will fail. This parameter is properly documented in the Configuration Guide.

So here I have a OLXMAPackage.xml replacement that you can drop in your “%ProgramFiles%\Microsoft Identity Integration Server\UIShell\XMLs\PackagedMAs” directory overwriting the existing one. It just removes SASL parameter, fixes the typo error for namingContexts parameter and sets default pagedSearch to “PAGING”, rather than “true” .

Connecting to Critical Path Directory Server with ILM 2007 FP1

2009-04-15T10:57:28Z

In my current customer I have to extract identity information from Critical Path directory. I was not able to find much information about this LDAP directory from the vendor’s website (http://www.criticalpath.net/Services/Training/Directory%20Server.html) neither in WikiPedia (http://en.wikipedia.org/wiki/Directory_service), so I first gave a try to built-in ILM 2007 FP1 MAs.

In particular, I was previously able to connect ILM to some Netscape-based LDAP directories such as Fedora Directory Server and OpenLDAP by using the “Sun and Netscape directory servers” Management Agent just by tweaking the iPlanetMASupportedServers registry key (http://support.microsoft.com/?kbid=842531) with what LDP shows in VendorVersion RootDSE property when connecting to it, such as in this example:

I tried this, to discover that CP directory is not Netscape based directory, or at least it does not expose the VendorVersion property in the RootDSE that this MA looks for. I also gave a little chance to other built-in LDAP directories MAs such as “IBM Directory Server” or “Novell eDirectory” MA without success, but I had to try ;)

So before attempting to write an extensible MA or using a file-based MA, I then decided to give a try to OpenLDAP XMA (http://openldap-xma.sourceforge.net/). Despite of its name, this MA uses Microsoft .NET Framework 2.0 System.DirectoryServices.Protocols, so it works with other LDAP directories, even with Active Directory or ADAM if you want to.

After a few dives into this MA code and some testing, I was able to successfully import the identities I needed from Critical Path’s Directory Server. I will post the issues found during this MA usage in a future post.