I was recently installing a ConfigMgr 2012 site for a customer and ran into an unusual error towards the end of the installation that i hadn't seen before and wanted to share this incase others ran into this similar issue. The following error message will be displayed in your ConfigMgr Setup log if you have a Password Filter configured on your system. For more information on Password filters and what they are check out the following Link.
The error "Failed to set up SQL Server certificate for service broker" is due to the fact that setup is trying to create a SQL Server Broker certificate and cannot set the password for the certificate because of the installed Password Filter on the system.
In order to workaround this problem you will need to temporarily remove the password filter so that setup can create the SQL Server Broker certificate.
1) Take a backup of the following key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa
2) Open the "Notification Packages" Multi String Value at the above registry location
3) Remove your custom password filter from the list, by default scecli is the only option listed in the Notification Packages value.
4) Re-run setup and choose to uninstall your site to remove the failed installation after this has completed run setup again to complete your site server installation.
5) Optionally Restore your previously backed up registry key.
I am experiencing the exact same error message but we do not have "CustomPasswordFilter" set in our SCCM server, SQL server, nor AD server. Do you have any other suggestions?
sory for the late response have been backed up a bit... i would validate that you have connectivity to the SQL server and that you have a static port set for SQL if remote. Additionally the ConfigMgr Setup log might have more info.