There have been quite a few discussions, blogs, and articles on some of the challenges with deploying and running Exchange on a domain controller. In 2006, Robert Moir wrote a detailed blog titled “Running Exchange on a Domain Controller.” In his blog, he addressed, at a high level, some of the challenges with performance and security when running Exchange and Active Directory on a domain controller. Note that this article is not simply about deploying Exchange Server 2007 in Active Directory (not like what Marc Grote talks about in his article on “how Exchange Server 2007 extends the Active Directory Schema”). The distinction we are making here is that we are running Exchange on the domain controller.
The Small Business Server (SBS) was the first Microsoft product to officially deploy and support Exchange and Active Directory on a domain controller; however, due to the single domain controller nature of the product, it did not face some of the additional challenges described in this article. During the development of Essential Business Server (EBS), which deploys Exchange 2007 on a domain controller in a multiple domain controller environment, we faced a new set of challenges and developed resolutions for them. These resolutions can also be applied to any enterprise-level deployments with Exchange 2007 on a domain controller. In a mid-size and enterprise domain, it is strongly recommended to keep at least one replica domain controller around for fault tolerance and load balancing. However, due to the replication latency between domain controllers, deployment and maintenance of Exchange on a replica Domain Controller becomes more challenging. Moreover, Exchange disaster recovery becomes significantly different than recovery from a member server.
One of the main challenges we faced in running Exchange 2007 on a Domain Controller was its recovery after a disaster. The recovery of Exchange when installed on a replica Domain Controller has a special formula that is quite different from any other topology. Exchange 2007 supports the /mode:RecoverServer that will automatically deploy the server binaries and recover all server settings from Active Directory, however, the Exchange computer object must be preserved and untouched in Active Directory for Exchange recovery to succeed. On the contrary, Active Directory does not recommend “taking over an existing slot” and re-using the existing computer object when promoting a failed Domain Controller. The recommended approach in recovering a failed (or unsuccessfully demoted) Domain Controller is to run ntdsutils.exe and cleanup the leftover computer object, the related NTDS settings object, the replication channels, and all other traces of the failed Domain Controller from Active Directory. Clearly, this approach will render the Exchange server unrecoverable.
In EBS, we solve this problem by examining the state of the Active Directory after disaster, transferring and (if needed) seizing the FSMO roles, and only cleaning up the NTDS setting object of the server. This object can be found under the Sites container in the configuration partition of Active Directory. We keep the computer object untouched and join the new server to the domain with the same name as the failed server. Then we promote the new server to a Domain Controller while preserving all settings and permissions on the computer object and then attempt a recovery of Exchange. With this approach, the domain controller and Exchange server recover successfully. EBS providers this solution under the hood through its recovery process and the users need not to worry about the details.
Another challenge we faced in running Exchange on a Domain Controller was the service dependency between Active Directory and Exchange 2007 services. This dependency can cause some of the core Exchange services (such as the transport service and information Store) to not start properly after a reboot. The reason for this is that during a reboot, all of Active Directory and Exchange 2007 services on that server will be restarted. Upon restart, some of the Active Directory services such as Kerberos Domain Controller take longer to reach the running state if the domain controller is deployed in a domain with other replicas. This is probably associated to the fact that the booting Kerberos Domain Controller must complete a full replication cycle before serving clients and issuing tokens. In the meantime, Exchange Active Directory Topology services will attempt to contact the closest Kerberos Domain Controller, in this case the local host's Kerberos Domain Controller service which results in a hard-to-detect race condition. In a multi domain controller domain, it is very likely that the local Kerberos Domain Controller isn’t responsive in time for the Exchange Active Directory Topology service and unfortunately this service fails to start and gives up. Many key Exchange 2007 services such as transport and Information store depend on the Exchange Active Directory topology service and they all fail to start. Note that this problem is only visible when running Exchange on a domain controller and it is emphasized when running Exchange on a replica domain controller. There are a few ways to solve this race condition. In EBS, we chose to change all Exchange service start types from Automatic to Automatic delay start. This change allows Kerberos Domain Controller service plenty of time to come up before Exchange Active Directory Topology service and other subsequent Exchange services start querying it.
Another problem with installing Exchange on a domain controller surfaces during the installation of Exchange 2007. During installation, Exchange 2007 will first attempt to prepare the Active Directory and extend the schema. For this it will target the schema master domain controller. The schema master is one of the FSMO roles assigned to only one domain controller in the entire forest, which may even be in a remote location. After the schema has been extended, Exchange will proceed with the remainder of the installation, however, there is no guarantee that the remainder of Active Directory changes will also be targeted to the same domain controller, especially when Exchange installation is running on a local host domain controller. Due to delays in replication and depending on the proximity of the FSMO role owners to the local host domain controller, the Exchange 2007 installation can hiccup and fail with “object not found” and “no permission” errors. This is because the changes made during the previous stages of installation haven’t yet made it to the local domain controller. Note that Exchange 2007 installation is very resilient and retry-able, so almost any failure can be retried until it succeeds, however in a multi domain controller domain with scattered FSMO roles, Exchange 2007 installation on a domain controller can leave the administrator baffled with random race conditions, before it succeeds. To eliminate this problem, we (temporarily) gather all FSMO roles on the local domain controller and target the Exchange 2007 installation to the local domain controller. More specifically, after promoting the local domain controller, we ensure a full cycle of replication has completed, and the domain controller is properly advertising and responding, then we transfer the roles over to the local domain controller and ensure all the other domain controllers (at least the ones in local Active Directory site) are aware of the FSMO role ownership change, and then we install Exchange 2007 with the /DC:”local DC name.” At this point, we are completely confident that all Exchange changes will target the local domain controller and Exchange setup will complete faster (eliminating network traffic) and without any random failures.
Even though installing domain controller and Exchange 2007 might be resource intensive on the server, it somewhat helps reduce some network traffic as long as you make the domain controller a global catalog. This is because Exchange 2007 produces plenty of traffic between itself and the closest global catalog in the domain and this network traffic is eliminated if Exchange is installed on a Global catalog. Therefore, it is important to make the local domain controller a GC if you plan on installing Exchange on it.
Alireza Farhangi
There is an incentives program in the US and Canada called “The Big Easy”. The great news is that EBS is one of the products that is included in this program. From the web site “For every qualifying product you purchase, Microsoft pays you partner subsidy funds that can be used with a Microsoft Partner of your choice to help with the implementation of your Microsoft solution”.
With this program, Microsoft is subsidizing the cost of a partner that deploys EBS for a customer.
Check it out:
http://www.microsoftincentives.com/BigEasy3/
Ed Tremblay
Program Manager
EBS
Hi, I’m John Endter, president of E Squared C LLC, a Microsoft Gold Certified partner and Small Business Specialist located out of Minden, Nevada.
In today’s economy, organizations are spending money wisely. Investment in virtualization, mobility, and security is up and the Harris Poll survey supports this notion. As the economy improves, IT investment will lead us to more prosperous times.
The survey reiterates what I have been seeing in the technology sector. With today’s economy being what it is, companies must find ways to become more efficient, cut costs, and be more productive while at the same time maintaining a secure computing environment.
To begin, one way companies can gain efficiencies and cut costs is through virtualization. The survey reports that 42% of those surveyed plan on investing in virtualization. Much of the server infrastructure we are dealing with is approaching 6 years old and some as old as 8 years. IT departments are looking for ways to simultaneously replace their aging infrastructure and improve the efficiency of their systems. Virtualization brings this to table and the survey supports what we are seeing in the industry.
Next, increasing employee productivity is another area of focus for IT departments. The survey points out that an overwhelming 92% of IT pros surveyed cited improving end-user productivity as a reason for innovation. We are seeing many companies getting very creative with how and where their employees work. Much of the IT spending is budgeted for mobility and remote workers. Windows Server 2008 application virtualization is the hot topic here.
Finally, the survey points out the importance of maintaining a secure computing environment. 73% of those surveyed say that protecting customer and company data is their top priority. As more security breaches are reported in the media, companies are learning the importance of investing in technologies that protect their data. The cost of reactive security breach mitigation is far greater than the cost of proactive security protection and companies are finally realizing this fact.
-John
For more information about the survey, visit http://www.microsoft.com/infrastructure/resources/itprosurvey.mspx
Please join us tomorrow, June 23rd from 10:30am - 11:00am (PDT) for a teleconference with Bob Kelly, corporate VP of Infrastructure Server Marketing.
Feel free to submit questions for the QA session following the presentation over the phone, or via Twitter any time leading up to or during the teleconference by tweeting with the hashtag, #qs4ms.
If you are interested in attending, please REGISTER NOW.
Hi, my name is Ozan Eren Bilgen and I am a software developer on the Windows Essential Business Server 2008 team. Today I want to talk about how EBS 2008 can help IT administrators in email security and malware protection for Exchange Server 2007.
Email is one of the most important communication tools in our business and personal life. That itself makes email a popular tool for salesmen, scammers and hackers. As of April 2008, more than 90% of emails are categorized as email spam or infected by malware. In other words, companies spend 90% of the email budget for the emails that they actually prefer to not receive at all. EBS 2008 provides email security while keeping IT budgets under control.
Email spam is usually seen in forms of fraudulent or unsolicited bulk emails. If the company network is not protected by an Exchange Spam Filter or some other anti-spam products, the mail server disk space may be filled up with spam emails and important business emails may not be delivered. High email traffic will also consume network bandwidth and slow down Internet connection speed. Unwanted emails take employees’ time and can be misleading for them. Windows Essential Business Server 2008 prevents spam emails with Microsoft Exchange Server 2007 Anti-spam Protection, which stops the emails before they are delivered to inbox. Thus, companies that are upgraded to EBS 2008 can save IT budgets for improving the business, and not for storing spam emails.
Another email related threat comes with attachments that are classified as malicious software (a.k.a. malware). Malware is a generic term for any software that can infiltrate or damage a computer system, such as viruses, worms, spywares, rootkits or any other type of dangerous software. A single malware incident can quickly take down a whole IT infrastructure, which makes malware protection an important part of email security. Windows Essential Business Server 2008 ships with Microsoft Forefront Server Security for Exchange Server 2007 that protects the inboxes against infected emails. FSE scans incoming emails and purges malware before the email is stored. FSE’s file filters will block attachments that might potentially be malware, such as .exe and .scr files. FSE updates malware signatures and scans inboxes periodically so that a new virus outbreak cannot affect the company network.
Anti-malware and Anti-spam security adapters in Admin Console complete the email security integration in Windows Essential Business Server 2008. Both adapters display critical issues related to spam and malware protection and launch Exchange Server 2007 and FSE 2007 management consoles for fine tuning. The adapters can restore Windows Essential Business Server 2008 default settings for email spam and email malware protection in case the user decides to revert to initial configuration. Finally the adapters display usage counters, such as the number of rejected spam emails and the number of blocked malware attachments.
Companies that do not pay attention to email security may run into problems like spam floods and virus attacks, which may threaten the company’s business and information security. Windows Essential Business Server 2008 enables secure email experience and provides easy management interfaces. Powered by Exchange Server 2007 and FSE 2007, Windows Essential Business Server 2008 gives the control in email security back to the IT administrators.
For more information about email protection and other ways that Windows EBS 2008 helps secure a company's IT infrastructure, see Security and Protection in the Windows Essential Business Server Technical Library.
Ozan Eren Bilgen
Are you interested in hearing about how other IT pros are reacting to economic conditions and where they’re investing?
Do you have questions about Microsoft’s efforts to help IT be more cost effective and deliver new solutions to business?
Is there a connection between virtualization and cloud computing?
What is Microsoft doing in enterprise security?
On Tuesday, June 23rd from 10:30am - 11:00am (PDT), join a teleconference with Bob Kelly, corporate VP of Infrastructure Server Marketing. Bob will talk about the state of IT within the context of results from a new Harris Interactive study of 1,200 IT professionals from the U.S., United Kingdom, Japan and Germany. The study was commissioned by Microsoft's Server & Tools Business.
There will be time for your questions following the brief presentation. Submit questions over the phone or you can submit them at any time leading up to or during the teleconference by tweeting with the Twitter hashtag, #qs4ms.
If you are interested in attending, please REGISTER NOW. Once you open the invite box, you can save and close to your calendar.
The EBS product is unique in that it is intended to provide a complete Windows based IT infrastructure for medium sized companies that is deployed all at once as a single product. During the course of the EBS deployment Windows 2008, Exchange 2007, System Center Essentials, Forefront Threat Management Gateway along with unique EBS software are installed and configured in a “best practices” deployment across three servers.
There have been questions about how to approach planning the timing of the deployment of EBS, and where in the deployment timeline are there logical “break points” where a user can pause before continuing with the deployment. To phrase the question another way, people have been asking how long can a user can “take a break” between the different deployment stages and between the tasks that are within the stages.
Here is a framework for thinking about the logical stages of EBS deployment with rough estimates of time for each of the stages and where “break points” exist during the deployment process.
Note that your mile will vary on the timing of completing the tasks within the stages. These times are used as examples for planning purpose only.
EBS “Stages of Deployment”
Figure 1 EBS Deployment Stages
There are three major stages that occur during an EBS deployment (see Figure 1 – EBS Deployment Stages).
1. Stage 1 – Prepare Environment and Plan for the Deployment
2. Stage 2 – Install the EBS server roles
3. Stage 3 – Perform final Configurations steps & Migrate workloads to EBS servers
Stage 1 – Prepare and Plan
This is the stage where the user plans & prepares for the deployment and uses EBS’s Planning and Preparation Wizards to help with the process.
1. Prepare Environment – Run the EBS Preparation Wizard to detect conditions in the Windows Server IT infrastructure that need to be corrected to ensure a successful EBS installation. The Wizard will detect blocking conditions in the existing environment and provide guidance on how to correct the conditions. The total amount of time to run the Preparation Wizard and detect and correct conditions is dependent on the state of the environment that the wizard is being run in. Some environments have many issues that need to be corrected before continuing.
Timing - Plan on spending a minimum of three hours during the prepare environment phase.
Break Point(s) – The user can run the wizard as many times as necessary (and take breaks between each run) to correct detected environmental conditions.
2. Plan for EBS Installation – Run the EBS Planning Wizard to help you gather the parameters that will be required when you run the Installation Wizard. The Planning Wizard is useful to help you think through how EBS will be integrated into your environment. The output of the Planning Wizard is a checklist that can be used when running the EBS Installation wizard.
Timing - Plan on spending a minimum of three hours during the planning phase.
Break Point(s) –The Planning Wizard needs to be run within fourteen days of completing the Preparation Wizard.
Stage 2 – Install EBS
This is the stage where the user installs the EBS server roles (Management, Security and Messaging) on three servers.
1. Install Management Server Role – Run the EBS installation wizard to install the Management Server role.
Timing - Plan on approximately three hours to complete the Management Server installation wizard.
Break Point(s) – The Management Server installation wizard needs to be run within fourteen days of completing the Planning Wizard.
2. Install Security Server Role – Run the EBS installation wizard to install the Security Server role.
Timing – Plan on approximately two hours to complete the Security Server installation wizard.
Break Point(s) – The Security Server installation cannot be completed until the Management Server install has completed. There is no enforced timing between when the Management Server has completed and when you need to start install of the Security Server.
3. Install Messaging Server Role – Run the EBS installation wizard to install the Messaging Server role.
Timing – Plan on approximately two hours to complete the Messaging Server installation wizard.
Break Point(s) – The Messaging Server installation cannot be completed until the Security Server install has completed. There is no enforced timing between when the Security Server has completed and when you need to start the install of the Messaging Server.
Stage 3 – Configure and Migrate
This is the stage where the user performs the configuration tasks that complete the EBS deployment and to migrate data from existing workloads to the newly installed EBS environment.
1. Configure EBS – Perform the configuration steps to complete the EBS deployment such as configure System Center Essentials (SCE) and configure EBS licensing (CALS).
Timing - Plan on approximately two hours to complete the Management Server configuration tasks
Break Point(s) – The initial licensing configuration must be completed within 30 days. Otherwise, there are no dependencies or forced timing for completing the individual configuration tasks.
2. Migrate to EBS – Perform the steps to migrate the data from existing workloads to the equivalent workloads running in EBS.
Timing – The timing for migration is variable and depends on which workloads you are planning on migrating to EBS. For example, if the user plans on continuing to use an existing DHCP service, then there is no need to plan for the DHCP migration. Another timing variable to consider is how big the data sets are that you are migrating. For example, the timing of the Exchange mailbox migration is dependent on the size of each user’s mailboxes and how big the existing Exchange mailbox database is. Plan for a minimum of three hours to complete the Management Server migration tasks.
Break Point(s) – There are no dependencies or forced timing for completing the individual migration tasks. The user can complete them on a schedule that makes the most sense for their business.
[Today’s post comes to us courtesy of Justin Crosby and Mark Stanfill]
Symptoms
The EBS Administration Console may crash if illegal characters are present in devices reported back to WSUS and SCE. The symptoms are that the console will fail to launch, and C:\Program Files\Windows Essential Business Server\logs\AdminConsole.log will have a line similar to “System.Xml.XmlException: '', hexadecimal value 0x01, is an invalid character”. (If you don’t see this exception, this article does not apply).
C:\Program Files\Windows Essential Business Server\logs\AdminConsole.log will show an error like the following (scroll to the bottom and search up):
[16600],"2009/05/29 08:03:52.981","AdminConsole","Error","!!!!FATAL: Console shutting down due to unhandled exception: There is an error in XML document (1, 91809)."
[16600],"2009/05/29 08:03:53.022","AdminConsole","Error","System.InvalidOperationException: There is an error in XML document (1, 91809). ---> System.Xml.XmlException: '', hexadecimal value 0x01, is an invalid character. Line 1, position 91809.
at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.ThrowInvalidChar(Int32 pos, Char invChar)
at System.Xml.XmlTextReaderImpl.ParseNumericCharRefInline(Int32 startPos, Boolean expand, BufferBuilder internalSubsetBuilder, Int32& charCount, EntityType& entityType)
at System.Xml.XmlTextReaderImpl.ParseCharRefInline(Int32 startPos, Int32& charCount, EntityType& entityType)
at System.Xml.XmlTextReaderImpl.ParseText(Int32& startPos, Int32& endPos, Int32& outOrChars)
at System.Xml.XmlTextReaderImpl.ParseText()
at System.Xml.XmlTextReaderImpl.ParseElementContent()
at System.Xml.XmlReader.ReadStartElement()
at System.Xml.Serialization.XmlSerializationReader.ReadStringValue()
at System.Xml.Serialization.XmlSerializationReader.ReadTypedPrimitive(XmlQualifiedName type, Boolean elementCanBeType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderApiRemotingCompressionProxy.Read1_Object(Boolean isNullable, Boolean checkType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderApiRemotingCompressionProxy.Read2_GenericReadableRow(Boolean isNullable, Boolean checkType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderApiRemotingCompressionProxy.Read137_Item()
at Microsoft.Xml.Serialization.GeneratedAssembly.ArrayOfObjectSerializer204.Deserialize(XmlSerializationReader reader)
at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
--- End of inner exception stack trace ---
Cause
This exception happens when a client computer reports its inventory to WSUS, and an illegal character is present in part of the data returned to WSUS. SCE in turn synchronizes this data to its database, which is queried by the console. The cases we have seen have all involved printer location strings.
Resolution
To allow the console to load properly, it is necessary to identify the problem machines, edit the bad value, and then report back to WSUS.
Identifying the problem machine
Obtain the hex code of the number from the AdminConsole.log. In the example above, it is 0x1:
System.Xml.XmlException: '', hexadecimal value 0x01, is an invalid character
Save the code below as “badchar.sql”. If your hex number is other than 0x1, determine the decimal value and substitute it for CHAR(1) in the last line of the script.
USE SUSDB
SELECT tbInventoryProperty.Name AS Field, tbInventoryClassInstance.KeyValue AS Object, tbInventoryPropertyInstance.Value AS [Bad Character],
tbComputerTarget.FullDomainName
FROM tbTarget INNER JOIN
tbInventoryClassInstance ON tbTarget.TargetID = tbInventoryClassInstance.TargetID INNER JOIN
tbInventoryPropertyInstance ON tbInventoryClassInstance.ClassInstanceID = tbInventoryPropertyInstance.ClassInstanceID INNER JOIN
tbComputerTarget ON tbTarget.TargetID = tbComputerTarget.TargetID AND tbInventoryClassInstance.TargetID = tbComputerTarget.TargetID INNER JOIN
tbComputerTargetDetail ON tbTarget.TargetID = tbComputerTargetDetail.TargetID INNER JOIN
tbInventoryProperty ON tbInventoryPropertyInstance.PropertyID = tbInventoryProperty.PropertyID INNER JOIN
tbInventoryClass ON tbInventoryClassInstance.ClassID = tbInventoryClass.ClassID AND tbInventoryProperty.ClassID = tbInventoryClass.ClassID
WHERE (tbInventoryPropertyInstance.Value LIKE '%' + CHAR(1) + '%')
Save the script on the Management Server and run the following command:
sqlcmd -S mgmt\sce -E -i badchar.sql –Y 20
Note the value of the field, object, and bad character. Field will correspond to a registry value, object will correspond to a registry key, and bad character will correspond to the data.
Edit the bad value
On the affected client machine, open Regedit. Export the affected key to make a backup. Locate the corresponding registry value and verify the bad character by right-clicking and choosing ‘Modify Binary Data’. The hex character should be present in the output. Cancel the edit binary value window and right-click the value to choose ‘Modify’. Enter any generic text (i.e. “Home office”). and choose OK to save.
Change to a standard string using Modify: