My name is Satish Petwe and I wanted to share some details about our deployment experience of System Center 2012 Endpoint Protection (SCEP) in the client environment at Microsoft. We have deployed SCEP on about 280,000 machines as I write this post.
From 2009 to 2010, we deployed Forefront Endpoint Protection 2010 (FEP) for the malware and virus protection of the client environment and we managed the deployment of client and Antimalware policy with Configuration Manager 2007. In System Center 2012 Configuration Manager (ConfigMgr), SCEP is included in the ConfigMgr install package and thus implementation has become integral part of our deployment.
One of the important roles our group plays is to deploy pre-released products internally to our users at large scale and provide valuable feedback to the product group which can be used for product improvements. The migration process uninstalls the FEP2010 client and immediately installs the SCEP client. The SCEP client installer is pre-staged locally during ConfigMgr client installation and therefore no time is wasted in downloading client bits during migration. This ensured the continued security of our environment during migration with minimal interruption.
We are finding the new features introduced in SCEP to be very beneficial. Below are few that specifically helped us to reduce our pain points.
To start with, Client installation and AM policy application in FEP is accomplished via a script delivered through Package and Program. If policy didn’t get applied, that left us with a lot of moving parts to troubleshoot. With SCEP, Client installation and AM policy application is accomplished via standard Configuration Manager Policies delivered from the MPs. The process is more reliable and easier to troubleshoot if something goes wrong.
Further, we no longer need involvement of Configuration Manager Administrator to manage Antimalware policies thanks to the RBA feature. Our Corporate Security Engineers now have Endpoint Protection Manager Role access in the Administrator console and manage the custom Antimalware polices and initiate remediation actions such as Full scan, Quick scan and Definition updates.
AM/AV Client installation and Antimalware Policy application
SCEP client installation policy via Client Settings
Staged on the client machine by CCMSETUP
Client side monitoring
CCMEVAL task monitors and remediates Microsoft Antimalware Service and ensures that Microsoft Network Inspection (NIS) service is not disabled
High speed data channel
The AM infection messages are sent at higher priority and processed at higher priority
Role Based Administration (RBA)
Endpoint Protection Manager Role provides rights for AM policy, SCEP alert management and SCEP Dashboard monitoring
Endpoint Protection Point Site System Role
In ConfigMgr 2012 hierarchy, the administrator is not required to execute any separate installer for SCEP. In order to enable SCEP in the hierarchy, the first step is to configure a newly introduced site system role “Endpoint Protection Point” on the CAS (Central Administration Site) site server. This step installs the SCEP client on the CAS site server. The SCEP client syncs the latest definition from Windows Update/Microsoft Update every 8 hours. The EP Control Manager calls the SCEP client API to retrieve the Definition metadata every 15 minutes and creates a *.EPP file in the epmgr.box folder on the site server when a new Definition is found. EP Manager parses the file and inserts the metadata in the database.
SCEP Client Deployment Strategy
The Default Client Setting for Endpoint Protection is a site wide setting for enabling/disabling SCEP client installation on all the machines reporting to the site. When enabled, it creates the SCEP client installation policy for all the clients in the hierarchy. All the on-line clients will receive the policy at the next polling cycle (Default polling cycle: 60 minutes) and that could create a sudden load of state messages in the hierarchy.
Since we had large number of machines (280,000) in the hierarchy, we opted for the following strategy.
a) Keep the Default SCEP client setting disabled
b) Create Custom Client setting
In the custom client settings, an alternate source is enabled for first signature installation to ensure that the first signature is obtained from WSUS/MU/MMPC immediately after the client is installed.
Here are few details we’ve learned about how different settings can impact your SCEP client and first signature deployments:
1) DisableFirstSignatureUpdate: When an alternate source is disabled for the first signature, ConfigMgr software update deployment evaluation is triggered for installing the first Definition Update after SCEP client is installed. If it fails to install the first signature from CM (e.g. package being replicated or DP not available), an alternate source is used for installing first signature after 30 minutes. However, during these 30 minutes of waiting period, the system tray icon stays red (). We received few user escalations during early migration phases about the same and had to enable the first signature from alternative source (WSUS, WU/MU) during implementation.
2) InstallRetryPeriod: The property is not exposed in UI and the value defines the frequency in hours at which the client installation will be retried. The default value is 4 hours and that greatly helped in retrying the failed installations (though not many) without having to deploy any recurring Software distribution package.
3) Temporary exemption from EP migration: During migration, we had to temporarily exclude certain clients from EP install and management. You may come across similar business requirements in your organization. We found the best way to achieve this is by creating another Custom agent (Settings: Manage EP client – FALSE, Install EP client - FALSE) at higher priority and assign the settings to a collection of machines that need such exemption.
4) Since the deployment state messages are sent only at state change, if the client is constantly failing to install SCEP, there will not be change in time stamp in DB (Table: EP_DeploymentState) for failing clients. The log file, EndpointProtectionAgent.log, on client will have the following entry after failed retry attempt every six hours.
“EP State and Error Code didn't get changed, skip resend state message”
c) Create and Target Collection for SCEP migration
In Microsoft, the deployment of SCEP was closely associated with the deployment of ConfigMgr 2012 client. Initially, a pilot of 500 machines and then subsequently a pilot of 3,000 machines were initiated to ensure that the client migration did not impact the hierarchy and network negatively. Later, the deployment of ConfigMgr 2012 client happened in phases of ~ 5,000 clients per site. SCEP migration was managed by creating a dynamic collection of ConfigMgr 2012 clients (client version attribute) with Collection update schedule @ 12 hours and targeting the Custom Client Settings to the same. This provided the delay of up to 12 hours between ConfigMgr client install and SCEP client install and reduced flood of traffic such as inventory data, SUM and Application distribution state messages, SCEP state messages, status messages etc. that could be seen if both clients were migrated together in short duration.
d) Create and Target Custom Antimalware Policy
We first targeted the dynamic collection with our desired AM policy, which we imported from FEP 2010 Configuration Manager 2007 hierarchy. Since clients without the latest EP client version ignore this policy, it’s staged to be applied as soon as the EP client is installed, reducing the time between default and custom AM policy application to a few minutes.
SCEP Client installation flow
Based on the above configurations, here is the flow of events that occurred during the client migration process.
Various State messages are sent by the ConfigMgr client during SCEP installation process to the reporting site to report the installation status and SCEP client health status.
State Message Type
StateMsg Topic type
Number per client
Size per message (KB)
SCEP SU2 Deployment state: Installation status (installed, failed, Restart required, Unmanaged, Pending, not supported) and error details
Antimalware_health _status: Client version, last AM/AV signature and client configuration details
Antimalware_infection_status: Last infection status and pending action details
Policy Application: Policy name, application status, failure error code
Malware detection: Threat details, infection path, threat clean-up status
The SCEP client migration is expected to happen in the background silently. However, the pop notifications are generated by Action Center when it is configured to send/display Security messages if problems are found. We observed the following Action Center pop ups on desktop as the FEP 2010 client is uninstalled and SCEP client is installed during migration.
We had to get buy off on the user experience from Microsoft IT as it had a potential of creating large number of Helpdesk calls due to such behavior. Our Help Desk did receive a few calls and handled them efficiently by bringing awareness to the user of such behavior.
Stay tuned on my next blog on Dashboard Monitoring, Alerts and email subscription that we configured, client side details, troubleshooting tips, remediation actions……
Did you write anything related to scep troubleshooting? I am searching for article that tells about the definitions update flow via client side logs.