The Butterfly Effect applied to OCS Edge Server R2

Chances are that, if you live in one of the green countries from the picture below (courtesy of Wikipedia), and if you tried to setup the OCS 2007 R2 Edge server role, you probably felt the frustration, combined with a little bit of despair and asked yourself at some time: “Am I going nuts?”, “Did I suddenly lose my skills?” or “Did I fell into a wormhole and I am now living in a parallel Universe?”.

Why only the green countries? I will explain in detail shortly, but in order not to create too much suspense, let me just say that these countries use a comma "," as the decimal separator and not a period ".".

I've been planning on blogging about this issue for some months now, but for one reason or another, I didn't manage to do it until now. Meanwhile, some other people decide to share their experiences about the very same issue, writing them in some blog posts that I would like to recommend:

• A/V Authentication Service can't be reached...due to a comma
• OCS 2007 R2 and Localization

I would also like to add that there is now a permanent fix for the problem I'm about to describe.

The Symptoms

Imagine the perfect installation of OCS 2007 R2, where all the requirements were met, where every single detail was taken care of, where every step of the official deployment guide was followed. But, at the very end, in the test phase, you taste the bitter taste of failure as some errors emerge:

1. Users see a “Limited External Calling” error in Communicator. If they click details, they get the message: "Some calls to and from people outside of your corporate network may not connect due to server connectivity problems. Try signing out and signing back in. If this problem continues, contact your system administrator with this information".
   
   
   
2. When an external user tries to call an internal user, the call is unsuccessful with the error: "The call was disconnected because you stopped receiving audio from <user>. Please try the call again".
    
3. When you validate the A/V Conferencing Server from the OCS Front-End, you get a "Validation Wizard completed with failures".
   
   The Office Communications Server 2007 R2 Deployment Log shows the following error:
   Failure
   [0xC3FC200D] One or more errors were detected

   A/V Authentication Edge Server: Could not contact A/V Authentication Edge Server.
   To resolve this error, check for the following
   1. The outbound proxy is reachable.
   2. The outbound proxy and A/V Authentication Edge Server are in trusted server list of each other.
   3. The outbound proxy and A/V Authentication Edge Server have valid certificates.
   4. Conference Server certificate is valid.
   5. A/V Authentication Edge Server Gruu is correct.

4. At the Mediation server, the Event 25015 is logged (this is actually the responsible for the Limited External Calling issue previously described): 
   "The A/V Authentication Service returned invalid response. Connections that require Firewall traversal will not be successful.
   Response Code: 400
   Response Text: Bad Request
   Cause: Either the A/V Authentication Service is not running or unreachable."
    

The Cause

Although OCS has lots and lots of (good) troubleshooting tools, sometimes it's not easy to spot the error, even when it is right in front of your eyes.

Analyzing the Communicator log from the external user, revealed the "Version Mismatch" error. The error has to do with localization (decimal separator), which caused a string comparison to fail: "2,0" with a comma it's different from "2.0" with a period! I must confess I would never get there without the help from a Microsoft colleague.

Here's the relevant excerpt from the communicator log and a glance of the Snooper Tool:

A Workaround

To workaround this problem, change the locale setting to English/US for the local account that the A/V Edge service and the A/V Edge Authentication service use (RTCProxyService).

1. Give the RTCProxyService account local login rights (it’s quicker to just add it to the local Administrators group).
2. Logon to the OCS Edge server with the RTCProxyService credentials.
3. Open Control Panel > Regional and Language Options and change the Current format to English (United States).
   
4. Log off the RTCProxyService, log back in with the Administrator, remove the RTCProxyService from the Administrators group, restart all the OCS services and voilá! If everything else is well configured, you now have a fully functional OCS solution.

The (Definitive) Solution

Fortunately, the April updates for Communications Server 2007 R2 include a patch for this specific problem. Read the following KB article for more information: The Communications Server 2007 R2 - A/V Edge Authentication Server does not recognize a token request if the locale for RTCProxyService is not en-US/409 (and then apply the KB967831 hotfix).

The Wrong Way!

When you apparently reached a dead-end and start doubting about yourself, you usually start trying some silly things. One of the approaches I took was to add the OCS Edge server to the list of authorized hosts. Don’t do this!!! The Edge server should never be placed in the Host Authorization tab, doing so will break the communication workflow.

The (Wrong) Usual Suspect: NAT

At the beginning of the troubleshoot process, my prime suspect was NAT, since I was using for the first time one of the coolest features of OCS 2007 R2 (the A/V Edge interface can now be NAT'ed) and the symptoms were typical of NAT'ed A/V Edge interface.

There are some requirements for this particular configuration to work, I strongly recommend the reading of the excellent post by Rick Varvel: Configuring R2 A/V Edge Service for NAT.

At the end, NAT had nothing to do with the problem, I can assure you this features works perfectly.

The Butterfly Effect

The Butterfly Effect has origin in chaos theory and describes how small variations of the initial condition of a dynamical system may produce large variations in the long term behavior of the system. The phrase refers to the idea that a butterfly's wings might create tiny changes in the atmosphere that may ultimately create a tornado in a certain location.

In other words, sometimes a comma makes all the difference!