jigardani

OnPrem Lync Server 2010 cannot direct calls to O365 UM

Jigar Dani — Thu, 13 Oct 2011 20:14:45 GMT

Recently I have had the opportunity to work on multiple OnPrem Lync Server 2010 (Lync) deployments integrated with O365 Exchange Unified Messaging Servers (UM).

Issue

Unable to route calls from Lync to UM. If you look at traces from the Lync Edge Server (with SipStack component traced) You should see an INVITE outbound to exap.um.outlook.com (Well known UM Access Proxy).

To this request UM returns a 503 Service Unavailable bad response.

Cause

The INVITE from Lync Edge Server to exap.um.outlook.com will look like this -

INVITE sip:<ex-umcontact>@<domain>.com;ms-organization=<domain>.com

UM uses the domain recognized in ms-organization parameter to route this request to the correct UM server in the cloud where the tenant is hosted. If this domain is not set up as Authoritative for O365 then UM rejects this request with a 503.

The ms-organization domain is fetched from the Lync hosted voicemail policy set up by the administrator using New-CSHostedVoicemailPolicy or Set-CSHostedVoicemailPolicy.

Resolution

In my experience this domain needs to be set up as Authoritative and should be listed as one of the accepted domains in O365 . You can check your accepted domains in O365 using Get-AcceptedDomain.

You can change this to Authoritative on the O365 environment using -

Set-AcceptedDomain – Identity <domain in ms-organization> –DomainType Authoritative

Users unable to join Lync hosted conferences from Cisco Unified Communications Manager 8.5

Jigar Dani — Wed, 05 Oct 2011 00:41:00 GMT

In the past couple of months we have seen increase in DTMF issue relating to Cisco Call Manager v8.5 connecting to Lync Server 2010 (Lync). I worked on a few of these issues giving me an opportunity to dive deep into this integration. Cisco Unified Communication Manager (CUCM) v8.5 is supported with Lync Server 2010 only on minor build version 8.5.1.12900-7 as noted on Unified Communications Open Interoperability Program – Lync Server.

Symptom

Cisco phones and conference room endpoints, SIP Trunk terminating on CUCM en-route to Lync would not be able to join conferences hosted by Lync MCU's. You may experience that the user keeps getting prompted for entering the conference ID. Some of the phones which have a capability of RFC 2833 DTMF MTP Passthrough work just fine when that feature is turned on.

Cause

After code level investigation we found that CUCM, when transcoding DTMF digits does not send the digits in right format.

To identify the issue, you need to collect a network trace from the MediationServer of your Lync deployment. Then trace the attempt to login to a Lync hosted conference room. Below is one such trace (obfuscated from customer environment since I do not have a CUCM here), to see the packets of interest I set the wireshark filter to (udp.length != 24 && rtpevent && rtp.marker==1)

The lower window highlighted digits show the raw data captured for RFC 2833 RTP event.

I also captured a trace when a endpoint coming from CUCM was successfully able to login to Lync MCU. To trace these DTMF digits I set up the wireshark filter to (udp.length == 24 && rtpevent && rtp.marker==1)

Again I have highlighted the RTPEvent.

Basically we are capturing UDP packets with length not equal to 24, that are RTPEVENT’s (meaning DTMF) and have RTP marker bit set to 1 - meaning the first packet in the DTMF digit - which has all the information we need.

Event Duration 800 is signified by hex 03 20 raw data - the data 00 00 after that is basically Trailer for Ethernet II as you can see above. Similarly in the bad snapshot above the duration zero should have been represented by 00 00.
However, in that place we see some data after that –

We found that we could isolate all the good packets from bad packets by filtering for the right length. In this case udp length 24. So Why length 24?

A DTMF packet is – UDP header (8 octets) + RTP header (12 octets) + RTP Payload (RTPEVENT=DTMF) (4 octets) = 24 octets.

Here is what we should have for DTMF payload (RTP Payload) RFC 4733 –

Here is what we should have for RTP header (assuming no CSRC) RFC 1189 –

Here is what we should have for UDP header RFC 768 –

So the hex data beyond 24 octets is something that should not have been received by Lync in a DTMF digit. This is what causes Lync to ignore these digits.

Resolution

To resolve this issue upgrade to the supported CUCM version 8.5.1.12900-7.

LYNC Server 2010 and Communication Server 2007 R2 does not support NET VX 1800 when set up in Hybrid Mode - DTMF issues

Jigar Dani — Thu, 07 Apr 2011 23:53:28 GMT

Over the past year we have seen multiple issues with NET VX1800 deployed in Hybrid Gateway mode - Gateway + Communication Server 2007 R2 (CS)\LYNC Server 2010 (LS) Mediation Server (MS). Here I am going to talk about one such issue.

Symptoms - Unable to make calls outbound from CS\LS to PSTN using NET VX1800, Unable to parse through a DTMF driven IVR on calls through VX 1800.

Microsoft's Unified Communication Open Interoperability Program lists all the gateways supported. VX1800 is supported only in Basic Mode and not Hybrid Mode.

In the cases we have worked on so far with NET working in Hybrid Mode, we have seen that VX1800 has challenges recognizing DTMF when transmitted using RFC 2833. A single DTMF event is sent by CS\LS is at times read as multiple events depending on state of the call.

These issues first surfaced in Q1 2010. NET is aware of this issue and currently working on fixing this. For more information please contact NET.

If you need further technical details on this issue please contact Microsoft Support.

Solution -

In the cases we have worked on, our customers ended up introducing the MS to sit between CS\LS Front End servers and run the NET VX 1800 in Basic Mode.

Exchange 2010 co-existence Exchange 2007 Unified Messaging connected to Cisco Call Manager causes users to be prompted twice when calling in subscriber access\outlook voice access

Jigar Dani — Thu, 07 Apr 2011 23:22:00 GMT

Over the past few months we have seen an increase in number of interop cases with Exchange 2010 Unified Messaging (UM) and Cisco Call Manager 7.0 (Configuration Notes) and 8.0 (Configuration Notes) (CCM). In this article I am going to talk particularly about one case that has seen the most hits which has also been pointed out in the configuration notes.

During migration from UM 2007 to UM 2010 the UM design requires that CCM is pointed directly to UM 2010 for Outlook Voice Access\Subscriber Access. In this co-existence scenario, UM 2010 will transfer the call to UM 2007 if the mailbox happens to reside on 2007 and users are able to access their mailboxes. There are two ways in which the transfer could be achieved - Re-Direct or Transfer. In SIP world this translates to the following messages -

Re-Direct - 302 Moved Temporarily (using the Contact header)

Transfer - REFER (using the Refer-To and Referred-By header)

Whether UM 2010 uses Re-Direct or a Transfer to move the call to UM 2007 depends if or not it was able to resolve who the user was. If it is able to resolve the user and find the mailbox UM 2010 does a Re-Direct, if it is not able to resolve the user to a mailbox it does a Transfer.

So what is the problem?

CCM does not completely work well with RFC 3515. This RFC defines the call transfer mechanism using REFER.

User behavior would be like this -

User calls in to UM to access the mailbox,

UM 2010 tries to resolve the user using the From header in SIP message,

If UM 2010 is not able to resolve the user to a mailbox, the caller will be prompted to authenticate using extension and pin information.

If UM 2010 sees this mailbox is actually residing on UM 2007 it has to transfer the call to UM 2007. This is done using the REFER method and the extension information is included in the Referred-By header.

CCM, when configured properly, uses the Refer-To header information from the SIP REFER message to connect to UM 2007, however doesn’t include the Referred-By header which includes the extension information.

UM 2007 answers the call and does not see the Referred-By header, thus prompts the caller again to authenticate

Solution -

The technical challenge is that if UM 2010 is not able to resolve the user, it has to answer the call and then prompt the caller to authenticate. A SIP call that is answered cannot be redirected and has to be transferred using REFER.

To understand the Caller ID resolution mechanism in UM 2010 go to this article. The article also points out a number of ways you can resolve caller id. Any of the mechanisms listed in the article - Pattern Matching, AD changes, Heuristics could be used to avoid this issue.

Exchange 2010 SP1 Unified Messaging cannot transfer call to an extension

Jigar Dani — Fri, 04 Feb 2011 22:41:00 GMT

There are a number of reasons that may lead to calls not being transferred correctly to extensions homed on gateways - Cisco Call Manager, Audio Codes, OCS\LYNC amongst others, one of them blogged by me here. In this blog I am going to talk about another issue we have seen with our customers.

ISSUE

Unable to transfer calls to an extension using Auto Attendant with key mappings set.

CAUSE

Microsoft Exchange Unified Messaging (UM) 2010 SP1 currently relies on the From header from the initial INVITE (call request) into UM to form the URI to transfer the call to. It takes the extension from the Auto Attendant key mapping and takes the server part from the server part of the From header (which should indicate the server address of the gateway).

Example -

If the From header is From: "Test" <sip:4832883828@174.281.138.34>;tag=4839-293-2991-392943

A transfer to a key mapped extension 5748 may lead to a Refer-To header like Refer-To= <sip:5748@174.281.138.34>

Now, sometimes the gateways would include the right server address in the From header.

From: "Test" <sip:4832883828@anonymous.invalid>;tag=4839-293-2991-392943

This may result in Refer-to header -

Refer-To: <sip:5748@anonymous.invalid>

Which as you would imagine the gateway may not be able to handle. This will result in a prompt like "Sorry unable to transfer the call".

RESOLUTION

The gateway should be configured to send the INVITE with the From header having the server address of the gateway.
Microsoft will, in a future release try to use the server address from the Contact header in the INVITE instead of the From header, this should resolve most of the issues we have seen so far. Stay tuned to this blog to find out when this change is made.

Exchange 2010 SP1 Unified Messaging integration with Nortel CS1000

Jigar Dani — Thu, 03 Feb 2011 23:57:00 GMT

With the release of Exchange 2010 SP1 we have seen some issues integrating with Nortel CS 1000. Nortel CS 1000 is currently not part of the Telephony Advisory Program and not certified for Exchange 2010 & 2010 SP1. It is currently supported only when it connects to Exchange UM through one of the supported IP gateways - Dialogic DMG, Audiocodes, etc.

BACKGROUND

To connect a call to UM the gateway first connects to it the UMService running on port 5060\5061 and then the call is immediately transferred to the UMWorkerProcess on port 5065\5067. You can know more about the process here. This transfer is basically a redirect done by SIP message 302 Moved Temporarily containing a Contact header that tells the gateway where to connect to the UMWorkerProcess.

ISSUE

You may experience an issue where once a user connected to Nortel CS1000 calls into Exchange UM he hears dead air. The call is never connected to any of the UM features - voicemail\ subscriber access\ auto attendant. We have seen this issue with version 5.x and 6.x.

CAUSE

After investigation we have found that at times Nortel CS 1000 is not able to understand this Contact header needed to route the request to UMWorkerProcess. This occurs when the Contact header contains an optional maddr parameter (multicast address) containing the domain that needs to be communicated to continue this call.

Contact: <sip:3213;phone-context=ping@microsoft.com:5065;maddr=microsoft.loc>

Now, this maddr parameter is added to the Contact header by UM only if the incoming INVITE into UM contains the optional maddr parameter in the Request line -

Request-Line: INVITE sip:3213;phone-context=ping@microsoft.com;maddr=10.10.10.1;transport=tcp SIP/2.0

If you view this in Network Monitor\ Wireshark you will see that the second INVITE into the UM worker process is never received at the UM server -

Instead of -

The result will be dead air/unable to connect the call.

SOLUTION

Audio Codes, Dialogic among other supported gateways give the ability to either strip the optional maddr parameter or can route the traffic for that domain to a particular gateway. They have also been certified to work with Exchange UM 2010\2010 SP1. It is advised that a supported gateway be introduced to connect with Exchange UM 2010 SP1 till we are able to certify Nortel CS 1000.

Exchange 2010 SP1 Unified Messaging inter-op with OCS - Location Profiles

Jigar Dani — Mon, 18 Oct 2010 17:50:00 GMT

With Exchange Unified Messaging 2010 SP1 the call transfer to extension has improved to facilitate OCS to make the decision on how to route the call.

Prior to UM 2010 SP1, UM will send the actual location profile to be used to make the call to the user by OCS. This was indicated in the Refer-To header in the REFER (message used to transfer the call to the extension).
phone-context=<location profile>

This information came from the UM Dial plan, whose name should be exactly same as the OCS Location profile.

In UM 2010 SP1, this location profile information is no longer relayed. In the Refer-To header, you will now see -
phone-context=user-default

Which basically means use the default location profile. With this UM empowers OCS to select the best location profile it wants to use.

One caveat here is that if OCS has not already set the default location profile this will need to be done. If the location profile is not selected for this OCS Pool you may see the symptoms like -
1. Exchange UM 2010 SP1 Dial by extension does not work for OCS dial plan.
2. Key mappings that map to extensions like Operator do not work.

However, when users use the AA - Directory look up, call complete fine. This will work because with directory look up the call is actually made using the SIP URI of the user and not the extension. The Refer-To header in this case will contain the SIP URI.

If you are running into this kind of problem, using the OCS logger, get the TranslationApplication component logs with all flags while reproducing the issue. You should see -
TL_INFO(TF_COMPONENT) [3] <TimeStamp> (TranslationApplication,PhoneNumberTranslator.TranslateNumber:phonenumbertranslator.cs(250))( 0000000001794AAA )calledNumber='126'
TL_WARN(TF_COMPONENT) [3] <TimeStamp> (TranslationApplication,PhoneNumberTranslator.GetProfileByName:phonenumbertranslator.cs(628))( 0000000001794AAA )Phone-context is user-default but no user-level or pool-level default defined.
TL_WARN(TF_COMPONENT) [3] <TimeStamp> (TranslationApplication,PhoneNumberTranslator.TranslateNumber:phonenumbertranslator.cs(275))( 0000000001794AAA )Profile='user-default' not found (or invalid)

You can set this default location profile on the Pool Level - Front End Properties. Once the right location profile is selected, call should route immediately.

Pre - Requisites for OCS integration with Exchange UM 2010 SP1 -
1. Works only with OCS R2,
2. OCS R2 should be at least at CU5.
3. Set up a default location profile for the pool

Remote Call Control in Office Communications Server R2 uses UPDATE method to maintain persistent sessions between Communicator 2007 R2 and CSTA Gateway

Jigar Dani — Mon, 16 Aug 2010 23:06:00 GMT

For most of the administrators who have ran into issues with RCC my colleagues blog has been a great help. With the evolution of OCS and the SIP standard, there have been two changes in the behavior since the time Rajesh put that blog out. Hopefully this blog with help you look out for issues arising due to those changes.

As noted in Rajesh's blog entry, the Microsoft Communicator 2007 R2 (MOC) establishes a persistent connection between itself and the CSTA gateway. This connection is used to send notification\events\commands bi-directionally. MOC is responsible to keep this connection alive. The earlier blog states - "To ensure that the SIP/CSTA Gateway is running and the signaling link is available, Office Communicator refreshes the INVITE dialog every 10 minutes by sending a Re-INVITE with the RequestSystemStatus command." However, MOC now uses an updated mechanism to keep up this connection.

The changes are -

Instead of using re-INVITE to keep the connection alive, MOC now uses the UPDATE method by default.
Instead of refreshing the session every 10 minutes, MOC now refreshes it every 25 minutes.

After 25 minutes MOC tries to refresh the session. In case the CSTA gateway is not aware of UPDATE method, it may just ignore this request and the session is torn down as the session timer is elapsed. If you enable logging on the MOC and look at the log file in Snooper, you will see that after 25 minutes of inactivity MOC will send a refresh using UDPATE method. However, never get a response from the CSTA gateway.

This will lead to RCC failures where the client is no longer receiving a toast for incoming calls or can no longer originate calls from the desk phone using the MOC, say after 25 minutes of inactivity.

After code level investigation here is the current logical algorithm the MOC uses to decide whether to use UPDATE or reINVITE to refresh the session -

So if the CSTA gateway vendor does not support UPDATE method-

Include an Allow header in the 200OK response to the INVITE,

Do not include UPDATE as supported methods.