Monday, October 28, 2013

Kemp Load Balancer and Lync Unified Contact Store Integration Note

During the deployment of Lync 2013 for a customer who was integrating with Exchange 2013, we ran into somewhat of a strange issue with the Unified Contact Store (UCS).  Even though EWS access was working, and OAuth appeared to be working, users' contacts were not being migrated into the UCS.  After a bit of troubleshooting on our own, a conversation with Kemp support pointed us in the right direction.

Symptoms

All of the standard Lync/Exchange integration features were working correctly - including calendar-based presence updates, Exchange OWA Instant Messaging, and even OAuth appeared to be working because we could schedule an Online Meeting from within OWA as well.  However, we noticed that contacts were not being migrated into the UCS.  We tried all of the troubleshooting steps suggested by Saleesh in his post here:

http://blogs.technet.com/b/saleesh_nv/archive/2013/03/14/lync-2013-unified-contact-store-ucs-troubleshooting-tips.aspx
  • Running the Test-CsExchStorageConnectivity command resulted in a 501 error (truncated output):
Test-CsExStorageConnectivity -SipUri sip:psharp@domain.com -Verbose
VERBOSE: Successfully opened a connection to storage service at localhost using
 binding: NetNamedPipe.
VERBOSE: Create message.
VERBOSE: Execute Exchange Storage Command.
Test-CsExStorageConnectivity : ExCreateItem exchange operation failed,
code=574, reason=StoreContext{traceId=[18446744071743115001],
activityId=[777ecae0-4db4-4418-8ccd-b06c7f31a401]}StoreException:
code=ErrorUnhandledException, reason=Wrapped callback failed --->
System.InvalidOperationException: Client found response content type of
'text/html', but expected 'text/xml'.
The request failed with the error message:
--
<html><head><title>501 Invalid Request</title></head><body>Invalid Request:
??</body>
--.
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapCli
entMessage message, WebResponse response, Stream responseStream, Boolean
asyncCall)


  • Running the Test-CsUnifiedContactStore cmdlet would pass/fail intermittently. (This was very frustrating!)
  • Tracing the error on the Lync side, through CLS or Debug Tools, confirmed the 501 "Invalid Request" that we were seeing in the Test-CsExchStorageConnectivity command.
  • Again, reviewing the System Message File on the Kemp confirmed the 501, but gave us an additional hint as it was labeling of traffic as "bad requests" coming from the Lync Front Ends (output below):

Oct  7 09:50:57 KEMP kernel: L7: badrequest-client_read [172.31.2.40:60898->172.16.16.86:443] (-501): <?xml ? , 0 [hlen 3118, nhdrs 9]
Oct  7 09:54:19 KEMP kernel: L7: badrequest-client_read [172.31.2.41:62471->172.16.16.86:443] (-501): <?xml ? , 0 [hlen 3082, nhdrs 9]
Oct  7 09:54:44 KEMP kernel: device eth0 entered promiscuous mode
Oct  7 09:56:46 KEMP kernel: device eth0 left promiscuous mode
Oct  7 09:57:23 KEMP kernel: device eth0 entered promiscuous mode
Oct  7 10:02:13 KEMP kernel: L7: badrequest-client_read [172.31.2.40:61198->172.16.16.86:443] (-501): <?xml ? , 0 [hlen 3074, nhdrs 9]
Oct  7 10:02:40 KEMP kernel: L7: Connection timed out (70.198.68.197:5663->192.168.151.86:443->172.16.16.80:443)[0] (connected)
Oct  7 10:04:31 KEMP kernel: L7: Connection timed out (172.31.2.40:61227->172.16.16.86:443-><nodest>)[0] (waiting for initial client request)
Oct  7 10:04:31 KEMP kernel: L7: Connection timed out (172.31.2.40:61229->172.16.16.86:443-><nodest>)[0] (waiting for initial client request)
Oct  7 10:04:31 KEMP kernel: L7: Connection timed out (172.31.2.40:61230->172.16.16.86:443-><nodest>)[0] (waiting for initial client request)
Oct  7 10:04:31 KEMP kernel: L7: Connection timed out (172.31.2.40:61228->172.16.16.86:443-><nodest>)[0] (waiting for initial client request)

Resolution

After a bit of troubleshooting, we narrowed the issue down to the Kemp Load Balancer.  We did this by changing the DNS records for EWS and Autodiscover to point directly to one of the Exchange 2013 CAS Proxy servers (we were running split roles on Exchange 2013).  Upon doing so, the UCS began working consistently.  We then placed a call to Kemp support.  Based on their suggestion, we updated the L7 Configuration 100-Continue Handling.

If your Kemp is running a relatively recent version of the software, the 100-Continue Handling will be set to "Always Expect-100".  And though it isn't clear to me as to why this is the case, this prevents UCS from working.  Per support's suggestion, we updated this setting to "Ignore Continue-100".  Within a couple minutes, virtually everyone's contacts started migrating to the UCS!

Kemp L7 Configuration for 100-Continue Handling



If anyone can provide some insight as to why this would prevent the UCS from working properly, please let me know.  Otherwise, I hope this saves someone the headaches I went through on this issue!

5 comments:

  1. Hello Phil,

    Great post. I think it is important for me to add some context. The issue at hand is data passing through LM that isn't RFC conformant for "100 continue" handling. The testing the cmdlet is intermittent and you get error sometimes like you mentioned. This does not affect user experience from what I have seen in my deployments.

    However, if necessary, you can apply the change mentioned here which than forces KEMP LM to ignore RFC conformance and continue processing messages.

    Thanks,
    Bhargav

    ReplyDelete
    Replies
    1. Bhargav,

      Thanks for the insight. Very helpful.

      -Phil

      Delete
  2. Any idea why Ignore Continue-100 is not a selectable option in KEMP version 7.1-26-15?
    Is there another option that can be selected in order to fix this problem?

    Choices are RFC-2616 Compliant, Require 100-Continue, RFC-7231 Compliant.

    Thank you.

    ReplyDelete
  3. "Ignore Continue-100" option has been changed to "RFC-7231 Compliant" option as of version 7.1

    ReplyDelete
    Replies
    1. Mark,

      Thanks for following up. That was going to be my suggestion based on the changes Kemp has made. There was some contention around the interpretations of the previous RFC between Microsoft and Kemp. This was also impacting Exchange in a Hybrid/migration scenario, hence the new options available in 7.1.

      Glad you got it sorted out!

      Delete