SymptomsAll of the standard Lync/Exchange integration features were working correctly - including calendar-based presence updates, Exchange OWA Instant Messaging, and even OAuth appeared to be working because we could schedule an Online Meeting from within OWA as well. However, we noticed that contacts were not being migrated into the UCS. We tried all of the troubleshooting steps suggested by Saleesh in his post here:
- Running the Test-CsExchStorageConnectivity command resulted in a 501 error (truncated output):
Test-CsExStorageConnectivity -SipUri sip:firstname.lastname@example.org -Verbose
VERBOSE: Successfully opened a connection to storage service at localhost using
VERBOSE: Create message.
VERBOSE: Execute Exchange Storage Command.
Test-CsExStorageConnectivity : ExCreateItem exchange operation failed,
code=ErrorUnhandledException, reason=Wrapped callback failed --->
System.InvalidOperationException: Client found response content type of
'text/html', but expected 'text/xml'.
The request failed with the error message:
<html><head><title>501 Invalid Request</title></head><body>Invalid Request:
entMessage message, WebResponse response, Stream responseStream, Boolean
- Running the Test-CsUnifiedContactStore cmdlet would pass/fail intermittently. (This was very frustrating!)
- Tracing the error on the Lync side, through CLS or Debug Tools, confirmed the 501 "Invalid Request" that we were seeing in the Test-CsExchStorageConnectivity command.
- Again, reviewing the System Message File on the Kemp confirmed the 501, but gave us an additional hint as it was labeling of traffic as "bad requests" coming from the Lync Front Ends (output below):
Oct 7 09:50:57 KEMP kernel: L7: badrequest-client_read [172.31.2.40:60898->172.16.16.86:443] (-501): <?xml ? , 0 [hlen 3118, nhdrs 9]
Oct 7 09:54:19 KEMP kernel: L7: badrequest-client_read [172.31.2.41:62471->172.16.16.86:443] (-501): <?xml ? , 0 [hlen 3082, nhdrs 9]
Oct 7 09:54:44 KEMP kernel: device eth0 entered promiscuous mode
Oct 7 09:56:46 KEMP kernel: device eth0 left promiscuous mode
Oct 7 09:57:23 KEMP kernel: device eth0 entered promiscuous mode
Oct 7 10:02:13 KEMP kernel: L7: badrequest-client_read [172.31.2.40:61198->172.16.16.86:443] (-501): <?xml ? , 0 [hlen 3074, nhdrs 9]
Oct 7 10:02:40 KEMP kernel: L7: Connection timed out (22.214.171.124:5663->192.168.151.86:443->172.16.16.80:443) (connected)
Oct 7 10:04:31 KEMP kernel: L7: Connection timed out (172.31.2.40:61227->172.16.16.86:443-><nodest>) (waiting for initial client request)
Oct 7 10:04:31 KEMP kernel: L7: Connection timed out (172.31.2.40:61229->172.16.16.86:443-><nodest>) (waiting for initial client request)
Oct 7 10:04:31 KEMP kernel: L7: Connection timed out (172.31.2.40:61230->172.16.16.86:443-><nodest>) (waiting for initial client request)
Oct 7 10:04:31 KEMP kernel: L7: Connection timed out (172.31.2.40:61228->172.16.16.86:443-><nodest>) (waiting for initial client request)
After a bit of troubleshooting, we narrowed the issue down to the Kemp Load Balancer. We did this by changing the DNS records for EWS and Autodiscover to point directly to one of the Exchange 2013 CAS Proxy servers (we were running split roles on Exchange 2013). Upon doing so, the UCS began working consistently. We then placed a call to Kemp support. Based on their suggestion, we updated the L7 Configuration 100-Continue Handling.
If your Kemp is running a relatively recent version of the software, the 100-Continue Handling will be set to "Always Expect-100". And though it isn't clear to me as to why this is the case, this prevents UCS from working. Per support's suggestion, we updated this setting to "Ignore Continue-100". Within a couple minutes, virtually everyone's contacts started migrating to the UCS!
Kemp L7 Configuration for 100-Continue Handling
If anyone can provide some insight as to why this would prevent the UCS from working properly, please let me know. Otherwise, I hope this saves someone the headaches I went through on this issue!