Cisco AP can’t join the WLC controller… DTLS handshake failure.

The problem, network components and topology

I recently faced the issue with AP join to vWLC. Cisco 2700 AP could not join to newly installed Cisco vWLC controller. A colleague asked me to take a look and explained the topology. The vWLC was located in Grandmetric DC Testing Labs whereas Cisco CAP-2700 was located at Grandmetric HQ Office. Between the office and DC there is an IPSec VPN tunnel created on Cisco ASA (5506x and 5520, the older one) in EzVPN  hardware client mode. Normally after connecting AP to PoE powered Catalyst, the AP receives IP address from DHCP with option 43 that specifies the controller IP address. In turn, AP is able to establish Capwap tunnel to controller, download updated software and specific configuration. The problem was that AP was not able to join the vWLC changing the address in cycles. This is normal behaviour when option 43 is not properly set in DHCP or there is no connectivity between AP and controller. This time it was not the case.

At first glance this could be certificate or asymmetric routing issue

I sat behind the console, checked the option 43 decimal to hex translation. It was correct. Then quickly logged to gui of vWLC and pinged the AP. Was not successful. That was the first hint and by the way I fixed another issue not related to AP join but rather to NAT for return traffic. I quickly fixed this one and then took a look at log messages on the controller. I saw something like this DTLS negotiation failure (so nat did not do the fix):

*osapiBsnTimer: May 25 06:54:29.631: %DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:3047 Failed to complete DTLS handshake with peer 172.16.1.16

I remember similar problem from the past where asymmetric routing caused the issue and capwap was not able to establish. However, I have verified once again the possibility of asymmetric routing and ensured that everything was ok with routing. Then started looking for some certificate problem on the AP and AP policy on the vWLC looking at debug on AP console level. The problem was strange because i have verified that certificates were ok in terms of X.509 framework and dates validation and I saw requests on office ASA and also connections were established:

UDP Outside 10.253.51.100:5247 LAB-DEV 172.16.1.16:19576, idle 0:00:02, bytes 2531685, flags -
UDP Outside 10.253.51.100:5246 LAB-DEV 172.16.1.16:19576, idle 0:00:00, bytes 5306037, flags -

similar connections were present on DC ASA:

UDP OUTSIDE 172.16.1.16:19576 LAB-Domain 10.253.51.100:5247, idle 0:00:14, bytes 2534737, flags -
UDP OUTSIDE 172.16.1.16:19576 LAB-Domain 10.253.51.100:5246, idle 0:00:05, bytes 3928657, flags -

However, the problem still existed. I have troubleshooted the returning traffic on ASA office and this pointed me to assumption that the return traffic comes to the ASA office but not to the AP. AP then has this DTLS handshake incomplete and began process of finding the controller one more time. I saw that there is no strange log on ASA and even packet tracer was ok. So I deducted that ASA has internal problem with properly handling the DTLS session. I also found that there was some bug on asa941-lfbff-k8.SPA ASA-OS related to DTLS handshake. I decided to upgrade the ASA-OS to version Cisco Adaptive Security Appliance Software Version 9.8(2) <the one i had on disk> and right after the reload and bootup the i saw 2 access points connected.