Blog

IP and Mobile Trends and Education

 

Cisco AP can’t join the WLC controller… DTLS handshake failure.

Author:


30.05.2018

The problem, network components and topology

I recently faced the issue with AP join to vWLC. Cisco 2700 AP could not join to newly installed Cisco vWLC controller. Coleague asked me to take a look and explained the topology. The vWLC was located in Grandmetric DC Testing Labs whereas Cisco CAP-2700 was located at Grandmetric HQ Office. Between the office and DC there is an IPSec VPN tunnel created on Cisco ASA (5506x and 5520, the older one) in EzVPN  hardware client mode. Normally after connecting AP to PoE powered Catalyst, the AP receives IP address from DHCP with option 43 that specifies the controller IP address. In turn, AP is able to establish Capwap tunnel to controller, download updated software and specfic configuration. The problem was that AP was not able to join the vWLC changing the address in cycles. This is normall behavior when option 43 is not properly set in DHCP or there is no connectivity between AP and controller. This time it was not the case.

At first glance this could be certificate or asymetric routing issue

I sat behind the console, checked the option 43 decimal to hex translation. It was correct. Then quickly logged to gui of vWLC and pinged the AP. Was not successful. That was the first hint and by the way I fixed another issue not related to AP join but rather to NAT for reutrn traffic. I quickly fixed this one and then took a look at log messages on the controller. I saw something like this DTLS negotiation failure (so nat did not do the fix):

*osapiBsnTimer: May 25 06:54:29.631: %DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:3047 Failed to complete DTLS handshake with peer 172.16.1.16

I remember similar problem from the past where asymetric routing caused the issue and capwap was not able to establish. Hovewer I have verified once again the possibility of asymetric routing and ensured that everything was ok with routing. Then started looking for some certificate problem on the AP and AP policy on the vWLC looking at debug on AP console level. The problem was strange because i have verified that certificates were ok in terms of X.509 framework and dates validation and I saw requests on office ASA and also connections were established:

UDP Outside 10.253.51.100:5247 LAB-DEV 172.16.1.16:19576, idle 0:00:02, bytes 2531685, flags -
UDP Outside 10.253.51.100:5246 LAB-DEV 172.16.1.16:19576, idle 0:00:00, bytes 5306037, flags -

 

similar connections were present on DC ASA:

UDP OUTSIDE 172.16.1.16:19576 LAB-Domain 10.253.51.100:5247, idle 0:00:14, bytes 2534737, flags -
UDP OUTSIDE 172.16.1.16:19576 LAB-Domain 10.253.51.100:5246, idle 0:00:05, bytes 3928657, flags -

 

However, the problem still existed. I have troubleshooted the returning traffic on ASA office and this pointed me to assumption that the return traffic comes to the ASA office but not to the AP. AP then has this DTLS handshake incomplete and began process of finding the controller one more time. I saw that there is no strange log on ASA and even packet tracer was ok. So I deducted that ASA has internal problem with properly handling the DTLS session. I also found that there was some bug on asa941-lfbff-k8.SPA ASA-OS related to DTLS handshake. I decided to upgrade the ASA-OS to version Cisco Adaptive Security Appliance Software Version 9.8(2) <the one i had on disk> and right after the reload and bootup the i saw 2 access points connected.

 

DTLS_issue_via_asa

Author

Marcin Bialy

Marcin Biały is Network and Security Architect with over 12 years of experience, with Service Provider and Enterprise networking background. He used to work for large service providers, global vendors and integration services companies as Network Architect, Leading Architect and Techincal Solution Manager positions. He designed, implemented and supported dozens large scale projects and infrastructure migrations, solved hundreds of tickets and spent hours with CLI and GUI of many flavors. Marcin is also holding industry recognizable certificates such as CCNP, CCNA, CCSI #35269, FCNSP #7207, FCNSA and more.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

code


 

Newsletter