Troubleshooting FMC and Cisco Firepower Sensor communication

In this post we are going to focus on the scripts included in FTD and FMC operating systems that help to troubleshoot connections between FTD sensors and Cisco Firepower Management Center. As they are run from the “expert mode” (super user), it is better that you have a deep understanding of any potential impact on the production environment.

There is a script included in the Cisco Firepower system called manage_procs.pl (use it wisely). It can be run from the FTD expert mode or the FMC. It allows you to restart the communication channel between both devices. Be careful, if you run it from the FMC and you have hundreds of sensors it will reestablish all communication channels to all of your sensors at once. If you run it from the FTD then only the particular sensor – FMC communication will be affected.


> expert
admin@FTDv:~$ sudo su
Password:
root@FTDv:/home/admin# manage_procs.pl
****************  Configuration Utility  **************
1   Reconfigure Correlator
2   Reconfigure and flush Correlator
3   Restart Comm. channel
4   Update routes
5   Reset all routes
6   Validate Network
0   Exit
**************************************************************
Enter choice:

I am using 3th, 4th and 5th option. It can take few seconds to proceed. This scripts are nice to be used when the FMC and FTD have communication problems like heartbeats are not received, policy deployment is failing or events are not received. These options reestablish the secure channels between both peers, verifying the certificates and creating new config file on the backend.

If you still have problems then you can see all the debugging messages in a separate SSH session to the sensor.

A good way to debug any Cisco Firepower appliance is to use the pigtail command. It gives real time outputs from a bunch of log files. So lets execute manage_procs.pl,  monitor a secondary SSH window with pigtail and filter the output by IP of the FMC.  Keep in mind that you may use the pigtail command during the registration process and monitor where the registration is failing.

root@FTDv:/home/admin# pigtail | grep 192.168.0.200
MSGS: 04-09 07:48:46 FTDv SF-IMS[9200]: [9200] sfmgr:sfmanager [INFO] MARK TO FREE peer 192.168.0.200
MSGS: 04-09 07:48:46 FTDv SF-IMS[9200]: [13244] sfmgr:sfmanager [INFO] WRITE_THREAD:Terminated sftunnel write thread for peer 192.168.0.200
MSGS: 04-09 07:48:48 FTDv SF-IMS[9200]: [13243] sfmgr:sfmanager [INFO] Stop child thread for peer 192.168.0.200
MSGS: 04-09 07:48:48 FTDv SF-IMS[9200]: [13243] sfmgr:sfmanager [INFO] Exiting child thread for peer 192.168.0.200
MSGS: 04-09 07:48:48 FTDv SF-IMS[9200]: [13243] sfmgr:sfmanager [INFO] free_peer 192.168.0.200.MSGS: 04-09 07:48:50 FTDv SF-IMS[9201]: [13428] sfmbservice:sfmb_service [INFO] TERM:Peer 192.168.0.200 removed
MSGS: 04-09 07:48:57 FTDv SF-IMS[5575]: [13337] SFDataCorrelator:EventStreamHandler [INFO] Reset: Closing estreamer connection to:192.168.0.200
SERR: 04-09 07:48:50 2018-04-09 07:48:58 sfmbservice[9201]:FTDvSF-IMS[9201]: [13428] sfmbservice:sfmb_service [INFO] TERM:Peer 192.168.0.200 removed
MSGS: 04-09 07:48:58 FTDv SF-IMS[14543]: [14546] sfmbservice:sfmb_service [INFO] Start getting MB messages for 192.168.0.200
MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14541] sftunneld:sf_peers [INFO] Using a 20 entry queue for 192.168.0.200 - 8104
MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14541] sftunneld:sf_peers [INFO] Using a 20 entry queue for 192.168.0.200 - 8121
MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14541] sftunneld:stream_file [INFO] Stream CTX initialized for 192.168.0.200
MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14551] sftunneld:sf_peers [INFO] Peer 192.168.0.200 needs a single connection
MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14551] sftunneld:sf_connections [INFO] Start connection to : 192.168.0.200 (wait 0 seconds is up)
MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_peers [INFO] Peer 192.168.0.200 needs a single connection
MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_ssl[INFO] Connect to 192.168.0.200 on port 8305 - br1
MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_ssl[INFO] Initiate IPv4 connection to 192.168.0.200 (via br1)
MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_ssl[INFO] Initiating IPv4 connection to 192.168.0.200:8305/tcp
MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_ssl[INFO] Wait to connect to 8305 (IPv6): 192.168.0.200
MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_ssl[INFO] Connect to 192.168.0.200 failed on port 8305 socket 11 (Connection refused)MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_ssl[INFO] No IPv4 connection to 192.168.0.200
MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_ssl[WARN] Unable to connect to peer '192.168.0.200'
MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_ssl[INFO] reconnect to peer '192.168.0.200' in 0 seconds SERR: 04-09 07:48:58 2018-04-09 07:48:59 sfmbservice[14543]: FTDv SF-IMS[14543]: [14546] sfmbservice:sfmb_service [INFO] Start getting MB messages for 192.168.0.200
MSGS: 04-09 07:49:00 FTDv SF-IMS[14541]: [14551] sftunneld:sf_peers [INFO] Peer 192.168.0.200 needs a single connection

Another great tool inherited by Sourcefire is sftunnel_status.pl. It is a script that shows all details related to the communication between the sensor and the FMC. The most important are the outputs showing the status of the Channel A and Channel B. These are the management and the eventing channels. In more complex Cisco Firepower designs these are two separate physical connections which enhance the policy push time and the logging features.

root@FTDv:/home/admin# sftunnel_status.pl
SFTUNNEL Start Time: Mon Apr  9 07:48:59 2018
Both IPv4 and IPv6 connectivity is supported
Broadcast count = 0
Reserved SSL connections: 0
Management Interfaces: 1
br1 (control events) 192.168.0.201,
*************************RUN STATUS****192.168.0.200*************
Key File    = /var/sf/peers/e5845934-1cb1-11e8-9ca8-c3055116ac45/sftunnel-key.pem
Cert File   = /var/sf/peers/e5845934-1cb1-11e8-9ca8-c3055116ac45/sftunnel-cert.pem
CA Cert     = /var/sf/peers/e5845934-1cb1-11e8-9ca8-c3055116ac45/cacert.pem
Cipher used = AES256-GCM-SHA384 (strength:256 bits)
ChannelA Connected: Yes, Interface br1
Cipher used = AES256-GCM-SHA384 (strength:256 bits)
ChannelB Connected: Yes, Interface br1
Registration: Completed.
IPv4 Connection to peer '192.168.0.200' Start Time: Mon Apr  9 07:49:01 2018

PEER INFO:
sw_version 6.2.2.2
sw_build 109
Management Interfaces: 1
eth0 (control events) 192.168.0.200,
Peer channel Channel-A is valid  type (CONTROL), using 'br1', connected to '192.168.0.200' via '192.168.0.201'
Peer channel Channel-B is valid  type (EVENT), using 'br1', connected to '192.168.0.200' via '192.168.0.201'

TOTAL TRANSMITTED MESSAGES <16> for IP(NTP) service
RECEIVED MESSAGES <8> for IP(NTP) service
SEND MESSAGES <8> for IP(NTP) service
HALT REQUEST SEND COUNTER <0> for IP(NTP) service
STORED MESSAGES for IP(NTP) service (service 0/peer 0)
STATE <Process messages> for IP(NTP) service
REQUESTED FOR REMOTE <Process messages> for IP(NTP) service
REQUESTED FROM REMOTE <Process messages> for IP(NTP) service

TOTAL TRANSMITTED MESSAGES <4> for Health Events service
RECEIVED MESSAGES <2> for Health Events service
SEND MESSAGES <2> for Health Events service
HALT REQUEST SEND COUNTER <0> for Health Events service
STORED MESSAGES for Health service (service 0/peer 0)
STATE <Process messages> for Health Events service
REQUESTED FOR REMOTE <Process messages> for Health Events service
REQUESTED FROM REMOTE <Process messages> for Health Events service

TOTAL TRANSMITTED MESSAGES <3> for Identity service
RECEIVED MESSAGES <2> for Identity service
SEND MESSAGES <1> for Identity service
HALT REQUEST SEND COUNTER <0> for Identity service
STORED MESSAGES for Identity service (service 0/peer 0)
STATE <Process messages> for Identity service
REQUESTED FOR REMOTE <Process messages> for Identity service
REQUESTED FROM REMOTE <Process messages> for Identity service

TOTAL TRANSMITTED MESSAGES <44> for RPC service
RECEIVED MESSAGES <22> for RPC service
SEND MESSAGES <22> for RPC service
HALT REQUEST SEND COUNTER <0> for RPC service
STORED MESSAGES for RPC service (service 0/peer 0)
STATE <Process messages> for RPC service
REQUESTED FOR REMOTE <Process messages> for RPC service
REQUESTED FROM REMOTE <Process messages> for RPC service
TOTAL TRANSMITTED MESSAGES <14> for IDS Events service
RECEIVED MESSAGES <7> for service IDS Events service
SEND MESSAGES <7> for IDS Events service
HALT REQUEST SEND COUNTER <0> for IDS Events service
STORED MESSAGES for IDS Events service (service 0/peer 0)
STATE <Process messages> for IDS Events service
REQUESTED FOR REMOTE <Process messages> for IDS Events service
REQUESTED FROM REMOTE <Process messages> for IDS Events service

TOTAL TRANSMITTED MESSAGES <23> for EStreamer Events service
RECEIVED MESSAGES <11> for service EStreamer Events service
SEND MESSAGES <12> for EStreamer Events service
HALT REQUEST SEND COUNTER <0> for EStreamer Events service
STORED MESSAGES for EStreamer Events service (service 0/peer 0)
STATE <Process messages> for EStreamer Events service
REQUESTED FOR REMOTE <Process messages> for EStreamer Events service
REQUESTED FROM REMOTE <Process messages> for EStreamer Events service

TOTAL TRANSMITTED MESSAGES <3> for Malware Lookup Service service
RECEIVED MESSAGES <2> for Malware Lookup Service) service
SEND MESSAGES <1> for Malware Lookup Service service
HALT REQUEST SEND COUNTER <0> for Malware Lookup Service service
STORED MESSAGES for Malware Lookup Service service (service 0/peer 0)
STATE <Process messages> for Malware Lookup Service service
REQUESTED FOR REMOTE <Process messages> for Malware Lookup Service) service
REQUESTED FROM REMOTE <Process messages> for Malware Lookup Service service

TOTAL TRANSMITTED MESSAGES <6> for service 7000
RECEIVED MESSAGES <3> for service 7000
SEND MESSAGES <3> for service 7000
HALT REQUEST SEND COUNTER <0> for service 7000
STORED MESSAGES for service 7000 (service 0/peer 0)
STATE <Process messages> for service 7000
REQUESTED FOR REMOTE <Process messages> for service 7000
REQUESTED FROM REMOTE <Process messages> for service 7000
TOTAL TRANSMITTED MESSAGES <58> for CSM_CCM service
RECEIVED MESSAGES <38> for CSM_CCM service
SEND MESSAGES <20> for CSM_CCM service
HALT REQUEST SEND COUNTER <0> for CSM_CCM service
STORED MESSAGES for CSM_CCM (service 0/peer 0)
STATE <Process messages> for CSM_CCM service
REQUESTED FOR REMOTE <Process messages> for CSM_CCM service
REQUESTED FROM REMOTE <Process messages> for CSM_CCM service

Priority UE Channel 1 service

TOTAL TRANSMITTED MESSAGES <228> for UE Channel service
RECEIVED MESSAGES <91> for UE Channel service
SEND MESSAGES <137> for UE Channel service
HALT REQUEST SEND COUNTER <0> for UE Channel service
STORED MESSAGES for UE Channel service (service 0/peer 0)
STATE <Process messages> for UE Channel service
REQUESTED FOR REMOTE <Process messages> for UE Channel service
REQUESTED FROM REMOTE <Process messages> for UE Channel service

Priority UE Channel 0 service

TOTAL TRANSMITTED MESSAGES <30> for UE Channel service
RECEIVED MESSAGES <3> for UE Channel service
SEND MESSAGES <27> for UE Channel service
HALT REQUEST SEND COUNTER <0> for UE Channel service
STORED MESSAGES for UE Channel service (service 0/peer 0)
STATE <Process messages> for UE Channel service
REQUESTED FOR REMOTE <Process messages> for UE Channel service
REQUESTED FROM REMOTE <Process messages> for UE Channel service

TOTAL TRANSMITTED MESSAGES <0> for FSTREAM service
RECEIVED MESSAGES <0> for FSTREAM service
SEND MESSAGES <0> for FSTREAM service

Heartbeat Send Time:     Mon Apr  9 07:59:08 2018
Heartbeat Received Time: Mon Apr  9 07:59:15 2018
************************RPC STATUS****192.168.0.200*************
‘ip’ => ‘192.168.0.200’,
‘uuid’ => ‘e5845934-1cb1-11e8-9ca8-c3055116ac45’,
‘ipv6’ => ‘IPv6 is not configured for management’,
‘name’ => ‘192.168.0.200’,
‘active’ => ‘1’,
‘uuid_gw’ => ”,
‘last_changed’ => ‘Mon Apr  9 07:07:16 2018’