Menu

US Region

Grandmetric LLC
Lewes DE 19958
16192 Coastal Hwy USA
EIN: 98-1615498
+1 302 691 94 10
info@grandmetric.com

EMEA Region

GRANDMETRIC Sp. z o.o.
ul. Metalowa 5, 60-118 Poznań, Poland
NIP 7792433527
+48 61 271 04 43
info@grandmetric.com

UK

Grandmetric LTD
Office 584b
182-184 High Street North
London
E6 2JA
+44 20 3321 5276
info@grandmetric.com

  • en
  • pl
  • Troubleshooting FMC and Cisco Firepower Sensor communication

    Troubleshooting FMC and Cisco Firepower Sensor communication

    Date: 23.04.2018

    Author:


    In this post we are going to focus on the scripts included in FTD and FMC operating systems that help to troubleshoot connections between FTD sensors and Cisco Firepower Management Center. As they are run from the “expert mode” (super user), it is better that you have a deep understanding of any potential impact on the production environment.

    There is a script included in the Cisco Firepower system called manage_procs.pl (use it wisely). It can be run from the FTD expert mode or the FMC. It allows you to restart the communication channel between both devices. Be careful, if you run it from the FMC and you have hundreds of sensors it will reestablish all communication channels to all of your sensors at once. If you run it from the FTD then only the particular sensor – FMC communication will be affected.


    > expert
    admin@FTDv:~$ sudo su
    Password:
    root@FTDv:/home/admin# manage_procs.pl
    ****************  Configuration Utility  **************
    1   Reconfigure Correlator
    2   Reconfigure and flush Correlator
    3   Restart Comm. channel
    4   Update routes
    5   Reset all routes
    6   Validate Network
    0   Exit
    **************************************************************
    Enter choice:

    I am using 3th, 4th and 5th option. It can take few seconds to proceed. This scripts are nice to be used when the FMC and FTD have communication problems like heartbeats are not received, policy deployment is failing or events are not received. These options reestablish the secure channels between both peers, verifying the certificates and creating new config file on the backend.

    If you still have problems then you can see all the debugging messages in a separate SSH session to the sensor.

    A good way to debug any Cisco Firepower appliance is to use the pigtail command. It gives real time outputs from a bunch of log files. So lets execute manage_procs.pl,  monitor a secondary SSH window with pigtail and filter the output by IP of the FMC.  Keep in mind that you may use the pigtail command during the registration process and monitor where the registration is failing.

     

    root@FTDv:/home/admin# pigtail | grep 192.168.0.200
    MSGS: 04-09 07:48:46 FTDv SF-IMS[9200]: [9200] sfmgr:sfmanager [INFO] MARK TO FREE peer 192.168.0.200
    MSGS: 04-09 07:48:46 FTDv SF-IMS[9200]: [13244] sfmgr:sfmanager [INFO] WRITE_THREAD:Terminated sftunnel write thread for peer 192.168.0.200
    MSGS: 04-09 07:48:48 FTDv SF-IMS[9200]: [13243] sfmgr:sfmanager [INFO] Stop child thread for peer 192.168.0.200
    MSGS: 04-09 07:48:48 FTDv SF-IMS[9200]: [13243] sfmgr:sfmanager [INFO] Exiting child thread for peer 192.168.0.200
    MSGS: 04-09 07:48:48 FTDv SF-IMS[9200]: [13243] sfmgr:sfmanager [INFO] free_peer 192.168.0.200.MSGS: 04-09 07:48:50 FTDv SF-IMS[9201]: [13428] sfmbservice:sfmb_service [INFO] TERM:Peer 192.168.0.200 removed
    MSGS: 04-09 07:48:57 FTDv SF-IMS[5575]: [13337] SFDataCorrelator:EventStreamHandler [INFO] Reset: Closing estreamer connection to:192.168.0.200
    SERR: 04-09 07:48:50 2018-04-09 07:48:58 sfmbservice[9201]:FTDvSF-IMS[9201]: [13428] sfmbservice:sfmb_service [INFO] TERM:Peer 192.168.0.200 removed
    MSGS: 04-09 07:48:58 FTDv SF-IMS[14543]: [14546] sfmbservice:sfmb_service [INFO] Start getting MB messages for 192.168.0.200
    MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14541] sftunneld:sf_peers [INFO] Using a 20 entry queue for 192.168.0.200 - 8104
    MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14541] sftunneld:sf_peers [INFO] Using a 20 entry queue for 192.168.0.200 - 8121
    MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14541] sftunneld:stream_file [INFO] Stream CTX initialized for 192.168.0.200
    MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14551] sftunneld:sf_peers [INFO] Peer 192.168.0.200 needs a single connection
    MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14551] sftunneld:sf_connections [INFO] Start connection to : 192.168.0.200 (wait 0 seconds is up)
    MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_peers [INFO] Peer 192.168.0.200 needs a single connection
    MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_ssl[INFO] Connect to 192.168.0.200 on port 8305 - br1
    MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_ssl[INFO] Initiate IPv4 connection to 192.168.0.200 (via br1)
    MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_ssl[INFO] Initiating IPv4 connection to 192.168.0.200:8305/tcp
    MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_ssl[INFO] Wait to connect to 8305 (IPv6): 192.168.0.200
    MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_ssl[INFO] Connect to 192.168.0.200 failed on port 8305 socket 11 (Connection refused)MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_ssl[INFO] No IPv4 connection to 192.168.0.200
    MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_ssl[WARN] Unable to connect to peer '192.168.0.200'
    MSGS: 04-09 07:48:58 FTDv SF-IMS[14541]: [14552] sftunneld:sf_ssl[INFO] reconnect to peer '192.168.0.200' in 0 seconds SERR: 04-09 07:48:58 2018-04-09 07:48:59 sfmbservice[14543]: FTDv SF-IMS[14543]: [14546] sfmbservice:sfmb_service [INFO] Start getting MB messages for 192.168.0.200
    MSGS: 04-09 07:49:00 FTDv SF-IMS[14541]: [14551] sftunneld:sf_peers [INFO] Peer 192.168.0.200 needs a single connection

     

    Another great tool inherited by Sourcefire is sftunnel_status.pl. It is a script that shows all details related to the communication between the sensor and the FMC. The most important are the outputs showing the status of the Channel A and Channel B. These are the management and the eventing channels. In more complex Cisco Firepower designs these are two separate physical connections which enhance the policy push time and the logging features.

     

    root@FTDv:/home/admin# sftunnel_status.pl
    SFTUNNEL Start Time: Mon Apr  9 07:48:59 2018
    Both IPv4 and IPv6 connectivity is supported
    Broadcast count = 0
    Reserved SSL connections: 0
    Management Interfaces: 1
    br1 (control events) 192.168.0.201,
    *************************RUN STATUS****192.168.0.200*************
    Key File    = /var/sf/peers/e5845934-1cb1-11e8-9ca8-c3055116ac45/sftunnel-key.pem
    Cert File   = /var/sf/peers/e5845934-1cb1-11e8-9ca8-c3055116ac45/sftunnel-cert.pem
    CA Cert     = /var/sf/peers/e5845934-1cb1-11e8-9ca8-c3055116ac45/cacert.pem
    Cipher used = AES256-GCM-SHA384 (strength:256 bits)
    ChannelA Connected: Yes, Interface br1
    Cipher used = AES256-GCM-SHA384 (strength:256 bits)
    ChannelB Connected: Yes, Interface br1
    Registration: Completed.
    IPv4 Connection to peer '192.168.0.200' Start Time: Mon Apr  9 07:49:01 2018


    PEER INFO:
    sw_version 6.2.2.2
    sw_build 109
    Management Interfaces: 1
    eth0 (control events) 192.168.0.200,
    Peer channel Channel-A is valid  type (CONTROL), using 'br1', connected to '192.168.0.200' via '192.168.0.201'
    Peer channel Channel-B is valid  type (EVENT), using 'br1', connected to '192.168.0.200' via '192.168.0.201'

    TOTAL TRANSMITTED MESSAGES <16> for IP(NTP) service
    RECEIVED MESSAGES <8> for IP(NTP) service
    SEND MESSAGES <8> for IP(NTP) service
    HALT REQUEST SEND COUNTER <0> for IP(NTP) service
    STORED MESSAGES for IP(NTP) service (service 0/peer 0)
    STATE <Process messages> for IP(NTP) service
    REQUESTED FOR REMOTE <Process messages> for IP(NTP) service
    REQUESTED FROM REMOTE <Process messages> for IP(NTP) service

    TOTAL TRANSMITTED MESSAGES <4> for Health Events service
    RECEIVED MESSAGES <2> for Health Events service
    SEND MESSAGES <2> for Health Events service
    HALT REQUEST SEND COUNTER <0> for Health Events service
    STORED MESSAGES for Health service (service 0/peer 0)
    STATE <Process messages> for Health Events service
    REQUESTED FOR REMOTE <Process messages> for Health Events service
    REQUESTED FROM REMOTE <Process messages> for Health Events service

    TOTAL TRANSMITTED MESSAGES <3> for Identity service
    RECEIVED MESSAGES <2> for Identity service
    SEND MESSAGES <1> for Identity service
    HALT REQUEST SEND COUNTER <0> for Identity service
    STORED MESSAGES for Identity service (service 0/peer 0)
    STATE <Process messages> for Identity service
    REQUESTED FOR REMOTE <Process messages> for Identity service
    REQUESTED FROM REMOTE <Process messages> for Identity service

    TOTAL TRANSMITTED MESSAGES <44> for RPC service
    RECEIVED MESSAGES <22> for RPC service
    SEND MESSAGES <22> for RPC service
    HALT REQUEST SEND COUNTER <0> for RPC service
    STORED MESSAGES for RPC service (service 0/peer 0)
    STATE <Process messages> for RPC service
    REQUESTED FOR REMOTE <Process messages> for RPC service
    REQUESTED FROM REMOTE <Process messages> for RPC service
    TOTAL TRANSMITTED MESSAGES <14> for IDS Events service
    RECEIVED MESSAGES <7> for service IDS Events service
    SEND MESSAGES <7> for IDS Events service
    HALT REQUEST SEND COUNTER <0> for IDS Events service
    STORED MESSAGES for IDS Events service (service 0/peer 0)
    STATE <Process messages> for IDS Events service
    REQUESTED FOR REMOTE <Process messages> for IDS Events service
    REQUESTED FROM REMOTE <Process messages> for IDS Events service

     

    TOTAL TRANSMITTED MESSAGES <23> for EStreamer Events service
    RECEIVED MESSAGES <11> for service EStreamer Events service
    SEND MESSAGES <12> for EStreamer Events service
    HALT REQUEST SEND COUNTER <0> for EStreamer Events service
    STORED MESSAGES for EStreamer Events service (service 0/peer 0)
    STATE <Process messages> for EStreamer Events service
    REQUESTED FOR REMOTE <Process messages> for EStreamer Events service
    REQUESTED FROM REMOTE <Process messages> for EStreamer Events service

     

    TOTAL TRANSMITTED MESSAGES <3> for Malware Lookup Service service
    RECEIVED MESSAGES <2> for Malware Lookup Service) service
    SEND MESSAGES <1> for Malware Lookup Service service
    HALT REQUEST SEND COUNTER <0> for Malware Lookup Service service
    STORED MESSAGES for Malware Lookup Service service (service 0/peer 0)
    STATE <Process messages> for Malware Lookup Service service
    REQUESTED FOR REMOTE <Process messages> for Malware Lookup Service) service
    REQUESTED FROM REMOTE <Process messages> for Malware Lookup Service service

     

    TOTAL TRANSMITTED MESSAGES <6> for service 7000
    RECEIVED MESSAGES <3> for service 7000
    SEND MESSAGES <3> for service 7000
    HALT REQUEST SEND COUNTER <0> for service 7000
    STORED MESSAGES for service 7000 (service 0/peer 0)
    STATE <Process messages> for service 7000
    REQUESTED FOR REMOTE <Process messages> for service 7000
    REQUESTED FROM REMOTE <Process messages> for service 7000
    TOTAL TRANSMITTED MESSAGES <58> for CSM_CCM service
    RECEIVED MESSAGES <38> for CSM_CCM service
    SEND MESSAGES <20> for CSM_CCM service
    HALT REQUEST SEND COUNTER <0> for CSM_CCM service
    STORED MESSAGES for CSM_CCM (service 0/peer 0)
    STATE <Process messages> for CSM_CCM service
    REQUESTED FOR REMOTE <Process messages> for CSM_CCM service
    REQUESTED FROM REMOTE <Process messages> for CSM_CCM service

    Priority UE Channel 1 service

    TOTAL TRANSMITTED MESSAGES <228> for UE Channel service
    RECEIVED MESSAGES <91> for UE Channel service
    SEND MESSAGES <137> for UE Channel service
    HALT REQUEST SEND COUNTER <0> for UE Channel service
    STORED MESSAGES for UE Channel service (service 0/peer 0)
    STATE <Process messages> for UE Channel service
    REQUESTED FOR REMOTE <Process messages> for UE Channel service
    REQUESTED FROM REMOTE <Process messages> for UE Channel service

     

    Priority UE Channel 0 service

     

    TOTAL TRANSMITTED MESSAGES <30> for UE Channel service
    RECEIVED MESSAGES <3> for UE Channel service
    SEND MESSAGES <27> for UE Channel service
    HALT REQUEST SEND COUNTER <0> for UE Channel service
    STORED MESSAGES for UE Channel service (service 0/peer 0)
    STATE <Process messages> for UE Channel service
    REQUESTED FOR REMOTE <Process messages> for UE Channel service
    REQUESTED FROM REMOTE <Process messages> for UE Channel service

     

    TOTAL TRANSMITTED MESSAGES <0> for FSTREAM service
    RECEIVED MESSAGES <0> for FSTREAM service
    SEND MESSAGES <0> for FSTREAM service

    Heartbeat Send Time:     Mon Apr  9 07:59:08 2018
    Heartbeat Received Time: Mon Apr  9 07:59:15 2018
    ************************RPC STATUS****192.168.0.200*************
    ‘ip’ => ‘192.168.0.200’,
    ‘uuid’ => ‘e5845934-1cb1-11e8-9ca8-c3055116ac45’,
    ‘ipv6’ => ‘IPv6 is not configured for management’,
    ‘name’ => ‘192.168.0.200’,
    ‘active’ => ‘1’,
    ‘uuid_gw’ => ”,
    ‘last_changed’ => ‘Mon Apr  9 07:07:16 2018’

    More in Cisco Firepower Online Training

    Let us guide you through Cisco Firepower Threat Defense technology (FTD) along with Firepower Management Center (FMC) as security management and reporting environment.

    Check training agenda

    Author

    Ivan Radev

    4 Comments
    Trinath Gogineni
    11 July 2019 at 22:04

    Thanks for your Information….

     
    Dmitri Mendeleev
    19 December 2019 at 20:43

    Thank you very much! I was looking for this.
    My Firepower ran out of space because of the bug CSCvb61055 and I wanted to restore communication without restarting it.
    Have a good one!

     
    DVAL NECEMON YANNICK DOH
    14 May 2020 at 15:53

    This is a top blog. Could you please share more scenarios and more troubleshooting commands?
    Thanks

     
    Ivan Espinoza
    5 June 2020 at 23:44

    Hello Ivan,

    Good joob, let me tell you I’m facing a similar issue with the FMC, this is not showing all events passing through it, I’m thinking to copy the backup to another FMC and check.

    Regards,

     

    Leave a Reply

    Your email address will not be published. Required fields are marked *


    Grandmetric