Troubleshooting VoIP issues over ASA/PIX/FWSM appliances.

Recently I've been again put in a situation where I had to troubleshoot voice over IP issues via firewall appliances by Cisco.

Here's a brief summary of types of issues:
- No Voice (including one way voice)
- Delay
- No communication
- Features not working (MoH etc).

Now I will be the first to admit - I am voice-over-IP-agnostic, I've touched CCM or CME once or twice in my life. I'm aware that h323, sip and skinny exist.

From my perspective only two things exist in voice over IP.
- signaling
- actual payload (RTP).

So what you should know about how ASA/FWSM interacts with voice traffic.
0. Before going into voice troubleshooting check routing. In many cases a few pings can save a few hours of "missing voice" or "broken signaling"
1. First of all you most likely want to inspect your signaling traffic to/from the internet
a) Be aware that some implementations do not use default ports:
Prime example - SIP - default udp/5060 but known also to use tcp/5060 and udp/5061 (yes for signaling)
b) Be aware that if you're running voice traffic over VPN tunnel you most likely do not need to inspect traffic (everything normally should be permitted). Inspecting over VPN is waste of time (delay) and can account for some one way voice issues.
2. What does the actual inspection do:
a) Dynamically opens pinholes in your security policy to allow incoming traffic (or outgoing if relevant)
b) Pre-allocates connections and translations for traffic.
Thing to take into account here
!!!!!!
If a given IP/phone is not mentioned in signaling the traffic will NOT be allowed in. This comes over and over with music on hold from secondary CCMs. Since none of the parties actually mentions that this particular CCM will be sending MoH the traffic is neatly being dropped because of security policy.
!!!!!!!


3. ASA/FWSM inspects AND changes payload of signaling packets - and that's a good thing.
Firewalls will change internal IP address to public ones according to translation rules in place.
Be mindful when changing inspection or translation rules. "clear xlate ..." or "clear conn ..." has a big chance of recovering if something went wrong after reconfiguration either in inspection rules or NAT.

4. RSVP packets. They usually contain IP options. ASA 8.0 will drop them. Check out ip-option inspection in ASA 8.2.

5. Signaling works but voice does not go properly (dead silence and one way voice issues).
This is most likely caused by a problem with inspection or some additional policies.
Ie. Firewall should automatically open pin holes in security policy for both way RTP traffic.

Check
------
show conn detail add x.x.x.x
show x det | i x.x.x.x
------
Where x.x.x.x is the source or destination of either signaling or voice.

You should have also enabled logging on informational level and monitor logs for entries mentioning IP address of either source of destination. 

Check also those (pick ones interesting)
h323
--------

show h225

show h245

show h323-ras

----------

SIP

-----
show sip

------
SCCP
-----
show skinny
------
I would not advise to get debugs at first - better leave those for interaction with TAC.
FYI - If it comes to TAC intervention, TAC will request "inside" and "outside" interfaces capture and debugs taken AT THE SAME TIME.

What goes without saying - !gathering packet capture helps!
If traffic is NATed and you see inside IP addresses in outside captures ... something went wrong. Maybe something similar to 4.?
Do you see packets coming into the appliance? Do you see them leaving? What has been changed.
FWSM note: Remember that inspected traffic goes straight to CPU!

Good luck.

DVTI, SVTI and routing protocols

edit:
Apparently dynamic routing will be supported with DVTI Cisco is in process of changing documentation to reflect this.


After several days of bugging Cisco SEs and TAC I finally got a clear answer on several questions I had regarding DVTI,SVTI and routing protocols.

It is true that you CAN run routing protocols over DVTI tunnels - however that is NOT recommended OR supported - because it was not tested.

Also according to Cisco SVTI and terminate on DVTI, but again routing protocols is not supported in this case, static routing should be fine though.

I'd like to stress out we're talking about Cisco support for those techs, it still might work but good luck getting support from TAC/SEs.
To quote rfc 1925, section 2 point 3:
"

(3)  With sufficient thrust, pigs fly just fine. However, this is
        not necessarily a good idea. It is hard to be sure where they
        are going to land, and it could be dangerous sitting under them
        as they fly overhead

 "

Read more: http://www.faqs.org/rfcs/rfc1925.html

Troubleshooting FWSM performance

FWSM is the firewall blade for Cat 6500 and 7600 routers.
The appliance has been out for a while and has been proven quite a performance bottle neck (AKA pain in the ***) very often. Specifically because of the placement (data centers) and increasing bandwidth demand there (Tengig links, Nexus). There's slowly a newer device looming on the horizon, but it may take several years.

So first of all, what is FWSM? It's a PIX in a chassis with very neat physical architecture.
You should also know that it has separate logical circuit for both connection for management from chassis (session ... command does a telnet on the backplane). There is also a 6 x gigabit etherchannel,
labeled PoXYZ.

There two dedicated proprietary chips (if you inspect the blade itself you will notice it's done by IBM) handling active connections. 

There is a dedicated network processor to perform access-list, route, xlate lookups.
Coincidentally it will also handle inspected ICMP traffic and ALL IPv6 traffic.

Check out:
http://www.scribd.com/doc/28698783/FWSM-Architecture (page 23)

The actual CPU, where layer 7 is performed and access-list compilation/optimization is done.
High CPU on FWSM does not cause performance degrade for non inspected traffic, in most cases anyway.

If you're checking for actual performance problems introduced by FWSM, please do this one test before making any assumptions.

Connect two gig capable laptop to two different line cards (gig capable obviously), put the laptops into two different VRFs and vlans, best if they are not advertised outside the chassis.
Add the vlans to FWSM apply an access-group for each vlan interface on the blade (it can be "permit ip any any") and create statics if nat-control is enabled.
Test of IPerf with TCP, single transfer, I've done some time ago was giving me just above 800 Mbit/s for outbound connections, around 650Mbit/s for inbound connections - if you want to have some baseline.

This method is of course not perfect, you could most likely tweak the physical and logical infrastructure to get better numbers, but my idea was to perform a test similar to real life.

Things to remember:
- make sure that the connection is not inspected if you want decent performance.
-  If this is TCP and you're getting poor performance consider enabling sysoption np completion-unit, this magic option is invoking special processing created to address scenarios in which FWSM was known to introduce out of order packets for TCP streams.
Please note that completion unit will not take care of out of order packets introduced elsewhere and it only work for traffic handled by first two chips.
- Get packet capture. Do not trust the FWSM capture, do local SPAN. FWSM capture is buggy, it's a bit more decent nowadays, but still is not to be relied on. Cisco TAC usually relies on FWSM capture functionality, but will ask you to do SPAN in some cases.
Please note that captures have to be done simultaneously.
What are we looking for in the captures? Out of order or dropped packets (to some extent you can trust dissector built into wireshark for this task), packets dropped (seen in one capture, but not the other), delay introduced by FWSM for packets passing through.
- FWSM does not play well with SACK option, check headers!
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCeb16752
In newer FWSM versions you can clear SACK option in TCP via sysopt!
http://www.cisco.com/en/US/docs/security/fwsm/fwsm40/command/reference/s8.html#wp2761175
- Things that can  improve performance and/or analysis: disable sequence randomization, state bypass (untested)

So you have a blade and you're getting packet drop and poor performance, where should you look:
- show np block (hardware buffer counters) - if they are non-zero and increasing it's bad. You're most likely running into hardware limitation of the FWSM. Consider running active/active failover to split the traffic into two physical devices, doing traffic shaping towards the FWSM on the chassis or bypass the FWSM completely for some traffic.
- show np all stats | i RTL  and show np all stats | i RL will show you if the packets are dropped because of software rate limiting mechanisms built into network processors.
- show np 3 stats can show if some redundant or extensive traffic is crossing the FWSM.
Good counters to start,
a) check if you're not being by one type of traffic
-------------
  TCP Fixups        
  UDP Fixups          
  ICMP Fixups        

--------------
those will show you how many connection "fixups" were done. 

b)  Everything starting from "Discard Statistics" is usually interesting if you're looking for packets dropped.

c) Take care of "Flow Control: Rate Limit Statistics" - packets software rate limited by the chip.

d) ARP miss indications - are some hosts on your network sending traffic to hosts which do not exist?

TAC and Cisco System Engineers can tell you what's going on and what some of the counters mean. Believe me it's NOT intuitive.

While troubleshooting FWSM performance ALWAYS start with checking the switching path.
CAM entries should point either to PoXYZ or trunk between two chassis', ARP entries for active unit should bear the MAC address of primary unit.

edited on 21/Jun/2010 - added bug for SACK.
edited on 13/Aug/2010 - clarifications, more about NP3.

Troubleshooting DMVPN

DMVPN is a great suite of protocols, from time to time something goes wrong though. Here's a few tips on how to troubleshoot it. It will not tell you how to troubleshoot every part but will rather guide you in narrowing the problem down.

In typical DMVPN scenario you will have following "layers" each dependent on all the ones before.
- Physical and IP
- Crypto
- GRE
- NHRP
- Routing protocol

1. Physical and IP - I'm putting those together since they are not really specific to DMVPN but you need to check if they work.
1.1 Check reachability, from spoke to hub by a simple ping or traceroute.
1.2 Typical problem: IPsec not starting to establish.
Do some basic testing - ping from spoke to hub, make sure not firewall on the way is blocking - UDP/500, UDP/4500 - if NAT-T is needed, ESP/AH.
If everything is configured but tunnel is not initiating... Did you configured NHRP network id?

A typical exercise here and at any level will be to verify CEF switching statistics "show cef drop" in old IOSes or "show cef switching statistic feature" on newer ones.

2. Crypto (IPsec), once you know that nothing is blocked and crypto show start establishing.
- Check that you have phase 1 SAs "show crypto isakmp sa det" the state you're looking for usually is QM_IDLE (or no IKE SA if lifetime is very short)
- Check IPsec SAs show crypto ipsec sa - both inbound and outbound SPI should be there and should be mirrored on other side of the tunnel. (Inbound SPI on spoke will be outbound SPI on hub and vice versa)
debug crypto ipsec and debug cry isakmp are your friends.

Two notable mentions here:
2.1 If in debugs you see tunnels establishing properly but they get torn down in few minutes it most likely means that NHRP relation is not establishing.
2.2 Crypto socket - there is a magic being called a crypto socket that is what is binding crypto and nhrp together - you can debug it - debug crypto socket. Problems with crypto socket can cause 2.1 but can be usually mitigated in short term by removing tunnel interface configuration and adding it back again. Many cases, different IOS versions affected, multiple bugs on Cisco side.

There is also a whole subset of problems with crypto accelerator cards that can show themselves here. Verify "show crypto engine accelerator statistic" and "show crypto engine configurtion" or "show crypto eli" - this will show you statistics and which accelerator is currently being used. You generally check for errors.

3. GRE - here's a fun fact, I've never seen a problem with GRE encapsulation or processing. But I would start by monitoring show interface tunnel X for input or output drops.
One problem you may encounter is .... NAT.
3.1 I've seen a scenario on a fairly recent 12.4T software where NAT was done for GRE traffic (no tunnel protection scenario). Check "sh ip nat trans".
3.2 If by any chance you're using "ip nat outside/inside" on tunnel interface, please check if you're not NATing too much.

4. NHRP - remember that even though the spoke has static NHRP mapping and "show ip nhrp brief" will always show you a mapping present (as opposed to the hub) it is the spoke that is initiating NHRP registration by sending registration request.
Useful debugs:
debug nhrp pack
debug nhrp ext 
debug nhrp err
debug nhrp rate
For each NHRP registration request you should see a packet encapsulated into IPsec (show crypto ipsec sa), if it's not the case enable debug from 2.2 and get in touch with Cisco TAC.

A hood value for NHRP holdtime would be around 300 seconds (as opposed to 7200 default).

5. Routing protocol - once you know all the "layers" below there is the RP level that makes it all tick. I've seen a range of problems here, some bugs, some platform specific (ASR hub taking longer to converge comparing to 7200 with same config). They will range from RP flapping (can be driven by NHRP or load) to downright instability of RP once spokes start connecting to a hub. It can be bug or platform limitation, one can write a book about this :-)

This post was meant to show you what are some common problems and how to track down the failing component. Hope it helps. If you're interested to learn more let me know.

DMVPN phase 3 - basic configuration example.

Phase 3 DMVPN is not a new topic. But Cisco documentation on this matter is bit lacking.

Two best articles are:
http://www.cisco.com/en/US/docs/ios/sec_secure_connectivity/configuration/guide/sec_DMVPN_ps6350_TSD_Products_Configuration_Guide_Chapter.html
combined with:
http://www.cisco.biz/en/US/prod/collateral/iosswrel/ps6537/ps6586/ps6660/ps6808/prod_white_paper0900aecd8055c34e_ps6658_Products_White_Paper.html

I've seen probably around 30 deployments with DMVPN many of them with configuration mistakes. It does not mean that it will not work - DMVPN is a robust beast - but wrong configuration is just asking for trouble later on.

So let's see it. (No crypto configuration, at this point)

First of all OSPF.
Please note that point-to-multipoint OSPF network type configuration has it's drawbacks (/32 for each tunnel address, re-computation on every flap), majority of people will want to use network type of broadcast.

Spoke config:
--------

interface Tunnel1
ip address 172.25.1.2 255.255.255.0
no ip redirects
ip nhrp map multicast 10.1.1.1
ip nhrp map 172.25.1.1 10.1.1.1
ip nhrp network-id 1
ip nhrp nhs 172.25.1.1
ip nhrp shortcut
ip nhrp redirect
ip ospf network point-to-multipoint
tunnel source Loopback0
tunnel mode gre multipoint

Both NHRP redirect and shortcut present.
OSPF network type set to point-to-multipoint.


Hub config:
-------

interface Tunnel1
ip address 172.25.1.4 255.255.255.0
no ip redirects
ip nhrp map multicast dynamic
ip nhrp network-id 1
ip nhrp redirect
ip ospf network point-to-multipoint
tunnel source Loopback0
tunnel mode gre multipoint

Only ip nhrp redirect is configured.

EIGRP spoke
--------

interface Tunnel1
bandwidth 64000
ip address 172.25.1.4 255.255.255.0
no ip redirects
ip nhrp map multicast 10.1.1.3
ip nhrp map 172.25.1.3 10.1.1.3
ip nhrp network-id 1
ip nhrp nhs 172.25.1.3
ip nhrp shortcut
ip nhrp redirect
tunnel source Loopback0
tunnel mode gre multipoint

EIGRP hub
---------

interface Tunnel1
bandwidth 64000
ip address 172.25.1.1 255.255.255.0
no ip redirects
ip nhrp map multicast dynamic
ip nhrp network-id 1
ip nhrp redirect
no ip split-horizon eigrp 1
ip summary-address eigrp 1 10.20.0.0 255.255.0.0 5
tunnel source Loopback0
tunnel mode gre multipoint

Note that in this particular case all the networks that this DMVPN clouds is "protecting" can be summarized into 10.20.0.0/16.

Please note increased bandwidth on tunnel interfaces for EIGRP. Default BW is 8kbit and will be used to calculate metrics for EIGRP.

So how would an intermediate/regional hub configuration look like?

interface Tunnel1
ip address 172.25.1.3 255.255.255.0
no ip redirects
ip nhrp map multicast dynamic
ip nhrp map 172.25.1.1 10.1.1.1
ip nhrp map multicast 10.1.1.1
ip nhrp network-id 1
ip nhrp nhs 172.25.1.1
ip nhrp redirect
no ip split-horizon eigrp 1
ip summary-address eigrp 1 10.20.10.0 255.255.255.0 5
tunnel source Loopback0
tunnel mode gre multipoint

FWSM - routing considerations or "Why clearing xlates solves it?"

edited on 18th May 2010.

Yet another things people tend not to realize (or maybe just those that do not attend networkers?).

All Cisco firewall appliances (ASA/PIX/FWSM) consider xlates before routing information. If some traffic has created an xlate all subsequent traffic will follow the path of xlate and not the route.

What I would like to point out is a problem that has been reported to me quite often.

There is a whole class of problems that manifests itself on the FWSM (mostly, but will also impact ASA/PIX) for which "clear xlate" is the only and temporary solution and here's what you need to know about it.

The problem shows in most cases where you have (pick as many as you like)
- no nat-control
- same security interfaces

There is one cure, which you should configure anyway on your appliance as a best practice.
Configure unicast RPF on ALL interfaces.
http://www.cisco.com/en/US/docs/security/fwsm/fwsm40/configuration/guide/protct_f.html#wp1042625

Why did I mention that the problem MANIFESTS itself on the FWSM?
Because the FWSM is working as expected - when a packet with same IP comes through an interface an xlate will be created. Once the xlate is created the traffic will consider existing xlates before routes.
So bottom line, if you have an improper xlate already installed in your xlate table, things may not work until you clear xlate(s), or it times out. However if you have continuous traffic keeping that xlate alive, the only way to clear this is to clear xlates. 

Unicast RPF will prohibit the bogus packet to create a xlate in the first place.

Why is this mostly seen on the FWSM? Because of the placement of FWSM - it's usually a datacenter with a mix of layer 2 and layer 3 traffic - probably proxy arp enabled on layer 3 interfaces, maybe some route leaking from a VRF that is normally protected by the FWSM and many many others.
Most of the time it's easier to fix the symptom then the root cause.

Enable unicast RPF on the FWSM in all your new deployment or consider contacting your local account team to get following bug integrated:
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsi14227

If you want to get to the bottom of this problem enable information level messages and monitor what connection is causing this bogus xlate to be created. Usually you will see two sequential messages, creating connection, creating xlate. You will need to match the "outage" to creation of the xlate. From there you trace the packets to L2/L3 interfaces all over the network.

ASA/PIX PKI implementation. Mupliple trustpoints considerations.

Not sure if Cisco documents it anywhere, but here goes.

What happens if you have multiple trustpoints defined on the ASA.

When a certificate is presented to the ASA, the appliance can use ANY trust point configured on the device and will use first one matching provided client type is matching.

You cannot change this behavior, except for specifying different certificate usage:
http://www.cisco.com/en/US/docs/security/asa/asa80/command/reference/c4.html#wp2124040

You do however have control over which certificate is being SENT to the peers, this is what you configure under tunnel-groups and ssl CLIs.