FWSM is the firewall blade for Cat 6500 and 7600 routers.
The appliance has been out for a while and has been proven quite a performance bottle neck (AKA pain in the ***) very often. Specifically because of the placement (data centers) and increasing bandwidth demand there (Tengig links, Nexus). There's slowly a newer device looming on the horizon, but it may take several years.
So first of all, what is FWSM? It's a PIX in a chassis with very neat physical architecture.
You should also know that it has separate logical circuit for both connection for management from chassis (session ... command does a telnet on the backplane). There is also a 6 x gigabit etherchannel,
labeled PoXYZ.
There two dedicated proprietary chips (if you inspect the blade itself you will notice it's done by IBM) handling active connections.
There is a dedicated network processor to perform access-list, route, xlate lookups.
Coincidentally it will also handle inspected ICMP traffic and ALL IPv6 traffic.
Check out:
http://www.scribd.com/doc/28698783/FWSM-Architecture (page 23)
The actual CPU, where layer 7 is performed and access-list compilation/optimization is done.
High CPU on FWSM does not cause performance degrade for non inspected traffic, in most cases anyway.
If you're checking for actual performance problems introduced by FWSM, please do this one test before making any assumptions.
Connect two gig capable laptop to two different line cards (gig capable obviously), put the laptops into two different VRFs and vlans, best if they are not advertised outside the chassis.
Add the vlans to FWSM apply an access-group for each vlan interface on the blade (it can be "permit ip any any") and create statics if nat-control is enabled.
Test of IPerf with TCP, single transfer, I've done some time ago was giving me just above 800 Mbit/s for outbound connections, around 650Mbit/s for inbound connections - if you want to have some baseline.
This method is of course not perfect, you could most likely tweak the physical and logical infrastructure to get better numbers, but my idea was to perform a test similar to real life.
Things to remember:
- make sure that the connection is not inspected if you want decent performance.
- If this is TCP and you're getting poor performance consider enabling sysoption np completion-unit, this magic option is invoking special processing created to address scenarios in which FWSM was known to introduce out of order packets for TCP streams.
Please note that completion unit will not take care of out of order packets introduced elsewhere and it only work for traffic handled by first two chips.
- Get packet capture. Do not trust the FWSM capture, do local SPAN. FWSM capture is buggy, it's a bit more decent nowadays, but still is not to be relied on. Cisco TAC usually relies on FWSM capture functionality, but will ask you to do SPAN in some cases.
Please note that captures have to be done simultaneously.
What are we looking for in the captures? Out of order or dropped packets (to some extent you can trust dissector built into wireshark for this task), packets dropped (seen in one capture, but not the other), delay introduced by FWSM for packets passing through.
- FWSM does not play well with SACK option, check headers!
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCeb16752
In newer FWSM versions you can clear SACK option in TCP via sysopt!
http://www.cisco.com/en/US/docs/security/fwsm/fwsm40/command/reference/s8.html#wp2761175
- Things that can improve performance and/or analysis: disable sequence randomization, state bypass (untested)
So you have a blade and you're getting packet drop and poor performance, where should you look:
-
show np block (hardware buffer counters) - if they are non-zero and increasing it's bad. You're most likely running into hardware limitation of the FWSM. Consider running active/active failover to split the traffic into two physical devices, doing traffic shaping towards the FWSM on the chassis or bypass the FWSM completely for some traffic.
-
show np all stats | i RTL and
show np all stats | i RL will show you if the packets are dropped because of software rate limiting mechanisms built into network processors.
-
show np 3 stats can show if some redundant or extensive traffic is crossing the FWSM.
Good counters to start,
a) check if you're not being by one type of traffic
-------------
TCP Fixups
UDP Fixups
ICMP Fixups
--------------
those will show you how many connection "fixups" were done.
b) Everything starting from "Discard Statistics" is usually interesting if you're looking for packets dropped.
c) Take care of "Flow Control: Rate Limit Statistics" - packets software rate limited by the chip.
d) ARP miss indications - are some hosts on your network sending traffic to hosts which do not exist?
TAC and Cisco System Engineers can tell you what's going on and what some of the counters mean. Believe me it's NOT intuitive.
While troubleshooting FWSM performance ALWAYS start with checking the switching path.
CAM entries should point either to PoXYZ or trunk between two chassis', ARP entries for active unit should bear the MAC address of primary unit.
edited on 21/Jun/2010 - added bug for SACK.
edited on 13/Aug/2010 - clarifications, more about NP3.
