FWSM - routing considerations or "Why clearing xlates solves it?"

edited on 18th May 2010.

Yet another things people tend not to realize (or maybe just those that do not attend networkers?).

All Cisco firewall appliances (ASA/PIX/FWSM) consider xlates before routing information. If some traffic has created an xlate all subsequent traffic will follow the path of xlate and not the route.

What I would like to point out is a problem that has been reported to me quite often.

There is a whole class of problems that manifests itself on the FWSM (mostly, but will also impact ASA/PIX) for which "clear xlate" is the only and temporary solution and here's what you need to know about it.

The problem shows in most cases where you have (pick as many as you like)
- no nat-control
- same security interfaces

There is one cure, which you should configure anyway on your appliance as a best practice.
Configure unicast RPF on ALL interfaces.
http://www.cisco.com/en/US/docs/security/fwsm/fwsm40/configuration/guide/protct_f.html#wp1042625

Why did I mention that the problem MANIFESTS itself on the FWSM?
Because the FWSM is working as expected - when a packet with same IP comes through an interface an xlate will be created. Once the xlate is created the traffic will consider existing xlates before routes.
So bottom line, if you have an improper xlate already installed in your xlate table, things may not work until you clear xlate(s), or it times out. However if you have continuous traffic keeping that xlate alive, the only way to clear this is to clear xlates. 

Unicast RPF will prohibit the bogus packet to create a xlate in the first place.

Why is this mostly seen on the FWSM? Because of the placement of FWSM - it's usually a datacenter with a mix of layer 2 and layer 3 traffic - probably proxy arp enabled on layer 3 interfaces, maybe some route leaking from a VRF that is normally protected by the FWSM and many many others.
Most of the time it's easier to fix the symptom then the root cause.

Enable unicast RPF on the FWSM in all your new deployment or consider contacting your local account team to get following bug integrated:
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsi14227

If you want to get to the bottom of this problem enable information level messages and monitor what connection is causing this bogus xlate to be created. Usually you will see two sequential messages, creating connection, creating xlate. You will need to match the "outage" to creation of the xlate. From there you trace the packets to L2/L3 interfaces all over the network.