ASA 8.3 part 2 - webvpn

Stuck at the airport - flight delayed. So I might as well write a bit more about my findings on ASA 8.3.

This time I'm going to talk about webvpn.

Frankly speaking webvpn is the only reason why you should consider 8.3.

Here are my highlights:
- Finally! you can smart tunnel your home page (without any hacks and tricks), downside to this - no webvpn toolbar, java needed, only windows and mac support for smart tunnel.
- A very extended OS/Browser support list, check out:
http://www.cisco.com/en/US/docs/security/asa/asa83/asdm63/configuration_guide/vpn_web.html#wp1052888
- Smart tunnel notification (now this might have been there before, but I noticed it just now)
When you enable smart tunneling (on both mac and windows) a notification icon can appear:
http://www.cisco.com/en/US/docs/security/asa/asa83/asdm63/configuration_guide/vpn_web.html#wp1055586
- Auto signon for smart tunnel:
http://www.cisco.com/en/US/docs/security/asa/asa83/asdm63/configuration_guide/vpn_web.html#wp1055462
The functionality is there but please be aware it has very limited scope.
- You'll be also (pleasantly?) surprised by the new way one can customize their webvpn portal.


- A very "important" change in webvpn config guide regarding APCF:
"Cisco TAC may help with APCF to address specific rendering issues if the smart tunneling feature is not working or cannot be used."

Things still to be addressed, from top of my head - webvpn cache "memory leak":
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtb68311


And here's the complete config guide:
http://www.cisco.com/en/US/docs/security/asa/asa83/asdm63/configuration_guide/vpn_web.html

ASA 8.3 part 1 - NAT

edit:
In the meantime Cisco published ASA 8.3 migration guide:
http://www.cisco.com/en/US/docs/security/asa/asa83/upgrading/migrating.html
Please refer to it for all NAT migration questions, otherwise do go on, there's some info compiled below :-)

Please be aware of following bugs (once I stumbled upon):

- NAT RPF failure in ASA 8.3

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCth72642

- Access-list migration based on NAT config.

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtf57830

- "Unidirectional" keyword added:

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCti36048
If any of the above links doesn't work go  to bugtoolkit directly:
http://www.cisco.com/cgi-bin/Support/Bugtool/launch_bugtool.pl

Recently I found quite a few discussion on supportforums regarding ASA 8.3 and related changes.

While majority of changes are good and much anticipated (ACL, webvpn - which I will try to give highlights at some other day) some changes are confusing at best especially for people who already knew NAT that came from PIX - or everything on ASA prior to 8.3.1 release.

In this post I do not intend to re-write configuration guide, it's there and it has ALL the information you might need, I intend to show how you CAN (and believe me there are multiple ways) how one can translate old PIX/ASA NAT CLI to ASA 8.3. That being said don't take my words for granted and have a look at configuration guide:

http://www.cisco.com/en/US/docs/security/asa/asa83/configuration/guide/nat_overview.html



Let's take a look at changes having in mind old order of NAT operation.
1) Nat exemption (this is NOT the only way to do it, but one that will give you least trouble)

Regarding this one ... you might be careful to check  CSCth72642 and some other bugs.
You might need to add reverse rule! 
ie. nat (out,in) sour  sta REMOTE REMOTE dest sta LOCAL LOCAL

Old Syntax:
access-list NAT0 permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0
nat (inside_interface) 0 access-list NAT0

New syntax:
object network MY_NETWORK
subnet 192.168.1.0 255.255.255.0

object network FOREIGN_NETWORK
subnet 192.168.2.0 255.255.255.0
nat (inside_interface,any) source static MY_NETWORK MY_NETWORK destination static FOREIGN_NETWORK FOREIGN_NETWORK





Another example, we're assuming:
- "inside" is the inside interface
- 192.168.0.0/24 is local subnet
- 172.16.0.0/24 is subnet for our remote location (or remote access VPN clients - SSL/IPsec ..)



Old syntax:
access-list NAT0 permit ip 192.168.0.0 255.255.255.0 172.16.0.0 255.255.255.0
nat (inside) 0 access-l NAT0


New syntac:
object network HOME
subnet 192.168.0.0 255.255.0.0


object network REMOTE
subnet 172.16.0.0 255.255.0.0


nat (inside,outside) source static HOME HOME destination static REMOTE REMOTE
It's not exempting anything from NAT but it's making sure there's a static translation when HOME is going REMOTE , for both HOME and REMOTE.

2) Static

- NAT
Old syntax
static (inside_interface,outside_interface) 192.0.2.1 192.168.1.2 netmask 255.255.255.255

New syntax:
object network INTERNAL_IP1
host 192.168.1.2
nat (inside_interface,outside_interface) static 192.0.2.1

- PAT
Old syntax:
static (inside_interface,outside_interface) udp 192.0.2.3 1611 192.168.1.3 161 netmask 255.255.255.255

New syntax:
object network INTERNAL_IP2
host 192.168.1.3
nat (inside_interface,outside_interface) static 192.0.2.3 service udp 161 1611

3. NAT
Old syntax:
nat (inside_interface) 1 192.168.1.0 255.255.255.0
global (outside_interface) 1 192.0.2.254

New syntax: object network MY_SUBNET
subnet 192.168.1.0 255.255.255.0
nat (inside_interface,outside_interface) dynamic 192.0.2.254
However consider, instead, adding a rule similar to this:
nat (any,outside) after-auto source dynamic any interface
Translate everything ELSE to interface via PAT.

Here's order of operation for NAT on 8.3
http://www.cisco.com/en/US/docs/security/asa/asa83/configuration/guide/nat_overview.html#wp1118157
if you check it's NOT that much different comparing to older ones.

So what's the big benefit of 8.3 NAT:
- "any" keyword in NAT (any source, any interface )
- A few new thing are available now man-to-few translations ... possibly nothing ground breaking.

Disadvantages:
- not readable via CLI - everything is an object. ASDM oriented. ("show nat" command DOES help A LOT)
- expanded ("exploding") configuration listing.
- lack of ACL support in NAT rules.
- no additional option on NAT rules (norandom, maximum connection count for example), need to use MPF.

There's of course more to NAT on ASA 8.3 in comparison - and I can post more examples if someone is interested.


Disclaimer:
The above worked on my ASA connected to the lab without REQUIRED memory upgrade.


edit:
18th Aug 2010 - Added some ASA 8.3 NAT bugs and link to Cisco's ASA migration.

PPPoE - config example.

Thanks to Oliver P. from Cisco TAC for this one.

Yet another configuration I keep to quickly test PPPoE or particular features over PPPoE on Cisco routers.

Server

-----------
aaa new-model
!
aaa authentication ppp default local
!
virtual-template 1 pre-clone 20
!
user cisco password cisco
!
bba-group pppoe global
 virtual-template 1
!
interface faX/Y
 pppoe enable
!
interface Loopback0
 ip address 6.6.66.66 255.255.255.255
!
interface Virtual-Template1
 ip unnumbered Loopback0
 peer default ip address pool mypool
 no keepalive
 ppp authentication pap
!
ip local pool mypool 66.66.66.66 66.66.66.166

-----------

Client

-----------

bba-group pppoe global

int  faX/Y
 description PPPOE interface
 no ip address
 duplex auto
 speed auto
 pppoe enable group global
 pppoe-client dial-pool-number 1
!
!
interface Dialer1
 mtu 1454
 ip address negotiated
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 encapsulation ppp
 dialer pool 1
 dialer-group 1
 no keepalive
 ppp authentication pap callin
 ppp pap sent-username cisco password 0 cisco
 ppp ipcp dns request accept
!
!
dialer-list 1 protocol ip permit
!

ip route 0.0.0.0 0.0.0.0 dialer1

-----------

L2tp over IPsec - testing configuration.

This is a configuration based on an actual customer's setup.
I keep it whenever I need to implement L2tp over IPsec on Cisco routers.

Spoke config 
--------------

l2tp-class l2tpclass1
!
pseudowire-class pwclass1
 encapsulation l2tpv2
 ip local interface FastEthernet0/0
!         
interface Virtual-PPP10
 ip address negotiated
 no peer neighbor-route
 no cdp enable
 ppp authentication pap callin
 ppp pap sent-username cisco password 0 cisco
 pseudowire 1.1.1.1 31 pw-class pwclass1
!         
dialer-list 31 protocol ip permit 

ip route 10.0.0.0 255.0.0.0 Virtual-PPP10

--------------

HUB config 
--------------

vpdn enable
vpdn tunnel accounting network default
vpdn session accounting network default
!
vpdn-group DailIn-L2TP/IPsec
! Default L2TP VPDN group
 accept-dialin
  protocol l2tp
  virtual-template 100
 lcp renegotiation on-mismatch
 no l2tp tunnel authentication
 l2tp tunnel timeout setup 60
 ip pmtu
!
!
username cisco privilege 15 password cisco
!
crypto keyring RING
  pre-shared-key address 0.0.0.0 0.0.0.0 key cisco
!
crypto isakmp policy 10
 encr aes 256
 authentication pre-share
 group 5
crypto isakmp key cisco address 0.0.0.0 0.0.0.0 no-xauth
crypto isakmp keepalive 30 5
crypto isakmp aggressive-mode disable
crypto isakmp profile L2TP
   description L2TP/IPsec HW CLients Profile
   keyring RING
   match identity address 0.0.0.0
!
!
crypto ipsec transform-set ICW esp-aes 256 esp-sha-hmac
 mode transport

crypto dynamic-map CryDynMapICW 10
 set nat demux
 set security-association lifetime kilobytes 3145728
 set transform-set ICW
 set pfs group5
 set isakmp-profile L2TP
 reverse-route
!
!
crypto map CryMapICW local-address Ethernet0/0
crypto map CryMapICW 10 ipsec-isakmp dynamic CryDynMapICW
!
!
interface Loopback100
 description L2TP Address
 ip address 10.255.0.1 255.255.255.255
!
interface Ethernet0/0
 ip address 1.1.1.1 255.255.255.0
 crypto map CryMapICW
!
interface Virtual-Template100
 ip unnumbered Loopback100
 peer default ip address pool L2TPOOL
 ppp chap hostname cisco
 ppp chap password 0 cisco
!
ip local pool L2TPOOL 5.5.5.1 5.5.5.254

------------------

Troubleshooting VoIP issues over ASA/PIX/FWSM appliances.

Recently I've been again put in a situation where I had to troubleshoot voice over IP issues via firewall appliances by Cisco.

Here's a brief summary of types of issues:
- No Voice (including one way voice)
- Delay
- No communication
- Features not working (MoH etc).

Now I will be the first to admit - I am voice-over-IP-agnostic, I've touched CCM or CME once or twice in my life. I'm aware that h323, sip and skinny exist.

From my perspective only two things exist in voice over IP.
- signaling
- actual payload (RTP).

So what you should know about how ASA/FWSM interacts with voice traffic.
0. Before going into voice troubleshooting check routing. In many cases a few pings can save a few hours of "missing voice" or "broken signaling"
1. First of all you most likely want to inspect your signaling traffic to/from the internet
a) Be aware that some implementations do not use default ports:
Prime example - SIP - default udp/5060 but known also to use tcp/5060 and udp/5061 (yes for signaling)
b) Be aware that if you're running voice traffic over VPN tunnel you most likely do not need to inspect traffic (everything normally should be permitted). Inspecting over VPN is waste of time (delay) and can account for some one way voice issues.
2. What does the actual inspection do:
a) Dynamically opens pinholes in your security policy to allow incoming traffic (or outgoing if relevant)
b) Pre-allocates connections and translations for traffic.
Thing to take into account here
!!!!!!
If a given IP/phone is not mentioned in signaling the traffic will NOT be allowed in. This comes over and over with music on hold from secondary CCMs. Since none of the parties actually mentions that this particular CCM will be sending MoH the traffic is neatly being dropped because of security policy.
!!!!!!!


3. ASA/FWSM inspects AND changes payload of signaling packets - and that's a good thing.
Firewalls will change internal IP address to public ones according to translation rules in place.
Be mindful when changing inspection or translation rules. "clear xlate ..." or "clear conn ..." has a big chance of recovering if something went wrong after reconfiguration either in inspection rules or NAT.

4. RSVP packets. They usually contain IP options. ASA 8.0 will drop them. Check out ip-option inspection in ASA 8.2.

5. Signaling works but voice does not go properly (dead silence and one way voice issues).
This is most likely caused by a problem with inspection or some additional policies.
Ie. Firewall should automatically open pin holes in security policy for both way RTP traffic.

Check
------
show conn detail add x.x.x.x
show x det | i x.x.x.x
------
Where x.x.x.x is the source or destination of either signaling or voice.

You should have also enabled logging on informational level and monitor logs for entries mentioning IP address of either source of destination. 

Check also those (pick ones interesting)
h323
--------

show h225

show h245

show h323-ras

----------

SIP

-----
show sip

------
SCCP
-----
show skinny
------
I would not advise to get debugs at first - better leave those for interaction with TAC.
FYI - If it comes to TAC intervention, TAC will request "inside" and "outside" interfaces capture and debugs taken AT THE SAME TIME.

What goes without saying - !gathering packet capture helps!
If traffic is NATed and you see inside IP addresses in outside captures ... something went wrong. Maybe something similar to 4.?
Do you see packets coming into the appliance? Do you see them leaving? What has been changed.
FWSM note: Remember that inspected traffic goes straight to CPU!

Good luck.

DVTI, SVTI and routing protocols

edit:
Apparently dynamic routing will be supported with DVTI Cisco is in process of changing documentation to reflect this.


After several days of bugging Cisco SEs and TAC I finally got a clear answer on several questions I had regarding DVTI,SVTI and routing protocols.

It is true that you CAN run routing protocols over DVTI tunnels - however that is NOT recommended OR supported - because it was not tested.

Also according to Cisco SVTI and terminate on DVTI, but again routing protocols is not supported in this case, static routing should be fine though.

I'd like to stress out we're talking about Cisco support for those techs, it still might work but good luck getting support from TAC/SEs.
To quote rfc 1925, section 2 point 3:
"

(3)  With sufficient thrust, pigs fly just fine. However, this is
        not necessarily a good idea. It is hard to be sure where they
        are going to land, and it could be dangerous sitting under them
        as they fly overhead

 "

Read more: http://www.faqs.org/rfcs/rfc1925.html

Troubleshooting FWSM performance

FWSM is the firewall blade for Cat 6500 and 7600 routers.
The appliance has been out for a while and has been proven quite a performance bottle neck (AKA pain in the ***) very often. Specifically because of the placement (data centers) and increasing bandwidth demand there (Tengig links, Nexus). There's slowly a newer device looming on the horizon, but it may take several years.

So first of all, what is FWSM? It's a PIX in a chassis with very neat physical architecture.
You should also know that it has separate logical circuit for both connection for management from chassis (session ... command does a telnet on the backplane). There is also a 6 x gigabit etherchannel,
labeled PoXYZ.

There two dedicated proprietary chips (if you inspect the blade itself you will notice it's done by IBM) handling active connections. 

There is a dedicated network processor to perform access-list, route, xlate lookups.
Coincidentally it will also handle inspected ICMP traffic and ALL IPv6 traffic.

Check out:
http://www.scribd.com/doc/28698783/FWSM-Architecture (page 23)

The actual CPU, where layer 7 is performed and access-list compilation/optimization is done.
High CPU on FWSM does not cause performance degrade for non inspected traffic, in most cases anyway.

If you're checking for actual performance problems introduced by FWSM, please do this one test before making any assumptions.

Connect two gig capable laptop to two different line cards (gig capable obviously), put the laptops into two different VRFs and vlans, best if they are not advertised outside the chassis.
Add the vlans to FWSM apply an access-group for each vlan interface on the blade (it can be "permit ip any any") and create statics if nat-control is enabled.
Test of IPerf with TCP, single transfer, I've done some time ago was giving me just above 800 Mbit/s for outbound connections, around 650Mbit/s for inbound connections - if you want to have some baseline.

This method is of course not perfect, you could most likely tweak the physical and logical infrastructure to get better numbers, but my idea was to perform a test similar to real life.

Things to remember:
- make sure that the connection is not inspected if you want decent performance.
-  If this is TCP and you're getting poor performance consider enabling sysoption np completion-unit, this magic option is invoking special processing created to address scenarios in which FWSM was known to introduce out of order packets for TCP streams.
Please note that completion unit will not take care of out of order packets introduced elsewhere and it only work for traffic handled by first two chips.
- Get packet capture. Do not trust the FWSM capture, do local SPAN. FWSM capture is buggy, it's a bit more decent nowadays, but still is not to be relied on. Cisco TAC usually relies on FWSM capture functionality, but will ask you to do SPAN in some cases.
Please note that captures have to be done simultaneously.
What are we looking for in the captures? Out of order or dropped packets (to some extent you can trust dissector built into wireshark for this task), packets dropped (seen in one capture, but not the other), delay introduced by FWSM for packets passing through.
- FWSM does not play well with SACK option, check headers!
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCeb16752
In newer FWSM versions you can clear SACK option in TCP via sysopt!
http://www.cisco.com/en/US/docs/security/fwsm/fwsm40/command/reference/s8.html#wp2761175
- Things that can  improve performance and/or analysis: disable sequence randomization, state bypass (untested)

So you have a blade and you're getting packet drop and poor performance, where should you look:
- show np block (hardware buffer counters) - if they are non-zero and increasing it's bad. You're most likely running into hardware limitation of the FWSM. Consider running active/active failover to split the traffic into two physical devices, doing traffic shaping towards the FWSM on the chassis or bypass the FWSM completely for some traffic.
- show np all stats | i RTL  and show np all stats | i RL will show you if the packets are dropped because of software rate limiting mechanisms built into network processors.
- show np 3 stats can show if some redundant or extensive traffic is crossing the FWSM.
Good counters to start,
a) check if you're not being by one type of traffic
-------------
  TCP Fixups        
  UDP Fixups          
  ICMP Fixups        

--------------
those will show you how many connection "fixups" were done. 

b)  Everything starting from "Discard Statistics" is usually interesting if you're looking for packets dropped.

c) Take care of "Flow Control: Rate Limit Statistics" - packets software rate limited by the chip.

d) ARP miss indications - are some hosts on your network sending traffic to hosts which do not exist?

TAC and Cisco System Engineers can tell you what's going on and what some of the counters mean. Believe me it's NOT intuitive.

While troubleshooting FWSM performance ALWAYS start with checking the switching path.
CAM entries should point either to PoXYZ or trunk between two chassis', ARP entries for active unit should bear the MAC address of primary unit.

edited on 21/Jun/2010 - added bug for SACK.
edited on 13/Aug/2010 - clarifications, more about NP3.