Friday, September 24, 2010

GET VPN with a VRF twist.

I couldn't find this properly documented with an example so here goes.
 
Since GET was designed to secure large private networks, like MPLS clouds,  it is understandable that some VRF features would soon make it to this world.

Cisco documents it here:
http://www.cisco.com/en/US/docs/ios/sec_secure_connectivity/configuration/guide/sec_encrypt_trns_vpn_ps10591_TSD_Products_Configuration_Guide_Chapter.html#wp1235754

You can have crypto map applied on a particular interface in VRF and still reach your KS via global VRF.

KS is completely oblivious to VRF configuration, so it will not be shown here.

GM configuration (relevant parts)
----------------------------
crypto isakmp policy 10
 authentication pre-share
crypto isakmp key cisco address 0.0.0.0 0.0.0.0 no-xauth
!
!
crypto gdoi group GET
 identity number 1111
 server address ipv4 172.16.1.1
 client registration interface Loopback0
!
!
crypto map MAP 10 gdoi
 set group GET
-------------

interfaces' config:


---------
interface Loopback0
 ip address 172.16.1.2 255.255.255.255
!
interface Loopback1
 ip vrf forwarding RED
 ip address 172.16.1.2 255.255.255.255
!
interface FastEthernet0/0
 ip address 10.0.0.2 255.255.255.0
!
interface FastEthernet0/0.10
 encapsulation dot1Q 10
 ip vrf forwarding RED
 ip address 10.0.0.2 255.255.255.0
 crypto map MAP
-----------------

OSPF:
------------
GM1-VRF#sh run | s router
router ospf 100 vrf RED
 network 10.0.0.0 0.0.0.255 area 0
 network 172.16.1.0 0.0.0.255 area 0
router ospf 1001
 network 10.0.0.0 0.0.0.255 area 0
 network 172.16.1.0 0.0.0.255 area 0
------------

Result:
----------------
GM1-VRF#sh crypto gdoi
GROUP INFORMATION

    Group Name               : GET
    Group Identity           : 1111
    Rekeys received          : 0
    IPSec SA Direction       : Both

     Group Server list       : 172.16.1.1

    Group member             : 172.16.1.2       vrf: None
       Registration status   : Registered
       Registered with       : 172.16.1.1
       Re-registers in       : 2568 sec
       Succeeded registration: 1
       Attempted registration: 2
       Last rekey from       : 0.0.0.0
       Last rekey seq num    : 0
       Unicast rekey received: 0
       Rekey ACKs sent       : 0
       Rekey Received        : never
       allowable rekey cipher: any
       allowable rekey hash  : any
       allowable transformtag: any ESP

    Rekeys cumulative
       Total received        : 0
       After latest register : 0
       Rekey Acks sents      : 0

 ACL Downloaded From KS 172.16.1.1:
   access-list  deny ospf any any
   access-list  permit ip any any

KEK POLICY:
    Rekey Transport Type     : Unicast
    Lifetime (secs)          : 86400
    Encrypt Algorithm        : 3DES
    Key Size                 : 192
    Sig Hash Algorithm       : HMAC_AUTH_SHA
    Sig Key Length (bits)    : 1024

TEK POLICY for the current KS-Policy ACEs Downloaded:
  FastEthernet0/0.10:
    IPsec SA:
        spi: 0x730377E0(1929607136)
        transform: esp-3des esp-sha-hmac
        sa timing:remaining key lifetime (sec): (2687)
        Anti-Replay :  Disabled

GM1-VRF#ping vrf RED 172.16.1.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.1.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/8/12 ms
---------------

Thursday, September 16, 2010

IOS - Lan-to-Lan with two dynamic peers. (Both can initiate)

Original credit goes to Zsolt Horvath from Cisco TAC.

Goal:
Establish L2L tunnel between two hosts with dynamic IP. In this case both are registered to a dynamic DNS service (dyndns, no-ip or others). 

Problem.
IOS does not do name resolution when it's IKE responder. Documented here:
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsc92917

Usual resolution:
Use "dynamic" keyword on one peer side and dynamic crypto map on other.
Resulting in only one peer being able to initiate.

A solution when both peers are able to initiate:
This will require having both static and dynamic crypto maps, but with dynamic being first (to account for responding) and static crypto map later (to account for initiating).

Configuration in a nutshell:
R1:
crypto dynamic-map DynamicMAP 10
  set transform TRANSFORM
crypto map StaticMAP 100 ipsec-isakmp dynamic DynamicMAP
crypto map StaticMAP 110 ipsec-isakmp
 set peer R2.provider.tld dynamic
 set transform-set TRANSFORM
 match address 150 
 
R2: 
crypto dynamic-map DynamicMAP 10
  set transform TRANSFORM
crypto map StaticMAP 100 ipsec-isakmp dynamic DynamicMAP
crypto map StaticMAP 110 ipsec-isakmp
 set peer R1.provider.tld dynamic
 set transform-set TRANSFORM
match address 150


Please note:
Note1: You need to have DNS resolution enabled on routers.
Note2: Remote access IPsec VPN should also work.
Note3: While in theory, isakmp key and identity with hostname should work. This has only been tested with identity address and a wildcard isakmp key. 

Sunday, July 25, 2010

IKEv2 - Basic Lan-to-Lan tunnel with crypto maps and pre shared key

Here's a very easy example for site-to-site tunnel using IKEv2 and crypto maps.
In fact it's very easy to "upgrade" your existing L2L tunnel to use IKEv2. You'll notice that the trick is to apply ike version 2 profile to existing crypto map.

Topology:
192.168.1.0/24 - Site1 router --(L2 connection)--- Site2 router - 192.168.2.0/24
Both site1 and site2 routers use 15.1(2)T IOS.

So how does the configuration work.
First of all you define IKEv2 proposals.
----------
crypto ikev2 proposal PROPOSAL_1
 encryption aes-cbc-128 3des
 integrity sha256 md5
 group 5 2
----------

You apply this(or those) proposals to a policy.
-----------
crypto ikev2 policy POLICY_1
 proposal PROPOSAL_1
-----------

Define a keyring (this comes from site2 router, site_1's public IP is 10.0.0.1)
-----------
crypto ikev2 keyring KEYRING_1
 peer SITE_1
  address 10.0.0.1
  pre-shared-key cisco
-----------

Following this you define a profile:
-----------
crypto ikev2 profile PROFILE_1
 match identity remote address 10.0.0.1 255.255.255.255
 authentication local pre-share
 authentication remote pre-share
 keyring KEYRING_1
-----------

Last thing to do is to apply the profile to you crypto map.
-----------
crypto map MAP 10 ipsec-isakmp
 set peer 10.0.0.1
 set transform-set TRANSFORM_1
 set ikev2-profile PROFILE_1
 match address VPN
-------------

Result?
########################################
Site2#deb cry ikev2 event
IKEV2 event debugging is on
Site2#ping 192.168.1.1 sou e0/1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.2.1

*Jul 26 10:46:07.931: IKEv2:% Getting preshared key from profile keyring KEYRING_1
*Jul 26 10:46:07.931: IKEv2:% Matched peer block 'SITE_1'
*Jul 26 10:46:07.931: IKEv2:Found Policy POLICY_1
*Jul 26 10:46:07.931: IKEv2:(1): Getting configured policies
*Jul 26 10:46:07.931: IKEv2:(1): Setting configured policies
*Jul 26 10:46:07.931: IKEv2:(1): Computing DH public key
*Jul 26 10:46:07.931: IKEv2:(1):
*Jul 26 10:46:07.931: IKEv2:(1): Sending initial message
*Jul 26 10:46:07.931: IKEv2:  IKE Proposal: 1, SPI size: 0 (initial negotiation),
Num. transforms: 8
   AES-CBC   3DES   SHA256   MD5   SHA256   MD596   DH_GROUP_1536_MODP/Group 5   DH_GROUP_1024_MODP/Group 2
*Jul 26 10:46:07.931:
*Jul 26 10:46:07.931: IKEv2:(1): Checking if request will fit in peer window
*Jul 26 10:46:07.931: IKEv2:Tx [L 10.0.0.2:500/R 10.0.0.1:500/VRF i0:f0] m_id: 0x0

*Jul 26 10:46:07.931: IKEv2:HDR[i:8984BC53859B1136 - r: 0000000000000000]
 SA KE N NOTIFY(NAT_DETECTION_SOURCE_IP) NOTIFY(NAT_DETECTION_DESTINATION_IP)
*Jul 26 10:46:07.931:
*Jul 26 10:46:07.931: IKEv2:(1): Insert SA
*Jul 26 10:46:08.007: IKEv2:Rx [L 10.0.0.2:500/R 10.0.0.1:500/VRF i0:f0] m_id: 0x0

*Jul 26 10:46:08.007: IKEv2:HDR[i:8984BC53859B1136 - r: 153FE7D81F5F453A]

*Jul 26 10:46:08.007:  SA KE N NOTIFY(NAT_DETECTION_SOURCE_IP) NOTIFY(NAT_DETECTION_DESTINATION_IP)
*Jul 26 10:46:08.007:
*Jul 26 10:46:08.007: IKEv2:(1): Processing initial message
*Jul 26 10:46:08.007: IKEv2:(1): Verify SA init message
*Jul 26 10:46:08.007: IKEv2:(1): Processing initial message
*Jul 26 10:46:08.007: IKEv2:(1): Process NAT discovery notify
*Jul 26 10:46:08.007: IKEv2:(1): Check NAT discovery
*Jul 26 10:46:08.007: IKEv2:(1): Computing DH secret key
*Jul 26 10:46:08.019: IKEv2:(1):
*Jul 26 10:46:08.019: IKEv2:(1): Generate skeyid
*Jul 26 10:46:08.019: IKEv2:(1): Complete SA init exchange
*Jul 26 10:46:08.019: IKEv2:(1): Check for EAP exchange
*Jul 26 10:46:08.019: IKEv2:(1): Generate my authentication data
*Jul 26 10:46:08.019: IKEv2:(1): Use preshared key for id 10.0.0.2, key len 5
*Jul 26 10:46:08.019: IKEv2:(1): Get my authentication method
*Jul 26 10:46:08.019: IKEv2:(1): Check for EAP exchange
*Jul 26 10:46:08.019: IKEv2:(1): Sending auth message
*Jul 26 10:46:08.019: IKEv2:  ESP Proposal: 1, SPI size: 4 (IPSec negotiation),
Num. transforms: 3
   AES-CBC   SHA96
*Jul 26 10:46:08.019:
*Jul 26 10:46:08.019: IKEv2:(1): Building packet for encryption; contents are:  VID IDi AUTH SA TSi TSr NOTIFY(INITIAL_CONTACT) NOTIFY(SET_WINDOW_SIZE) NOTIFY(ESP_TFC_NO_SUPPORT) NOTIFY(NON_FIRST_FRAGS)
*Jul 26 10:46:08.019:
*Jul 26 10:46:08.019: IKEv2:(1): Checking if request will fit in peer window
*Jul 26 10:46:08.019: IKEv2:Tx [L 10.0.0.2:500/R 10.0.0.1:500/VRF i0:f0] m_id: 0x1

*Jul 26 10:46:08.019: IKEv2:HDR[i:8984BC53859B1136 - r: 153FE7D81F5F453A]
 ENCR
*Jul 26 10:46:08.019:
*Jul 26 10:46:08.071: IKEv2:Rx [L 10.0.0.2:500/R 10.0.0.1:500/VRF i0:f0] m_id: 0x1

*Jul 26 10:46:08.071: IKEv2:HDR[i:8984BC53859B1136 - r: 153FE7D81F5F453A]

*Jul 26 10:46:08.071:  VID IDr AUTH SA TSi TSr NOTIFY(SET_WINDOW_SIZE) NOTIFY(ESP_TFC_NO_SUPPORT) NOTIFY(NON_FIRST_FRAGS)
*Jul 26 10:46:08.071:
*Jul 26 10:46:08.071: IKEv2:(1): Process auth response notify
*Jul 26 10:46:08.071: IKEv2:(1): Peer is a Cisco device  Platform:IOS Capabilities: Granite

*Jul 26 10:46:08.071: IKEv2:(1): Getting configured policies
*Jul 26 10:46:08.071: IKEv2:Found Policy POLICY_1
*Jul 26 10:46:08.071: IKEv2:(1): Verify peer's policy
*Jul 26 10:46:08.071: IKEv2:(1): Get peer authentication method
*Jul 26 10:46:08.071: IKEv2:(1): Get peer's preshared key for 10.0.0.1
*Jul 26 10:46:08.071: IKEv2:(1): Verify authentication data
*Jul 26 10:46:08.071: IKEv2:(1): Use preshared key for id 10.0.0.1, key len 5
*Jul 26 10:46:08.071: IKEv2:(1): Check for EAP exchange
*Jul 26 10:46:08.071: IKEv2:(1): Processing auth message
*Jul 26 10:46:08.071: IKEv2:(1): Closing the PKI session
*Jul 26 10:46:08.071: IKEv2:(1): SA created; inserting SA into database
*Jul 26 10:46:08.071: IKEv2:(1): Load IPSEC key material
*Jul 26 10:46:08.071: IKEv2:(1): Checking for duplicate SA.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 8/12/20 ms
Site2#sh cry ikev2 sa det
Tunnel-id Local                 Remote                fvrf/ivrf            Status
1         10.0.0.2/500          10.0.0.1/500          (none)/(none)        READY
      Encr: AES-CBC, keysize: 128, Hash: SHA256, DH Grp:5, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/163 sec
      CE id: 1004, Session-id: 4
      Status Description: Negotiation done
      Local spi: 8984BC53859B1136       Remote spi: 153FE7D81F5F453A
      Local id: 10.0.0.2
      Remote id: 10.0.0.1
      Local req msg id:  2              Remote req msg id:  0
      Local next msg id: 2              Remote next msg id: 0
      Local req queued:  2              Remote req queued:  0
      Local window:      5              Remote window:      5
      DPD configured for 0 seconds, retry 0
      NAT-T is not detected

########################################

While on the responder:
########################################
Site1#
*Jul 26 10:46:07.839: IKEv2:Rx [L 10.0.0.1:500/R 10.0.0.2:500/VRF i0:f0] m_id: 0x0

*Jul 26 10:46:07.839: IKEv2:HDR[i:8984BC53859B1136 - r: 0000000000000000]

*Jul 26 10:46:07.839:  SA KE N NOTIFY(NAT_DETECTION_SOURCE_IP) NOTIFY(NAT_DETECTION_DESTINATION_IP)
*Jul 26 10:46:07.839:
*Jul 26 10:46:07.839: IKEv2:Verify SA init message
*Jul 26 10:46:07.839: IKEv2:Insert SA
*Jul 26 10:46:07.839: IKEv2:(1): Getting configured policies
*Jul 26 10:46:07.839: IKEv2:Found Policy POLICY_1
*Jul 26 10:46:07.839: IKEv2:(1): Processing initial message
*Jul 26 10:46:07.839: IKEv2:(1): Process NAT discovery notify
*Jul 26 10:46:07.839: IKEv2:(1): Setting configured policies
*Jul 26 10:46:07.839: IKEv2:Failed to retrieve Certificate Issuer list
*Jul 26 10:46:07.839: IKEv2:(1): Computing DH public key
*Jul 26 10:46:07.839: IKEv2:(1):
*Jul 26 10:46:07.839: IKEv2:(1): Computing DH secret key
*Jul 26 10:46:07.907: IKEv2:(1):
*Jul 26 10:46:07.907: IKEv2:(1): Generate skeyid
*Jul 26 10:46:07.907: IKEv2:(1): Sending initial message
*Jul 26 10:46:07.907: IKEv2:  IKE Proposal: 1, SPI size: 0 (initial negotiation),
Num. transforms: 4
   AES-CBC   SHA256   SHA256   DH_GROUP_1536_MODP/Group 5
*Jul 26 10:46:07.907:
*Jul 26 10:46:07.907: IKEv2:Failed to retrieve Certificate Issuer list
*Jul 26 10:46:07.907: IKEv2:Tx [L 10.0.0.1:500/R 10.0.0.2:500/VRF i0:f0] m_id: 0x0

*Jul 26 10:46:07.907: IKEv2:HDR[i:8984BC53859B1136 - r: 153FE7D81F5F453A]
 SA KE N NOTIFY(NAT_DETECTION_SOURCE_IP) NOTIFY(NAT_DETECTION_DESTINATION_IP)
*Jul 26 10:46:07.907:
*Jul 26 10:46:07.907: IKEv2:(1): Complete SA init exchange
*Jul 26 10:46:07.907: IKEv2:(1): Starting timer to wait for auth message
*Jul 26 10:46:07.975: IKEv2:Rx [L 10.0.0.1:500/R 10.0.0.2:500/VRF i0:f0] m_id: 0x1

*Jul 26 10:46:07.975: IKEv2:HDR[i:8984BC53859B1136 - r: 153FE7D81F5F453A]

*Jul 26 10:46:07.975:  VID IDi AUTH SA TSi TSr NOTIFY(INITIAL_CONTACT) NOTIFY(SET_WINDOW_SIZE) NOTIFY(ESP_TFC_NO_SUPPORT) NOTIFY(NON_FIRST_FRAGS)
*Jul 26 10:46:07.975:
*Jul 26 10:46:07.975: IKEv2:(1): Stopping timer to wait for auth message
*Jul 26 10:46:07.975: IKEv2:(1): Check NAT discovery
*Jul 26 10:46:07.975: IKEv2:(1): Recieved valid parameteres in process id
*Jul 26 10:46:07.975: IKEv2:(1): Peer is a Cisco device  Platform:IOS Capabilities: Granite

*Jul 26 10:46:07.975: IKEv2:(1): Getting configured policies
*Jul 26 10:46:07.975: IKEv2:found matching IKEv2 profile 'PROFILE_1'
*Jul 26 10:46:07.975: ISAKMP:(0):: peer matches PROFILE_1 profile
*Jul 26 10:46:07.975: IKEv2:% Getting preshared key from profile keyring KEYRING_1
*Jul 26 10:46:07.975: IKEv2:% Matched peer block 'SITE_2'
*Jul 26 10:46:07.975: IKEv2:Found Policy POLICY_1
*Jul 26 10:46:07.975: IKEv2:(1): Setting configured policies
*Jul 26 10:46:07.975: IKEv2:(1): Verify peer's policy
*Jul 26 10:46:07.975: IKEv2:(1): Get peer authentication method
*Jul 26 10:46:07.975: IKEv2:(1): Get peer's preshared key for 10.0.0.2
*Jul 26 10:46:07.975: IKEv2:(1): Verify authentication data
*Jul 26 10:46:07.975: IKEv2:(1): Use preshared key for id 10.0.0.2, key len 5
*Jul 26 10:46:07.975: IKEv2:(1): Processing initial contact
*Jul 26 10:46:07.975: IKEv2:(1): Processing auth message
*Jul 26 10:46:07.975: IKEv2:(1): Get my authentication method
*Jul 26 10:46:07.975: IKEv2:(1): Get peer's preshared key for 10.0.0.2
*Jul 26 10:46:07.975: IKEv2:(1): Generate my authentication data
*Jul 26 10:46:07.975: IKEv2:(1): Use preshared key for id 10.0.0.1, key len 5
*Jul 26 10:46:07.975: IKEv2:(1): Get my authentication method
*Jul 26 10:46:07.975: IKEv2:(1): Sending auth message
*Jul 26 10:46:07.975: IKEv2:  ESP Proposal: 1, SPI size: 4 (IPSec negotiation),
Num. transforms: 3
   AES-CBC   SHA96
*Jul 26 10:46:07.975:
*Jul 26 10:46:07.975: IKEv2:(1): Building packet for encryption; contents are:  VID IDr AUTH SA TSi TSr NOTIFY(SET_WINDOW_SIZE) NOTIFY(ESP_TFC_NO_SUPPORT) NOTIFY(NON_FIRST_FRAGS)
*Jul 26 10:46:07.975:
*Jul 26 10:46:07.975: IKEv2:
Site1#Tx [L 10.0.0.1:500/R 10.0.0.2:500/VRF i0:f0] m_id: 0x1

*Jul 26 10:46:07.975: IKEv2:HDR[i:8984BC53859B1136 - r: 153FE7D81F5F453A]
 ENCR
*Jul 26 10:46:07.975:
*Jul 26 10:46:07.975: IKEv2:(1): Closing the PKI session
*Jul 26 10:46:07.975: IKEv2:(1): SA created; inserting SA into database
*Jul 26 10:46:07.975: IKEv2:(1): Load IPSEC key material
*Jul 26 10:46:07.975: IKEv2:(1): Checking for duplicate SA
*Jul 26 10:46:07.975: IKEv2:(1): Starting timer to delete negotiation context
Site1#sh cry ikev2 sa det
Tunnel-id Local                 Remote                fvrf/ivrf            Status
1         10.0.0.1/500          10.0.0.2/500          (none)/(none)        READY
      Encr: AES-CBC, keysize: 128, Hash: SHA256, DH Grp:5, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/132 sec
      CE id: 1004, Session-id: 4
      Status Description: Negotiation done
      Local spi: 153FE7D81F5F453A       Remote spi: 8984BC53859B1136
      Local id: 10.0.0.1
      Remote id: 10.0.0.2
      Local req msg id:  0              Remote req msg id:  2
      Local next msg id: 0              Remote next msg id: 2
      Local req queued:  0              Remote req queued:  2
      Local window:      5              Remote window:      5
      DPD configured for 0 seconds, retry 0
      NAT-T is not detected

######################################## 



Configuration guide is located here.
http://www.cisco.com/en/US/docs/ios/sec_secure_connectivity/configuration/guide/sec_cfg_ikev2_ps10592_TSD_Products_Configuration_Guide_Chapter.html
Related RFC:
http://www.ietf.org/rfc/rfc4306.txt

Wednesday, July 21, 2010

ASA 8.3 part 2 - webvpn

Stuck at the airport - flight delayed. So I might as well write a bit more about my findings on ASA 8.3.

This time I'm going to talk about webvpn.

Frankly speaking webvpn is the only reason why you should consider 8.3.

Here are my highlights:
- Finally! you can smart tunnel your home page (without any hacks and tricks), downside to this - no webvpn toolbar, java needed, only windows and mac support for smart tunnel.
- A very extended OS/Browser support list, check out:
http://www.cisco.com/en/US/docs/security/asa/asa83/asdm63/configuration_guide/vpn_web.html#wp1052888
- Smart tunnel notification (now this might have been there before, but I noticed it just now)
When you enable smart tunneling (on both mac and windows) a notification icon can appear:
http://www.cisco.com/en/US/docs/security/asa/asa83/asdm63/configuration_guide/vpn_web.html#wp1055586
- Auto signon for smart tunnel:
http://www.cisco.com/en/US/docs/security/asa/asa83/asdm63/configuration_guide/vpn_web.html#wp1055462
The functionality is there but please be aware it has very limited scope.
- You'll be also (pleasantly?) surprised by the new way one can customize their webvpn portal.


- A very "important" change in webvpn config guide regarding APCF:
"Cisco TAC may help with APCF to address specific rendering issues if the smart tunneling feature is not working or cannot be used."

Things still to be addressed, from top of my head - webvpn cache "memory leak":
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtb68311


And here's the complete config guide:
http://www.cisco.com/en/US/docs/security/asa/asa83/asdm63/configuration_guide/vpn_web.html

ASA 8.3 part 1 - NAT

edit:
In the meantime Cisco published ASA 8.3 migration guide:
http://www.cisco.com/en/US/docs/security/asa/asa83/upgrading/migrating.html
Please refer to it for all NAT migration questions, otherwise do go on, there's some info compiled below :-)


Please be aware of following bugs (once I stumbled upon):
- NAT RPF failure in ASA 8.3
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCth72642
- Access-list migration based on NAT config.
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtf57830
- "Unidirectional" keyword added:
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCti36048
If any of the above links doesn't work go  to bugtoolkit directly:
http://www.cisco.com/cgi-bin/Support/Bugtool/launch_bugtool.pl



Recently I found quite a few discussion on supportforums regarding ASA 8.3 and related changes.

While majority of changes are good and much anticipated (ACL, webvpn - which I will try to give highlights at some other day) some changes are confusing at best especially for people who already knew NAT that came from PIX - or everything on ASA prior to 8.3.1 release.

In this post I do not intend to re-write configuration guide, it's there and it has ALL the information you might need, I intend to show how you CAN (and believe me there are multiple ways) how one can translate old PIX/ASA NAT CLI to ASA 8.3. That being said don't take my words for granted and have a look at configuration guide:

http://www.cisco.com/en/US/docs/security/asa/asa83/configuration/guide/nat_overview.html



Let's take a look at changes having in mind old order of NAT operation.
1) Nat exemption (this is NOT the only way to do it, but one that will give you least trouble)

Regarding this one ... you might be careful to check  CSCth72642 and some other bugs.
You might need to add reverse rule! 
ie. nat (out,in) sour  sta REMOTE REMOTE dest sta LOCAL LOCAL

Old Syntax:
access-list NAT0 permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0
nat (inside_interface) 0 access-list NAT0

New syntax:
object network MY_NETWORK
subnet 192.168.1.0 255.255.255.0

object network FOREIGN_NETWORK
subnet 192.168.2.0 255.255.255.0
nat (inside_interface,any) source static MY_NETWORK MY_NETWORK destination static FOREIGN_NETWORK FOREIGN_NETWORK





Another example, we're assuming:
- "inside" is the inside interface
- 192.168.0.0/24 is local subnet
- 172.16.0.0/24 is subnet for our remote location (or remote access VPN clients - SSL/IPsec ..)



Old syntax:
access-list NAT0 permit ip 192.168.0.0 255.255.255.0 172.16.0.0 255.255.255.0
nat (inside) 0 access-l NAT0


New syntac:
object network HOME
subnet 192.168.0.0 255.255.0.0


object network REMOTE
subnet 172.16.0.0 255.255.0.0


nat (inside,outside) source static HOME HOME destination static REMOTE REMOTE
It's not exempting anything from NAT but it's making sure there's a static translation when HOME is going REMOTE , for both HOME and REMOTE.

2) Static

- NAT
Old syntax
static (inside_interface,outside_interface) 192.0.2.1 192.168.1.2 netmask 255.255.255.255

New syntax:
object network INTERNAL_IP1
host 192.168.1.2
nat (inside_interface,outside_interface) static 192.0.2.1

- PAT
Old syntax:
static (inside_interface,outside_interface) udp 192.0.2.3 1611 192.168.1.3 161 netmask 255.255.255.255

New syntax:
object network INTERNAL_IP2
host 192.168.1.3
nat (inside_interface,outside_interface) static 192.0.2.3 service udp 161 1611

3. NAT
Old syntax:
nat (inside_interface) 1 192.168.1.0 255.255.255.0
global (outside_interface) 1 192.0.2.254

New syntax: object network MY_SUBNET
subnet 192.168.1.0 255.255.255.0
nat (inside_interface,outside_interface) dynamic 192.0.2.254
However consider, instead, adding a rule similar to this:
nat (any,outside) after-auto source dynamic any interface
Translate everything ELSE to interface via PAT.

Here's order of operation for NAT on 8.3
http://www.cisco.com/en/US/docs/security/asa/asa83/configuration/guide/nat_overview.html#wp1118157
if you check it's NOT that much different comparing to older ones.

So what's the big benefit of 8.3 NAT:
- "any" keyword in NAT (any source, any interface )
- A few new thing are available now man-to-few translations ... possibly nothing ground breaking.

Disadvantages:
- not readable via CLI - everything is an object. ASDM oriented. ("show nat" command DOES help A LOT)
- expanded ("exploding") configuration listing.
- lack of ACL support in NAT rules.
- no additional option on NAT rules (norandom, maximum connection count for example), need to use MPF.

There's of course more to NAT on ASA 8.3 in comparison - and I can post more examples if someone is interested.


Disclaimer:
The above worked on my ASA connected to the lab without REQUIRED memory upgrade.


edit:
18th Aug 2010 - Added some ASA 8.3 NAT bugs and link to Cisco's ASA migration.

Saturday, June 05, 2010

PPPoE - config example.

Thanks to Oliver P. from Cisco TAC for this one.

Yet another configuration I keep to quickly test PPPoE or particular features over PPPoE on Cisco routers.

Server
-----------
aaa new-model
!
aaa authentication ppp default local
!
virtual-template 1 pre-clone 20
!
user cisco password cisco
!
bba-group pppoe global
 virtual-template 1
!
interface faX/Y
 pppoe enable
!
interface Loopback0
 ip address 6.6.66.66 255.255.255.255
!
interface Virtual-Template1
 ip unnumbered Loopback0
 peer default ip address pool mypool
 no keepalive
 ppp authentication pap
!
ip local pool mypool 66.66.66.66 66.66.66.166
-----------
Client
-----------
bba-group pppoe global

int  faX/Y
 description PPPOE interface
 no ip address
 duplex auto
 speed auto
 pppoe enable group global
 pppoe-client dial-pool-number 1
!
!
interface Dialer1
 mtu 1454
 ip address negotiated
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 encapsulation ppp
 dialer pool 1
 dialer-group 1
 no keepalive
 ppp authentication pap callin
 ppp pap sent-username cisco password 0 cisco
 ppp ipcp dns request accept
!
!
dialer-list 1 protocol ip permit
!
ip route 0.0.0.0 0.0.0.0 dialer1
-----------

L2tp over IPsec - testing configuration.

This is a configuration based on an actual customer's setup.
I keep it whenever I need to implement L2tp over IPsec on Cisco routers.

Spoke config 
--------------
l2tp-class l2tpclass1
!
pseudowire-class pwclass1
 encapsulation l2tpv2
 ip local interface FastEthernet0/0
!         
interface Virtual-PPP10
 ip address negotiated
 no peer neighbor-route
 no cdp enable
 ppp authentication pap callin
 ppp pap sent-username cisco password 0 cisco
 pseudowire 1.1.1.1 31 pw-class pwclass1
!         
dialer-list 31 protocol ip permit 

ip route 10.0.0.0 255.0.0.0 Virtual-PPP10
--------------
 
HUB config 
--------------
vpdn enable
vpdn tunnel accounting network default
vpdn session accounting network default
!
vpdn-group DailIn-L2TP/IPsec
! Default L2TP VPDN group
 accept-dialin
  protocol l2tp
  virtual-template 100
 lcp renegotiation on-mismatch
 no l2tp tunnel authentication
 l2tp tunnel timeout setup 60
 ip pmtu
!
!
username cisco privilege 15 password cisco
!
crypto keyring RING
  pre-shared-key address 0.0.0.0 0.0.0.0 key cisco
!
crypto isakmp policy 10
 encr aes 256
 authentication pre-share
 group 5
crypto isakmp key cisco address 0.0.0.0 0.0.0.0 no-xauth
crypto isakmp keepalive 30 5
crypto isakmp aggressive-mode disable
crypto isakmp profile L2TP
   description L2TP/IPsec HW CLients Profile
   keyring RING
   match identity address 0.0.0.0
!
!
crypto ipsec transform-set ICW esp-aes 256 esp-sha-hmac
 mode transport
crypto dynamic-map CryDynMapICW 10
 set nat demux
 set security-association lifetime kilobytes 3145728
 set transform-set ICW
 set pfs group5
 set isakmp-profile L2TP
 reverse-route
!
!
crypto map CryMapICW local-address Ethernet0/0
crypto map CryMapICW 10 ipsec-isakmp dynamic CryDynMapICW
!
!
interface Loopback100
 description L2TP Address
 ip address 10.255.0.1 255.255.255.255
!
interface Ethernet0/0
 ip address 1.1.1.1 255.255.255.0
 crypto map CryMapICW
!
interface Virtual-Template100
 ip unnumbered Loopback100
 peer default ip address pool L2TPOOL
 ppp chap hostname cisco
 ppp chap password 0 cisco
!
ip local pool L2TPOOL 5.5.5.1 5.5.5.254
------------------




Wednesday, June 02, 2010

Troubleshooting VoIP issues over ASA/PIX/FWSM appliances.

Recently I've been again put in a situation where I had to troubleshoot voice over IP issues via firewall appliances by Cisco.

Here's a brief summary of types of issues:
- No Voice (including one way voice)
- Delay
- No communication
- Features not working (MoH etc).

Now I will be the first to admit - I am voice-over-IP-agnostic, I've touched CCM or CME once or twice in my life. I'm aware that h323, sip and skinny exist.

From my perspective only two things exist in voice over IP.
- signaling
- actual payload (RTP).

So what you should know about how ASA/FWSM interacts with voice traffic.
0. Before going into voice troubleshooting check routing. In many cases a few pings can save a few hours of "missing voice" or "broken signaling"
1. First of all you most likely want to inspect your signaling traffic to/from the internet
a) Be aware that some implementations do not use default ports:
Prime example - SIP - default udp/5060 but known also to use tcp/5060 and udp/5061 (yes for signaling)
b) Be aware that if you're running voice traffic over VPN tunnel you most likely do not need to inspect traffic (everything normally should be permitted). Inspecting over VPN is waste of time (delay) and can account for some one way voice issues.
2. What does the actual inspection do:
a) Dynamically opens pinholes in your security policy to allow incoming traffic (or outgoing if relevant)
b) Pre-allocates connections and translations for traffic.
Thing to take into account here
!!!!!!
If a given IP/phone is not mentioned in signaling the traffic will NOT be allowed in. This comes over and over with music on hold from secondary CCMs. Since none of the parties actually mentions that this particular CCM will be sending MoH the traffic is neatly being dropped because of security policy.
!!!!!!!


3. ASA/FWSM inspects AND changes payload of signaling packets - and that's a good thing.
Firewalls will change internal IP address to public ones according to translation rules in place.
Be mindful when changing inspection or translation rules. "clear xlate ..." or "clear conn ..." has a big chance of recovering if something went wrong after reconfiguration either in inspection rules or NAT.

4. RSVP packets. They usually contain IP options. ASA 8.0 will drop them. Check out ip-option inspection in ASA 8.2.

5. Signaling works but voice does not go properly (dead silence and one way voice issues).
This is most likely caused by a problem with inspection or some additional policies.
Ie. Firewall should automatically open pin holes in security policy for both way RTP traffic.

Check
------
show conn detail add x.x.x.x
show x det | i x.x.x.x
------
Where x.x.x.x is the source or destination of either signaling or voice.

You should have also enabled logging on informational level and monitor logs for entries mentioning IP address of either source of destination. 

Check also those (pick ones interesting)
h323
--------
----------
SIP
-----
show sip
------
SCCP
-----
show skinny
------
I would not advise to get debugs at first - better leave those for interaction with TAC.
FYI - If it comes to TAC intervention, TAC will request "inside" and "outside" interfaces capture and debugs taken AT THE SAME TIME.

What goes without saying - !gathering packet capture helps!
If traffic is NATed and you see inside IP addresses in outside captures ... something went wrong. Maybe something similar to 4.?
Do you see packets coming into the appliance? Do you see them leaving? What has been changed.
FWSM note: Remember that inspected traffic goes straight to CPU!

Good luck.

Friday, April 16, 2010

DVTI, SVTI and routing protocols

edit:
Apparently dynamic routing will be supported with DVTI Cisco is in process of changing documentation to reflect this.


After several days of bugging Cisco SEs and TAC I finally got a clear answer on several questions I had regarding DVTI,SVTI and routing protocols.

It is true that you CAN run routing protocols over DVTI tunnels - however that is NOT recommended OR supported - because it was not tested.

Also according to Cisco SVTI and terminate on DVTI, but again routing protocols is not supported in this case, static routing should be fine though.

I'd like to stress out we're talking about Cisco support for those techs, it still might work but good luck getting support from TAC/SEs.
To quote rfc 1925, section 2 point 3:
"
(3)  With sufficient thrust, pigs fly just fine. However, this is
        not necessarily a good idea. It is hard to be sure where they
        are going to land, and it could be dangerous sitting under them
        as they fly overhead

Tuesday, February 16, 2010

Troubleshooting FWSM performance

FWSM is the firewall blade for Cat 6500 and 7600 routers.
The appliance has been out for a while and has been proven quite a performance bottle neck (AKA pain in the ***) very often. Specifically because of the placement (data centers) and increasing bandwidth demand there (Tengig links, Nexus). There's slowly a newer device looming on the horizon, but it may take several years.

So first of all, what is FWSM? It's a PIX in a chassis with very neat physical architecture.
You should also know that it has separate logical circuit for both connection for management from chassis (session ... command does a telnet on the backplane). There is also a 6 x gigabit etherchannel,
labeled PoXYZ.

There two dedicated proprietary chips (if you inspect the blade itself you will notice it's done by IBM) handling active connections. 

There is a dedicated network processor to perform access-list, route, xlate lookups.
Coincidentally it will also handle inspected ICMP traffic and ALL IPv6 traffic.

Check out:
http://www.scribd.com/doc/28698783/FWSM-Architecture (page 23)

The actual CPU, where layer 7 is performed and access-list compilation/optimization is done.
High CPU on FWSM does not cause performance degrade for non inspected traffic, in most cases anyway.

If you're checking for actual performance problems introduced by FWSM, please do this one test before making any assumptions.

Connect two gig capable laptop to two different line cards (gig capable obviously), put the laptops into two different VRFs and vlans, best if they are not advertised outside the chassis.
Add the vlans to FWSM apply an access-group for each vlan interface on the blade (it can be "permit ip any any") and create statics if nat-control is enabled.
Test of IPerf with TCP, single transfer, I've done some time ago was giving me just above 800 Mbit/s for outbound connections, around 650Mbit/s for inbound connections - if you want to have some baseline.

This method is of course not perfect, you could most likely tweak the physical and logical infrastructure to get better numbers, but my idea was to perform a test similar to real life.

Things to remember:
- make sure that the connection is not inspected if you want decent performance.
-  If this is TCP and you're getting poor performance consider enabling sysoption np completion-unit, this magic option is invoking special processing created to address scenarios in which FWSM was known to introduce out of order packets for TCP streams.
Please note that completion unit will not take care of out of order packets introduced elsewhere and it only work for traffic handled by first two chips.
- Get packet capture. Do not trust the FWSM capture, do local SPAN. FWSM capture is buggy, it's a bit more decent nowadays, but still is not to be relied on. Cisco TAC usually relies on FWSM capture functionality, but will ask you to do SPAN in some cases.
Please note that captures have to be done simultaneously.
What are we looking for in the captures? Out of order or dropped packets (to some extent you can trust dissector built into wireshark for this task), packets dropped (seen in one capture, but not the other), delay introduced by FWSM for packets passing through.
- FWSM does not play well with SACK option, check headers!
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCeb16752
In newer FWSM versions you can clear SACK option in TCP via sysopt!
http://www.cisco.com/en/US/docs/security/fwsm/fwsm40/command/reference/s8.html#wp2761175
- Things that can  improve performance and/or analysis: disable sequence randomization, state bypass (untested)

So you have a blade and you're getting packet drop and poor performance, where should you look:
- show np block (hardware buffer counters) - if they are non-zero and increasing it's bad. You're most likely running into hardware limitation of the FWSM. Consider running active/active failover to split the traffic into two physical devices, doing traffic shaping towards the FWSM on the chassis or bypass the FWSM completely for some traffic.
- show np all stats | i RTL  and show np all stats | i RL will show you if the packets are dropped because of software rate limiting mechanisms built into network processors.
- show np 3 stats can show if some redundant or extensive traffic is crossing the FWSM.
Good counters to start,
a) check if you're not being by one type of traffic
-------------
  TCP Fixups        
  UDP Fixups          
  ICMP Fixups        
--------------
those will show you how many connection "fixups" were done. 
b)  Everything starting from "Discard Statistics" is usually interesting if you're looking for packets dropped.
c) Take care of "Flow Control: Rate Limit Statistics" - packets software rate limited by the chip.
d) ARP miss indications - are some hosts on your network sending traffic to hosts which do not exist?

TAC and Cisco System Engineers can tell you what's going on and what some of the counters mean. Believe me it's NOT intuitive.

While troubleshooting FWSM performance ALWAYS start with checking the switching path.
CAM entries should point either to PoXYZ or trunk between two chassis', ARP entries for active unit should bear the MAC address of primary unit.

edited on 21/Jun/2010 - added bug for SACK.
edited on 13/Aug/2010 - clarifications, more about NP3.

Monday, January 25, 2010

Troubleshooting DMVPN

DMVPN is a great suite of protocols, from time to time something goes wrong though. Here's a few tips on how to troubleshoot it. It will not tell you how to troubleshoot every part but will rather guide you in narrowing the problem down.

In typical DMVPN scenario you will have following "layers" each dependent on all the ones before.
- Physical and IP
- Crypto
- GRE
- NHRP
- Routing protocol

1. Physical and IP - I'm putting those together since they are not really specific to DMVPN but you need to check if they work.
1.1 Check reachability, from spoke to hub by a simple ping or traceroute.
1.2 Typical problem: IPsec not starting to establish.
Do some basic testing - ping from spoke to hub, make sure not firewall on the way is blocking - UDP/500, UDP/4500 - if NAT-T is needed, ESP/AH.
If everything is configured but tunnel is not initiating... Did you configured NHRP network id?

A typical exercise here and at any level will be to verify CEF switching statistics "show cef drop" in old IOSes or "show cef switching statistic feature" on newer ones.

2. Crypto (IPsec), once you know that nothing is blocked and crypto show start establishing.
- Check that you have phase 1 SAs "show crypto isakmp sa det" the state you're looking for usually is QM_IDLE (or no IKE SA if lifetime is very short)
- Check IPsec SAs show crypto ipsec sa - both inbound and outbound SPI should be there and should be mirrored on other side of the tunnel. (Inbound SPI on spoke will be outbound SPI on hub and vice versa)
debug crypto ipsec and debug cry isakmp are your friends.

Two notable mentions here:
2.1 If in debugs you see tunnels establishing properly but they get torn down in few minutes it most likely means that NHRP relation is not establishing.
2.2 Crypto socket - there is a magic being called a crypto socket that is what is binding crypto and nhrp together - you can debug it - debug crypto socket. Problems with crypto socket can cause 2.1 but can be usually mitigated in short term by removing tunnel interface configuration and adding it back again. Many cases, different IOS versions affected, multiple bugs on Cisco side.

There is also a whole subset of problems with crypto accelerator cards that can show themselves here. Verify "show crypto engine accelerator statistic" and "show crypto engine configurtion" or "show crypto eli" - this will show you statistics and which accelerator is currently being used. You generally check for errors.

3. GRE - here's a fun fact, I've never seen a problem with GRE encapsulation or processing. But I would start by monitoring show interface tunnel X for input or output drops.
One problem you may encounter is .... NAT.
3.1 I've seen a scenario on a fairly recent 12.4T software where NAT was done for GRE traffic (no tunnel protection scenario). Check "sh ip nat trans".
3.2 If by any chance you're using "ip nat outside/inside" on tunnel interface, please check if you're not NATing too much.

4. NHRP - remember that even though the spoke has static NHRP mapping and "show ip nhrp brief" will always show you a mapping present (as opposed to the hub) it is the spoke that is initiating NHRP registration by sending registration request.
Useful debugs:
debug nhrp pack
debug nhrp ext 
debug nhrp err
debug nhrp rate
For each NHRP registration request you should see a packet encapsulated into IPsec (show crypto ipsec sa), if it's not the case enable debug from 2.2 and get in touch with Cisco TAC.

A hood value for NHRP holdtime would be around 300 seconds (as opposed to 7200 default).

5. Routing protocol - once you know all the "layers" below there is the RP level that makes it all tick. I've seen a range of problems here, some bugs, some platform specific (ASR hub taking longer to converge comparing to 7200 with same config). They will range from RP flapping (can be driven by NHRP or load) to downright instability of RP once spokes start connecting to a hub. It can be bug or platform limitation, one can write a book about this :-)

This post was meant to show you what are some common problems and how to track down the failing component. Hope it helps. If you're interested to learn more let me know.

Friday, January 22, 2010

DMVPN phase 3 - basic configuration example.

Phase 3 DMVPN is not a new topic. But Cisco documentation on this matter is bit lacking.

Two best articles are:
http://www.cisco.com/en/US/docs/ios/sec_secure_connectivity/configuration/guide/sec_DMVPN_ps6350_TSD_Products_Configuration_Guide_Chapter.html
combined with:
http://www.cisco.biz/en/US/prod/collateral/iosswrel/ps6537/ps6586/ps6660/ps6808/prod_white_paper0900aecd8055c34e_ps6658_Products_White_Paper.html

I've seen probably around 30 deployments with DMVPN many of them with configuration mistakes. It does not mean that it will not work - DMVPN is a robust beast - but wrong configuration is just asking for trouble later on.

So let's see it. (No crypto configuration, at this point)

First of all OSPF.
Please note that point-to-multipoint OSPF network type configuration has it's drawbacks (/32 for each tunnel address, re-computation on every flap), majority of people will want to use network type of broadcast.

Spoke config:
--------
interface Tunnel1
ip address 172.25.1.2 255.255.255.0
no ip redirects
ip nhrp map multicast 10.1.1.1
ip nhrp map 172.25.1.1 10.1.1.1
ip nhrp network-id 1
ip nhrp nhs 172.25.1.1
ip nhrp shortcut
ip nhrp redirect
ip ospf network point-to-multipoint

tunnel source Loopback0
tunnel mode gre multipoint

Both NHRP redirect and shortcut present.
OSPF network type set to point-to-multipoint.


Hub config:
-------
interface Tunnel1
ip address 172.25.1.4 255.255.255.0
no ip redirects
ip nhrp map multicast dynamic
ip nhrp network-id 1
ip nhrp redirect
ip ospf network point-to-multipoint

tunnel source Loopback0
tunnel mode gre multipoint

Only ip nhrp redirect is configured.

EIGRP spoke
--------
interface Tunnel1
bandwidth 64000
ip address 172.25.1.4 255.255.255.0
no ip redirects
ip nhrp map multicast 10.1.1.3
ip nhrp map 172.25.1.3 10.1.1.3
ip nhrp network-id 1
ip nhrp nhs 172.25.1.3
ip nhrp shortcut
ip nhrp redirect

tunnel source Loopback0
tunnel mode gre multipoint



EIGRP hub
---------
interface Tunnel1
bandwidth 64000
ip address 172.25.1.1 255.255.255.0
no ip redirects
ip nhrp map multicast dynamic
ip nhrp network-id 1
ip nhrp redirect
no ip split-horizon eigrp 1
ip summary-address eigrp 1 10.20.0.0 255.255.0.0 5

tunnel source Loopback0
tunnel mode gre multipoint

Note that in this particular case all the networks that this DMVPN clouds is "protecting" can be summarized into 10.20.0.0/16.

Please note increased bandwidth on tunnel interfaces for EIGRP. Default BW is 8kbit and will be used to calculate metrics for EIGRP.

So how would an intermediate/regional hub configuration look like?
interface Tunnel1
ip address 172.25.1.3 255.255.255.0
no ip redirects
ip nhrp map multicast dynamic
ip nhrp map 172.25.1.1 10.1.1.1
ip nhrp map multicast 10.1.1.1
ip nhrp network-id 1
ip nhrp nhs 172.25.1.1
ip nhrp redirect
no ip split-horizon eigrp 1
ip summary-address eigrp 1 10.20.10.0 255.255.255.0 5

tunnel source Loopback0
tunnel mode gre multipoint

Friday, January 08, 2010

FWSM - routing considerations or "Why clearing xlates solves it?"

edited on 18th May 2010.

Yet another things people tend not to realize (or maybe just those that do not attend networkers?).

All Cisco firewall appliances (ASA/PIX/FWSM) consider xlates before routing information. If some traffic has created an xlate all subsequent traffic will follow the path of xlate and not the route.

What I would like to point out is a problem that has been reported to me quite often.

There is a whole class of problems that manifests itself on the FWSM (mostly, but will also impact ASA/PIX) for which "clear xlate" is the only and temporary solution and here's what you need to know about it.

The problem shows in most cases where you have (pick as many as you like)
- no nat-control
- same security interfaces

There is one cure, which you should configure anyway on your appliance as a best practice.
Configure unicast RPF on ALL interfaces.
http://www.cisco.com/en/US/docs/security/fwsm/fwsm40/configuration/guide/protct_f.html#wp1042625

Why did I mention that the problem MANIFESTS itself on the FWSM?
Because the FWSM is working as expected - when a packet with same IP comes through an interface an xlate will be created. Once the xlate is created the traffic will consider existing xlates before routes.
So bottom line, if you have an improper xlate already installed in your xlate table, things may not work until you clear xlate(s), or it times out. However if you have continuous traffic keeping that xlate alive, the only way to clear this is to clear xlates. 

Unicast RPF will prohibit the bogus packet to create a xlate in the first place.

Why is this mostly seen on the FWSM? Because of the placement of FWSM - it's usually a datacenter with a mix of layer 2 and layer 3 traffic - probably proxy arp enabled on layer 3 interfaces, maybe some route leaking from a VRF that is normally protected by the FWSM and many many others.
Most of the time it's easier to fix the symptom then the root cause.

Enable unicast RPF on the FWSM in all your new deployment or consider contacting your local account team to get following bug integrated:
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsi14227

If you want to get to the bottom of this problem enable information level messages and monitor what connection is causing this bogus xlate to be created. Usually you will see two sequential messages, creating connection, creating xlate. You will need to match the "outage" to creation of the xlate. From there you trace the packets to L2/L3 interfaces all over the network.

Wednesday, January 06, 2010

ASA/PIX PKI implementation. Mupliple trustpoints considerations.

Not sure if Cisco documents it anywhere, but here goes.

What happens if you have multiple trustpoints defined on the ASA.

When a certificate is presented to the ASA, the appliance can use ANY trust point configured on the device and will use first one matching provided client type is matching.

You cannot change this behavior, except for specifying different certificate usage:
http://www.cisco.com/en/US/docs/security/asa/asa80/command/reference/c4.html#wp2124040

You do however have control over which certificate is being SENT to the peers, this is what you configure under tunnel-groups and ssl CLIs.

Sunday, January 03, 2010

IPsec VPN on Catalyst 6500 or 7600.

First thing you need to know is that IPsec VPN on 6500 and 7600 will not work from SXE.

I've seen this too many times, problems with IPsec VPN on 6500 or 7600 with VPN SPA (Shared Port Adapter) or VPNSM (Service Module).

The place you need to start is:
http://www.cisco.com/en/US/docs/interfaces_modules/shared_port_adapters/configuration/6500series/76ovwvpn.html
Cross reference if the modules, configurations you're using are supported in the first place. They are not supported usually for very good reason - Cisco didn't deem it important enough.

If this is a new setup and you do not need any VRF features I would recommend going for CCA (Crypto Connect Alternative). In this mode the crypto engine operates in VRF mode but everything can remain in global VRF.
Here's a decent config example:
http://www.cisco.com/en/US/docs/interfaces_modules/shared_port_adapters/configuration/6500series/76cfvpna.html#wp2048824

Best software to run the VPN SPA with ... (3rd Jan 2010) SXI2a or SRC4, my personal types.
If you're considering VPN SPA as the platform for remote access, save yourself the trouble - use ASA instead (if you're into Cisco of course).

The problem with vlan 1
There is a known problem which is fixed by Cisco in a strange way. If you have vlan 1 configured on trunks to VPN SPA you might run into performance problems. So if you have problems with IPsec performance on 6500 and 7600 (only VRF and CCA mode) - remove vlan 1 from the trunks for VPN module (interface gigabitEthernet Module_slot/Subslot/0 and 1).
The fix implemented by Cisco:
For new installations do not add vlan 1 to the trunks for VPN SPA...
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsl28371

edited: 17th Jun 2010

Saturday, January 02, 2010

IPsec and VRFs. So who's doing the VRF handoff anyway?

VRF aware IPsec - complicated topic? Not really.

So what are the the options that we need to configured
1) what will be the front (FVRF) and back (IVRF) VRF?
2) Which VRF will we be matching a given identity on?
3) Post encapsulation which VRF will we be putting the packets on?
You can read all this in the configuration guide:
http://www.cisco.com/en/US/docs/ios/sec_secure_connectivity/configuration/guide/sec_vrf_aware_ipsec_ps10591_TSD_Products_Configuration_Guide_Chapter.html
Check out the excellent examples.

So let's consider something more advanced. GRE over IPsec with VRFs.
Very often people will not know what should happen with IVRF, should it be specified under isakmp profile or not?
How does FVRF behave?
Following also applied to SVTI tunnels, and to some extent DVTI, note that VRF integration with L2TP(/with IPsec) is a whole new world.

Consider this example.
I have VRF BB linking both sides of GRE tunnel. One side has VRF BLUE and the other RED.
I will configure one side using tunnel protection (TP) other using crypto maps.
Certificate authentication is easier (no keyring needed) but let have a look.
Note that I do not need to go to my CA server via any VRF so I do not specify VRF in trustpoints.

TP side:
------------------------------
crypto pki trustpoint cisco2
 enrollment url http://10.0.0.3:80
 serial-number
 revocation-check crl
crypto isakmp policy 10
 encr aes
 group 2
crypto ipsec transform-set ITS esp-3des esp-sha-hmac
crypto ipsec profile PRO
 set transform-set ITS
interface Tunnel0
 ip vrf forwarding BLUE
 ip address 172.16.0.1 255.255.255.252
 tunnel source Ethernet0/0.100
 tunnel destination 10.0.0.101
 tunnel vrf BB
 tunnel protection ipsec profile PRO
------------------------------
And yes, it's GRE/VTI that is doing the handoff. Not convinced? Let's have a look at crypto maps.

------------------------------
crypto isakmp policy 10
 encr aes
 group 2
crypto isakmp identity dn
crypto isakmp profile IPSEC_VRF
   ca trust-point cisco
   match identity address 10.0.0.100 255.255.255.255 BB
 crypto map MAP 100 ipsec-isakmp
 set peer 10.0.0.100
 set transform-set ITS
 match address GRE
interface Tunnel0
 ip vrf forwarding RED
 ip address 172.16.0.2 255.255.255.252
 tunnel source Ethernet0/0.100
 tunnel destination 10.0.0.100
 tunnel vrf BB
interface Ethernet0/0.100
 encapsulation dot1Q 100
 ip vrf forwarding BB
 ip address 10.0.0.101 255.255.255.0
 crypto map MAP
------------------------------

Is it working? And what are the main differences... Let's have a look at show crypto ipsec sa output.

TP side:
------------------------------
interface: Tunnel0
    Crypto map tag: Tunnel0-head-0, local addr 10.0.0.100
   protected vrf: BLUE
------------------------------

Crypto map side:
------------------------------
interface: Ethernet0/0.100
    Crypto map tag: MAP, local addr 10.0.0.101
   protected vrf: BB
------------------------------

Is that at all correct? Well check "show crypto map". All I can say "It's pinging!"

Friday, January 01, 2010

EZVPN with certificates. part3

In my previous post I used certificate maps rather then using match identity groups. Why?

I could have easily used "match identity group LAB" as you can see in debugs from previous post, but I prefer not to. Certificate maps are here to stay they offer much more flexibility then static matching.

Here's some background why you might want to consider using cert maps in production.

As of IOS 12.4(20)T (including and everything above) some connection - like L2L tunnels from example stopped sending Unity VID. So what? So, match identity group will not work in this case, it's only being used in case of ezvpn - where unity tag is set. Where is this documented? It's not, but feel free to check your debugs :)

A thought on virtual templates if you're not using them, you should start moving your setup to this. Cisco will be making this THE ezvpn setup (for both client and server). It offer hugle flexibility improvements and fixes some of the shortcomings of legacy configurations (NAT, firewall, access-lists), plus all other remote access methods (L2TP, PPTP and yes recently even webvpn) are using it already 
Virtual-template ezvpn setup is referenced as DVTI, while tunnel interfaces with "tunnel mode ipsec ipv4" is called SVTI - just in case you have to work with Cisco TAC.

EZVPN with certificates. part2

So I cheated a bit.

Previous post, that would not really work.... here's why.  Check out MM3/4 on server side and MM5 on client where it would actually fail.
-----------------------------------
*Dec 28 20:48:41.199: ISAKMP:(1006): processing CERT_REQ payload. message ID = 0
*Dec 28 20:48:41.199: ISAKMP:(1006): peer wants a CT_X509_SIGNATURE cert
*Dec 28 20:48:41.199: ISAKMP:(1006): peer wants cert issued by cn=SUBCA2.cisco.com,ou=LAB1
(....)
*Dec 28 20:48:41.199: ISAKMP:(1006):Old State = IKE_R_MM3  New State = IKE_R_MM3
*Dec 28 20:48:41.215: ISAKMP (1006): constructing CERT_REQ for issuer cn=SUBCA1.cisco.com,ou=LAB
-----------------------------------

Which results in client failing in MM5 because we cannot find a common CA.

Easy fix - enrolling to same CA.

So the end config
-----------------------------------
aaa new-model
aaa authentication login EZ local
aaa authorization network EZ local
crypto pki certificate map MAP1 10
 subject-name co lab
crypto isakmp policy 10
 encr aes
 group 2
crypto isakmp client configuration group EZ_GROUP
 domain cisco.com
 pool EZ
 save-password
 include-local-lan
 pfs
crypto isakmp profile ISAKMP_PROFILE
   match certificate MAP1
   client authentication list EZ
   isakmp authorization list EZ
   client configuration address respond
   client configuration group EZ_GROUP
   virtual-template 100
crypto ipsec transform-set ITS esp-3des esp-sha-hmac
crypto ipsec profile PRO
 set transform-set ITS

interface Virtual-Template100 type tunnel
 ip unnumbered Loopback0
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile PRO
-----------------------------------


-----------------------------------
crypto ipsec client ezvpn EZ_CLIENT
 connect manual
 mode client
 peer 192.168.0.1
 virtual-interface 100
 username cisco password cisco
 xauth userid mode local
interface Virtual-Template100 type tunnel
 no ip address
 tunnel mode ipsec ipv4
end
-----------------------------------


Server side debugs:
------------------------------------
*Jan  2 09:53:46.527: ISAKMP:(1003):Old State = IKE_R_MM3  New State = IKE_R_MM3
*Jan  2 09:53:46.547: ISAKMP (0:1003): constructing CERT_REQ for issuer cn=SUBCA2.cisco.com,ou=LAB1
*Jan  2 09:53:46.547: ISAKMP (0:1003): constructing CERT_REQ for issuer cn=SUBCA1.cisco.com,ou=LAB
(....)
*Jan  2 09:53:46.615: ISAKMP:(0):: UNITY's identity group: OU = LAB1
*Jan  2 09:53:46.615: ISAKMP:(0):: peer matches *none* of the profiles
*Jan  2 09:53:46.615: ISAKMP:(1003): processing CERT payload. message ID = 0
*Jan  2 09:53:46.615: ISAKMP:(1003): processing a CT_X509_SIGNATURE cert
*Jan  2 09:53:46.615: ISAKMP:(1003): peer's pubkey is cached
*Jan  2 09:53:46.615: ISAKMP:(1003): OU = LAB1
*Jan  2 09:53:46.615: ISAKMP:(0): certificate map matches ISAKMP_PROFILE profile
*Jan  2 09:53:46.615: ISAKMP:(0): Trying to re-validate CERT using new profile
*Jan  2 09:53:46.615: ISAKMP:(0): CERT validity confirmed.
--------------------------------