Innovation through Aggravation

Cisco ACE context erasing

Cameron — Fri, 19 Nov 2010 08:36:07 +0000

One of the most annoying things I found out about the ACE module was that I couldnt just ‘write erase’ a context from within it. For example, lets say I just wanted to erase the current config and paste in a new one because of massive changes or I’ve got a test context that changes a lot etc….well I couldnt find an easy way to do it.

What I do now to overcome this is use checkpoints. When I create a brand new context; the first thing I do is create a checkpoint – a blank one:

checkpoint create blankcontext

Then, paste in your new config and if its stable then create a new checkpoint. Checkpoints are also useful for instant rollbacks of any changes – just create a new checkpoint whenever you are about to make changes.

Having the blank checkpoint has come in handy a few times. Note that checkpoints are only visible within the same context – so you need to create them in multiple contexts as appropriate.

Call Manager 7 – admin password recovery

Cameron — Tue, 28 Sep 2010 08:44:56 +0000

I had a vm for cucm7.1(3) that I’d had off for a while – and I needed to fire it up again for some lab testing. Of course, my usual set of passwords didnt work to get into the thing. So, at the console of the vm, I did the following:

1. Login at the console using username: pwrecovery password: pwreset.

2. Edit the vm’s settings to ensure no cdrom images or physical cdroms are connected.

3. Follow the prompts, and re-enable the call manager iso image in the cdrom drive when prompted to ‘insert the cdrom’. Its basically asking for the call manager install media.

This resets the user admin account to ‘admin’ with a new password you specify.

Upgrading Cisco 4400 Wireless Controllers

Cameron — Fri, 24 Sep 2010 04:18:44 +0000

Make sure you absolutely check the release notes for the software as changes to the behaviour of the wlans and settings can occur between releases.

I found it is better to ssh into the controller and upgrade it from the CLI. This gives you a running output of what is going on at each stage.

debug transfer trace enable
transfer download datatype code
transfer download mode tftp
transfer download serverip 10.1.1.10
transfer download path /
transfer download filename AIR-WLC4400-K9-6-0-196-0.aes
transfer download start

After a successful download of the new image you should see something like:

TFTP File transfer is successful.
                                   Reboot the switch for update to complete.

Sat Jun 19 15:47:00 2010: Still waiting!  Status = 2 Sat Jun 19 15:47:03 2010: Still waiting!
Sat Jun 19 15:47:04 2010: finished umounting

(Cisco Controller) >

Issue a ‘reset system’ and say yes to saving the config before rebooting. Remember to repeat for the bootloader file should the release notes say to do so. It takes about 2 mins for the WLC to reboot. If there are access points registered to the WLC before rebooting, they will migrate to another controller.

Check release notes of the main image re bootloader. Similar procedure, however the version on the cisco web site has a ‘BOOT’ after the name. The example below was for version 5.2.

debug transfer trace enable
transfer download datatype code
transfer download mode tftp
transfer download serverip 10.1.1.10
transfer download path /
transfer download filename AIR-WLC4400-K9-5-2-157-0-ER.aes
transfer download start

The following output (show sysinfo) after reboot verifies the versions:

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 6.0.196.0
RTOS Version..................................... 6.0.196.0
Bootloader Version............................... 4.1.171.0
Emergency Image Version.......................... 5.2.157.0
Build Type....................................... DATA + WPS

Time to resurrect this

Cameron — Fri, 24 Sep 2010 03:41:52 +0000

Its been a while…:/

VLAN groups for fwsm and ace on 6513

Cameron — Wed, 30 Apr 2008 20:15:10 +0000

With the data centre move, I took the opportunity to clean up some of the 6513 config which had got out of control.

The original groupings looked all over the place:

svclc vlan-group 1 27,28,40,76
svclc vlan-group 2 44,45,48
svclc vlan-group 42 42
svclc vlan-group 43 43
svclc vlan-group 400 402
svclc vlan-group 427 427,428
svclc vlan-group 500 527,528,544,545,548
svclc vlan-group 600 612,614,618,620,621,622,623,624,625,626,628,632,636,699
svclc vlan-group 602 602
svclc vlan-group 700 720,721,724,725,732
svclc vlan-group 990 996,997
svclc vlan-group 999 9,999
svclc module 2 vlan-group 2,43,400,427,500,600,602,700,999,
firewall module 1 vlan-group 1,42,427,428,600,990,999,

….YUK. However I decided to break the FWSM and ACE’s visible vlans into groups with more meaning; specifically:

Group 1 = specific to FWSM
Group 2 = common to both
Group 3 = specific to ACE

In the end the config gets cleaned up and looks like:

svclc vlan-group 1 27,28,40,42,76,612,614,618,622,623,626,628,636,699,996,997
svclc vlan-group 2 9,427,428,620,621,624,625,632,999
svclc vlan-group 3 43,44,45,48,402,527,528,544,545,548,602,720,721,724,725,732

firewall module 1 vlan-group 1,2
svclc module 2 vlan-group 2,3

…much better ! A few of the other commands that got me out of a few dramas:

firewall autostate
firewall multiple-vlan-interfaces
svclc autostate
svclc multiple-vlan-interfaces

(The autostate command helps the devices track the loss of an access link).

Moved Data Centre

Cameron — Wed, 30 Apr 2008 20:04:24 +0000

The Data Centre move happened last weekend – all went well and we did it (a team of 11) in 24 hours. I had to look after the firewalls, ACEs, proxy, WAAS as a priority and chipped in whereever else when required. For a business that does several hundred million dollars of business through its web site (as one example) – It was a great effort to have the business back online 2 days prior to expectation. Some of the guys I work with really showed their expertise and planning skills.

One thing I learned was that if unsure, always bring up the standby FWSM or ACE when :

A. The primary unit is online and functional and is active.

B. The underlying network (vlans, switch configs etc) are up and stable.

I had a few dramas where the failover states of the FWSM or ACE seemed ok, but because a few things were changed during the move things started to go awry. In some cases both FWSMs or ACEs were up, but because of some underlying changes to the network they would both think they are active and the network can get turned into a banana. (eg. failover vlan removed!)

In hindsight, I should have left the secondary modules out until all other network work was completed. Just because you think the modules are working at one point in time doesn’t mean they will be behaving as expected at a later point if there are changes being made to the network.

Cisco ACE log bug

Cameron — Wed, 19 Mar 2008 07:08:42 +0000

That old chestnut – the weird ace log issue – now has a bug ID ! (CSCso15332). It only appears to happen if the rservers are offline in the original context. Bizarre.

Here is what Cisco has said on the matter:

Conditions:

The customer was using the same syslog host in two contexts, had configured http and https probes in both contexts, and was logging probe failures in both contexts. The rservers in the second context were unavailable when the issue began; later, those rservers became available and the inappropriate logging ceased.

Workaround:

None.

Further Problem Description:

The customer experienced this problem for approximately ten days sproadically on one rserver. Even during that period there were at most a moderate (20 to 30) number of messages a day.

NTP on the 6509

Cameron — Wed, 19 Mar 2008 07:03:43 +0000

Had some minor drama with the core switch clock being out by well over an hour. Not causing any issue except frustration when viewing logs and trying to compare timestamps across devices.

All the network devices get their time sync from the wan router (which in turn contacts our ntp server for its time).

To fix it up – remove all the ntp related statements from the cisco 6509, then manually set the clock to within 5 mins of the actual time as reported by the ntp server, then re-apply the ‘ntp server’ statement(s). After a short while the clock will re-sync correctly with the ntp server specified.

Never apply the ‘ntp clock-period’ statement – this is done by the device itself. I think someone manually applied this and that was why the time started to get way out.
ie. Make sure you never copy the ‘ntp clock-period’ statement from one device to another. The clock-period is set by NTP; to regulate the internal clock on the device. This is the number of ticks in one second. Therefore, NTP doesn’t change the time of the switch, it adjusts the clock-period so that the system clock is synchronized with the ntp server.

No luck so far with the ACE log issue

Cameron — Sat, 01 Mar 2008 23:55:15 +0000

I’ve sent more info to the TAC re that ACE logging issue – which has not shown up again for a few weeks. It stopped suddenly after the production rservers in the ‘correct’ context were finally started up (the context was prepared and active prior to server deployment). In any case, the TAC has escalated it to the engineering team for analysis. I agreed with one of Cisco’s engineers that said “this shouldn’t be possible”. Love it :)

We deployed a Cisco WAAS appliance to the Brisbane office – all seemed ok however the voice team said ‘at some stage after the WAAS was turned on’ they had messages-on-hold errors. Strange given that WAN optimisers ignore udp. Because the deployment is using wccp – it is likely something has gone wrong with the router – or it could be unrelated altogether.

Also, I learned a valuable lesson the other night – never gloss over the ‘clear xlate’ command on a FWSM. When messing around with static nats, hours can be saved troubleshooting by using that command … bah.

Riverbed vs WAAS … part one

Cameron — Thu, 21 Feb 2008 07:32:24 +0000

I’ve been re-visiting the whole Riverbed vs Cisco WAAS situation again. We have a few WAAS appliances and modules in production, and they do the job – for now. I am concerned about the scalability of their solution and it is my opinion that Riverbed has the edge here and in other areas – except price. I’m not one to really care about price usually because I am more concerned with the outcome – but sure it is a consideration for the business. Hence, I’ve gone back to have a look at it all again.

Watch the Riverbed video here

It is impressive stuff. Cisco has the edge on price – but not much else. Riverbed is a clear winner when it comes to ease of deployment, scalability and technology. I’ll come back and justify my comments with some data soon.