With the data centre move, I took the opportunity to clean up some of the 6513 config which had got out of control.

The original groupings looked all over the place:

svclc vlan-group 1 27,28,40,76
svclc vlan-group 2 44,45,48
svclc vlan-group 42 42
svclc vlan-group 43 43
svclc vlan-group 400 402
svclc vlan-group 427 427,428
svclc vlan-group 500 527,528,544,545,548
svclc vlan-group 600 612,614,618,620,621,622,623,624,625,626,628,632,636,699
svclc vlan-group 602 602
svclc vlan-group 700 720,721,724,725,732
svclc vlan-group 990 996,997
svclc vlan-group 999 9,999
svclc module 2 vlan-group 2,43,400,427,500,600,602,700,999,
firewall module 1 vlan-group 1,42,427,428,600,990,999,

….YUK. However I decided to break the FWSM and ACE’s visible vlans into groups with more meaning; specifically:

Group 1 = specific to FWSM
Group 2 = common to both
Group 3 = specific to ACE

In the end the config gets cleaned up and looks like:

svclc vlan-group 1 27,28,40,42,76,612,614,618,622,623,626,628,636,699,996,997
svclc vlan-group 2 9,427,428,620,621,624,625,632,999
svclc vlan-group 3 43,44,45,48,402,527,528,544,545,548,602,720,721,724,725,732

firewall module 1 vlan-group 1,2
svclc module 2 vlan-group 2,3

…much better ! A few of the other commands that got me out of a few dramas:

firewall autostate
firewall multiple-vlan-interfaces
svclc autostate
svclc multiple-vlan-interfaces

(The autostate command helps the devices track the loss of an access link).

The Data Centre move happened last weekend – all went well and we did it (a team of 11) in 24 hours. I had to look after the firewalls, ACEs, proxy, WAAS as a priority and chipped in whereever else when required. For a business that does several hundred million dollars of business through its web site (as one example) – It was a great effort to have the business back online 2 days prior to expectation. Some of the guys I work with really showed their expertise and planning skills.

One thing I learned was that if unsure, always bring up the standby FWSM or ACE when :

A. The primary unit is online and functional and is active.

B. The underlying network (vlans, switch configs etc) are up and stable.

I had a few dramas where the failover states of the FWSM or ACE seemed ok, but because a few things were changed during the move things started to go awry. In some cases both FWSMs or ACEs were up, but because of some underlying changes to the network they would both think they are active and the network can get turned into a banana. (eg. failover vlan removed!)

In hindsight, I should have left the secondary modules out until all other network work was completed. Just because you think the modules are working at one point in time doesn’t mean they will be behaving as expected at a later point if there are changes being made to the network.

That old chestnut – the weird ace log issue – now has a bug ID ! (CSCso15332). It only appears to happen if the rservers are offline in the original context. Bizarre.

Here is what Cisco has said on the matter:

Conditions:

The customer was using the same syslog host in two contexts, had configured http and https probes in both contexts, and was logging probe failures in both contexts. The rservers in the second context were unavailable when the issue began; later, those rservers became available and the inappropriate logging ceased.

Workaround:

None.

Further Problem Description:

The customer experienced this problem for approximately ten days sproadically on one rserver. Even during that period there were at most a moderate (20 to 30) number of messages a day.

Had some minor drama with the core switch clock being out by well over an hour. Not causing any issue except frustration when viewing logs and trying to compare timestamps across devices.

All the network devices get their time sync from the wan router (which in turn contacts our ntp server for its time).

To fix it up – remove all the ntp related statements from the cisco 6509, then manually set the clock to within 5 mins of the actual time as reported by the ntp server, then re-apply the ‘ntp server’ statement(s). After a short while the clock will re-sync correctly with the ntp server specified.

Never apply the ‘ntp clock-period’ statement – this is done by the device itself. I think someone manually applied this and that was why the time started to get way out.
ie. Make sure you never copy the ‘ntp clock-period’ statement from one device to another. The clock-period is set by NTP; to regulate the internal clock on the device. This is the number of ticks in one second. Therefore, NTP doesn’t change the time of the switch, it adjusts the clock-period so that the system clock is synchronized with the ntp server.

I’ve sent more info to the TAC re that ACE logging issue – which has not shown up again for a few weeks. It stopped suddenly after the production rservers in the ‘correct’ context were finally started up (the context was prepared and active prior to server deployment). In any case, the TAC has escalated it to the engineering team for analysis. I agreed with one of Cisco’s engineers that said “this shouldn’t be possible”. Love it

We deployed a Cisco WAAS appliance to the Brisbane office – all seemed ok however the voice team said ‘at some stage after the WAAS was turned on’ they had messages-on-hold errors. Strange given that WAN optimisers ignore udp. Because the deployment is using wccp – it is likely something has gone wrong with the router – or it could be unrelated altogether.

Also, I learned a valuable lesson the other night – never gloss over the ‘clear xlate’ command on a FWSM. When messing around with static nats, hours can be saved troubleshooting by using that command … bah.

I’ve been re-visiting the whole Riverbed vs Cisco WAAS situation again. We have a few WAAS appliances and modules in production, and they do the job – for now. I am concerned about the scalability of their solution and it is my opinion that Riverbed has the edge here and in other areas – except price. I’m not one to really care about price usually because I am more concerned with the outcome – but sure it is a consideration for the business. Hence, I’ve gone back to have a look at it all again.

Watch the Riverbed video here

It is impressive stuff. Cisco has the edge on price – but not much else. Riverbed is a clear winner when it comes to ease of deployment, scalability and technology. I’ll come back and justify my comments with some data soon.

I’ve logged a call a few days ago with Cerulean (hopefully it will make it to the TAC) regarding a strange log entry I am seeing in version 6.3 of the ACE software. A health probe is failing, however the target server is not actually defined in the context in which the probe resides. (It actually resides in a different context – so it should not be ’seen’ by this probe).

There’s no reference to this server in the context in question, yet the probe thinks it is. In any case, I hope the TAC will let us know whats going on. It isn’t affecting production, just a weird thing which I hope isn’t a sign of something more sinister.

Update: The TAC got back to me; they are equally baffled. They’ve got all the details they need and are trying to find out what’s going on.

Avamar (at the time of test) only has a client for OSX 10.4 (Tiger). We wanted to test the Avamar client on OSX to ensure that it could backup the ‘illegal’ folder and filenames seen by the OS. (‘Illegal’ in terms of what other OS’s normally expect – we actually wanted the files to be moved to another fileserver – that’s another story).

For example – the sample set of data is:

drwxr-xr-x 3 camerons camerons  4096 Feb 12 12:40 ⢠ALL FILES â¢
drwxr-xr-x 4 camerons camerons  4096 Feb 12 12:42 FONTS?????
drwxr-xr-x 2 camerons camerons  4096 Feb 12 12:42 DIRECT_LB | DIRECT
drwxr-xr-x 4 camerons camerons  4096 Feb 12 12:42 RANGES07:08
-rw-r----- 1 camerons camerons 74321 Feb 12 12:42 lanyard 1 ???.jpg
drwxr-xr-x 2 camerons camerons  4096 Feb 12 12:42 New_Change27:3

Obviously, some of the filenames above are fine – but some are a little wacko. The restore operation completes – but in the test the restored folder was not visible in the finder – only via terminal. Restart of the osx vm was necessary (the finder seemed to hang and I couldnt restart it) and then the restored files were visible. This could be a quirk with the ‘hacked’ osx vm.

Note that also the terminal shows some folder names as including a ‘:’, whereas in Finder they would appear as ‘/’ – for date format names. This is another quirk I suppose.

In summary, the Avamar client for OSX works as expected, even taking into account the non-standard filenames.

I got the 10.4.10 vmware appliance for some testing (find it on the torrent search sites) – works like a charm under VMware Fusion (v1.1) on my MBP. The cool thing about it is that it fires up just like a new install of OS X (Welcome prompts for user info etc) – so for all intents it is a clean install vm.

I’ll be testing the Avamar client for OS 10.4 tomorrow – we really need one for 10.5 but that is apparently some time off before we’ll see it (like the RHEL5 client!)….so I’ll be using the 10.4 guest as a guinea pig. All my macs that I can access are all on Leopard – so I’ve gotta resort to the vm method.

Good on the guy who made the vm appliance anyway. Champion effort.