I have a group colleagues who have a tendency to be fiercely independent until they are stuck in a bind, at which point they suddenly become completely dependent upon the very group with which they’ve attempted to maintain such fierce independence.
It’s been 16 months since we first made the decision to migrate from one vpn vendor to another. The design is complicated to be sure, but the end result is a fully redundant vpn solution that utilized routing protocols to ensure maximum uptime. (Hows that for BusinessSpeak 2.0)
My colleagues across the pond were supposed to install thier new infrastructure by December 31st 2007. Not one set of boxes was installed. We (I) provided ample guidance including complete config templates for the installation. Still nothing.
A few weeks ago, word came down that one of our european offices was moving. We have been agressively migrating off our old platform this year (with breakneck speed and minimal caution in my opinion) and have nearly replaced all our domestic boxes. Word was that the crew across the pond was planning on deploying the old solution again. Management thought that was an exceedingly bad idea since we are actively migrating away from it. Good call on management’s part.
Here’s where hair caught on fire.
I stood up the new boxes and pinned up VPNs in a non-standard inter-vendor solution a few weeks ago for my fiercely independent bretheren across the pond. Now, their main set of boxes is so over taxed because of ridiculously complicated an archaine deployment practices (packets are litterally manipulated 3 and 4 times as they cross the firewalls) and the reliance on 3DES (yes in 2008 they are still running 3DES) on the vpns. The boxes run at 99% CPU constantly.
I’m really surpised they have not caught on fire themselves.
So, not surpringly, at some point you reach the saturation point. At some point the last straw is placed on the camel’s back and it breaks. At some point the ballon bursts because you’ve put too much air in it. We reached that point.
The latest vpn is just too much for these boxes to handle. They’re choking. We know its not an interoperabilty issue or config issue, because the vpns to other boxes from this vendor are working fine from our new site.
So I get a call at 6:30 AM today. They’d like me to look at it. I say, “I’m pretty sure its a problem with the boxes in London. They’re running at 99% CPU. VPN is CPU intensive, especially when you are running 3DES.” I might as well have pissed into the wind. This must be a problem with the new vendor is the suspicion across the pond.
Nevertheless, I’ve convinced nearly everyone that we need to migrate the vpn to the new platform which is currently 99% IDLE in terms of CPU.
Of course it needs to be done immediately.
So I’ve been working up BGP configs and VPN configs since 7:00.
Oh, and its freakin raining again.
