Tomorrow marks the one year anniversary of the fiber-optic cable cut that left much of Santa Cruz in the dark for more than 24 hours. (Otherwise known as The Great Internet Outage of 2009.) In our debriefing meetings at Cruzio in the days and weeks following the incident we discussed what happened, why it had the impact that it did, and how to make sure it didn’t happen again.

So what did happen?

No one really knows. One or more vandals cut a major fiber-optic cable bundle that served Santa Cruz, Santa Clara and San Benito Counties. This bundle of lines carried backhaul data for landline telephone, internet and cellular phone communications. No one was ever caught, despite a $250,000 dollar reward. Cruzio system administrators were able to receive notifications with the final burst of traffic that got out of the county before the tiny redundant connection became completely saturated in the early hours of April 9th. Immediately Cruzio’s entire outage response team were mobilized and the clock was started on our response scenarios. Cruzio staff drove south to find an active connection so that they could post a system status update from a live connection for anyone viewing Cruzio’s website from outside of Santa Cruz county. Cruzio’s response team have various scenarios prepared that inform our decisions in the case of different types of outages, from power,  connectivity, and physical hardware failure. These plans served us well during this time—having the scenarios already planned out meant that we were able to calmly and confidently proceed along the path we had already prepared to restore services.

Why did it have such a huge impact?

We are all extremely accustomed to having persistent connections close at hand, from Internet at home, to full access to the web through devices like the iPhone and the Droid. When that persistent Internet presence disappeared on April 9th last year, along with all of the telephone communications out of the county, things got interesting very quickly. Cruzio’s servers were split between a facility in Santa Cruz and a facility in Fremont. This meant that server availability was inconsistent: some customer services were unaffected, some were totally offline. Cruzio DSL customers unfortunately have their traffic aggregated on AT&T’s equipment and the traffic is passed off to Cruzio at a termination point beyond where the fiber was cut.

In the days following the outage we at Cruzio decided that this kind of problem was not acceptable to us. If we’re unable to protect the connectivity to our systems then we’re unable to maintain the level of availability our customers have come to expect. At Cruzio we make sure we have spare hardware and redundant power systems so it was decided that a redundant architecture was needed. The major problem with redundant architecture for Santa Cruz is that all of our connectivity comes over the hill, most of it on a single path. Building out a true redundant infrastructure would mean finding another option.

What has Cruzio done to prepare for the future?

In late 2009 Cruzio completed the acquisition of Gatespeed Wireless. By combining Gatespeed’s network with Cruzio’s existing network in Santa Cruz we have been able to build a fully redundant system. If there’s another fiber cut, Cruzio’s systems will stay online and traffic will re-route over our wireless infrastructure. Cruzio is now also able to offer extremely high-speed wireless connections directly to customer locations.

Cruzio has also found a new home and our new location at the Sentinel building will allow us to consolidate our servers in a single data center where we can respond to issues more quickly than we could when they were located in Fremont. We’re also pulling our own fiber connection to our new home so we’ll no longer be dependent on a connection provided by AT&T over their fiber. This new link will allow us to connect a 10+ gigabit link to the Internet backbone in Silicon Valley. We’re also spearheading a bid to build yet another fiber loop that will add a third route out of the county for Cruzio traffic, as well as provide high bandwidth connectivity to areas traditionally underserved by the existing telecommunications infrastructure.

What’s Next?

With the combination of ultra-highspeed connectivity to our new headquarters, and our new local datacenter, Cruzio can offer services that have previously been unavailable in Santa Cruz. A fully redundant colocation facility, 100Mb and faster wireless connections to businesses around the county and beyond, gigabit Internet connections for businesses who decide to share our new space with us and more services to come.

What started out as a simple desire to protect our customers from another act of vandalism has helped Cruzio push to bring new services to Santa Cruz County that many wouldn’t have considered viable here just a year ago.

Cruzio Reporter: Chris Frost, Technical Services Manager