Aristata CEO Carlin Walsh made an appearance at the most recent Chaffee Board of County Commissioners (BoCC) meeting the week of Thanksgiving to provide an accounting of the technology meltdown that disrupted service to the Arkansas River Valley off and on, from November 7 through 19.
Carlin Walsh, CEO of Aristata Communications. Courtesy photo.
“I want to start by apologizing for the disruption to the county. This has disrupted lives and businesses and flies in the face of our Aristata mission,” Walsh began. “This incident was not small, or a routine problem. It involved the engineering network for the majors, including Verizon, and AT&T.”
Aristata had announced a technology upgrade ahead of time. Typically these happen in the middle of the night when usage is lowest. But then something happened. Walsh outlined what was diagnosed as an international network problem.
“By 4:00 a.m. the day it occurred, we had engineers from other countries, from South Africa, Britain, Australia, trying to discover what had gone on.”
“There is not a layman’s definition of what happened,” he explained. “We discovered that part of our inherited legacy network was unknowingly using an inaccurate network configuration. It had been there for years. When we made an equipment upgrade it took the entire network down.”
Walsh explained that at the core of the problem was an equipment malfunction at 910 Telecom Center in Denver. “We began to provision the network, we stabilized the network at 4:00 p.m. Nov. 9. We thought that was it.”
Then on Nov.16, the network went down again. “This demonstrated this wasn’t the equipment here. It happened here, but it also happened in Australia,” he explained. “We found it was the software configuration. We got the network back up and stable by 4:00 a.m. November 19.”
Walsh assured the BoCC and the public that Aristata will continue to tweak the network to optimize traffic over the next few weeks and apologized again for the service disruptions.
“What this highlights is how critical this infrastructure is to our daily lives,” said Commissioner P.T. Wood. “Is there a way that we can help you in sharing information when these things happen? It’s important to the entire county that we ensure that this piece of infrastructure is working seamlessly.”
“We were quick to get out [the word] on November 7 for the VoIP (Voice-over IP) service to the sheriff’s and EMS on the situation – we got out with communications. But the second [outage] on November 16 left a lot to be desired,” admitted Walsh. He agreed that better communications with the county, and other utilities [and we suggest, the news media] would be desirable.
One of the challenges it turns out, is today’s jumble of landlines and cell numbers across multiple counties served by Aristata. There is no actual list.
“We have everybody’s phone numbers, but we don’t know if they are landlines or cell numbers,” said Walsh. “We have got to get everyone registered, including the VoIP customers… for those in say Custer [county], it is challenging, they have no other option there.” He added that if a Short Messaging Service (SMS) text system could be instituted in Chaffee County, that it might have expansion potential into other counties.
The BoCC was not without empathy for the challenge the outage represented. “I’m still amazed every morning when I wake up to plumbing and hot water,” said Commissioner Felt. “It amazes me that the system works as well as it does. Going forward, we’ve had a window into what could happen…. we should use this to prepare to handle things in the future.”
“We appreciate that you’re local, and you’re invested in the community,” said Commissioner Keith Baker. “With any of our other providers, you’re getting a call center far away. Here we know we can reach you.”
Walsh, who was born and grew up here, and is known for his commitment to the valley, wrapped up his comments again expressing appreciation for those who have been supportive as Aristata worked through the crisis.
Recent Comments