Sixty-six emergency communications districts in Tennessee experienced service interruptions, lasting more than 97 hours, as a result of the Christmas morning suicide bombing on Second Avenue in downtown Nashville, according to a report released earlier this month by the Tennessee Emergency Communications Board.
Working in tandem with AT&T, the contractor for the state's emergency communications system, the board made contact with each 911 district impacted by the bombing and unveiled its report at a board meeting May 5. The AT&T hub was among the buildings blasted when, according to police, a suicide bomber detonated a device inside an RV parked just 20 feet from the building.
The goal of the report was to identify the impact of the Christmas morning bombing on 911 systems, what went wrong that the impact on the 66 districts and what can be done to avoid similar situations should disaster strike an emergency communications system again.
According to the report, which was overseen by board vice chairman Steve Martini, the cause of the sweeping communications outages was the loss of power at the site of the bombing. The bomb inside Anthony Quinn Warner's RV was detonated at 6:30 a.m., ripping through Nashville's famous historic district and severely damaging about a dozen buildings.
In the immediate aftermath of the bombing, there did not appear to be loss of 911 communication, cell phone or internet service.
"The blast damaged the physical exterior and destroyed the electrical power interface to the building," Martini said. "Without power, equipment automatically switched to the battery backup."
However, backup generators intended to support the batteries were damaged, or knocked offline due to the explosion or the ensuing fire suppression efforts by the Nashville Fire Department.
That created a race against time to fix the backup generators so that the batteries kept the vital 911 communications equipment online. But, according to the report, AT&T repair crews were denied access to the building for 12 hours by government officials because of safety concerns about the structural integrity of the building as well as the need to gather evidence.
"It prevented AT&T from timely utilizing its existing backup power sources, extra batteries, replacement generators or repairing generators," Martini said.
It was at this point, around 11:50 a.m., that 911 districts began experiencing service interruptions. Protocols called for AT&T to reach out about the situation to individual 911 districts through a communications system called everbridge. But, according to a key finding in the report, some 911 districts didn't receive those updates. According to Martini's report, 911 districts provide AT&T with phone numbers and/or email addresses to receive those everbridge communications. But, for the same reason citizens lost cellular and internet service in the days after the blast, some local 911 officials never received the everbridge messages.
"The internet losses prevented the messages from reaching their destination," Martini said. "Some [911 districts] may need to update their everbridge contacts in terms of both technology and carrier."
According to Martini's report, 86 911 communication systems experienced no known impacts and 10 had indirect impacts. AT&T was able to reroute calls to its communications center in Knoxville.
For the 66 impacted districts, service was not fully restored until 97.5 hours after the initial outage on Dec. 29.
Many local governments offered alternative numbers for citizens in need of emergency response to call.
"For those 66 [911 centers] with transport impacted, the timing of the impact and restoration may differ," Martini said.
The loss of 911 communications for four days forced the emergency communications board and AT&T to examine what can be done to avoid a similar outage.
Since the loss of power was the central cause of the issue, Martini said, "going forward AT&T is actively taking steps to additionally fortify power supply."
It's been a lesson learned not just for the system in Nashville, but for AT&T's communications hubs across the country, according to Martini's report.
"AT&T is evaluating re-engineering electrical power shutoff infrastructure into central offices nationwide," Martini said in his presentation to the board. "This evaluation addresses AT&T's network throughout the nation. But for the purposes of the Nashville location, the repairs at the site of the bombing have already included repairs to the electrical power service on Second Avenue."
The company is also conducting engineering studies to enhance backup generator "flexibility and capability," he said.
Since the situation was hampered by AT&T's inability to access the building, Martini said, the company is talking to government officials to establish national protocols to expedite access to communications centers in the event of future disasters. The company has already briefed Metro and state of Tennessee officials about ensuring access to its center in the event a future crime creates a communications shortage.
To improve backup communications in the event of an outage, AT&T will explore redundant network connections to include terrestrial options, such as diverse AVPN circuits, and non-terrestrial options like LTE wireless, so that local 911 districts have alternatives if the main communications system goes down. AT&T will work with local 911 districts to explore support and information needed to seek supplemental funding from state and local resources in the event that adding a backup 911 communications option costs extra money.
Finally, the report recommended the emergency communications board staff work with local 911 districts to keep their everbridge communications methods up to date, including multiple modes of communication, in the event of a future outage.
Read more at TennesseeLookout.com.