Network Outage Update

Colorado 811 Network Outage Update – September 6 and 7, 2018

  • On September 6, 2018 at approximately 10:35 am Colorado 811 experienced a network outage due to a failure of a T3/DS3 network circuit at one of CO811’s network provider’s operation centers.
  • The network outage resulted in the temporary loss of all inbound and outbound telecommunications and internet access for all onsite employees. Colorado 811 Teleworkers lost phone communications and access to internal Colorado 811 software applications. Our Members and the public temporally lost the ability to call the contact center at Colorado 811 via phone for twenty-five minutes.
  • The specific cause of the failure is unknown at this time; however, we continue to push for a detailed incident report from our network provider.
  • At 10:45 am Colorado 811 engaged the Network Operations Center’s management Team and escalated the support ticket to its highest level. After some conversation with network engineers and their managers it was determined that the problem was downstream at another location with the Local Exchange Provider (LEC). An additional ticket was opened with the LEC to further troubleshoot the issue impacting Colorado 811.
  • At 11:00 am Colorado 811 successfully routed all 811-phone traffic to an alternate phone server located offsite. This resulted in the ability for 5-6 teleworkers to process tickets and receive 811 phone calls from their homes. The first phone call was received and processed at 11:03 am. All Colorado 811 services were available at this point. Hold times may have increased due to the limited number of agents that were available.
  • Between 12:00 pm and 12:45 pm Colorado 811 successfully configured the contact center agent workstations and phones to route traffic over an alternate managed internet circuit. The result was the restoration of phone and internet service to the contact center agents. By 12:45pm every agent had been configured to run on the alternate network circuit and the contact center was fully operational.
  • At 12:47 pm Colorado 811’s Network provider indicated that the circuit should now be restored. The circuit was tested by IT, who concluded that the circuit was not yet fully capable of carrying the load of phone calls and network traffic required to operate. IT could still see network package loss and bandwidth issues on the circuit. IT escalated the support ticket again and Colorado 811 asked the provider to continue with their troubleshooting process. Our contact center remained on the alternate circuit for the remainder of the day.
  • From 4:00 pm to 8:00 pm Colorado 811 and the network provider performed troubleshooting throughout the evening. It was decided to leave the contact center on the alternate circuit until the restored circuit proved that it was ready for production.
  • On September 7, 2018, the restored MPLS circuit passed all checks for packet loss, latency and bandwidth. At 9:00 am Colorado 811 IT began the process of switching the contact center agents back to their original configuration. All Colorado 811 services are available.

CURRENT STATUS: We are seeing some slowness on the circuit creep back up. We have moved about 10 agents back, but we’ll pause for now. We just spoke to CenturyLink to escalate with a manager. We were told that a tech would be reassigned to the ticket again and that we may need to take the circuit down for testing. They will let us know first before they do that. We explained our business (public safety organization), our problem and our frustration.

Click here to download the official Network Outage Update PDF