Incident Number 20190219
Incident Type MPLS Circuit Outage
Internal Ticket Number CO811ITSD-1331
Description of Incident
At approximately 2:30 PM MST on Thursday February 19th 2020, Colorado 811 lost MPLS connectivity on our Level 3/Century Link managed circuit BBMD4611. At 2:47 PM MST Systems Administrator created CenturyLink Ticket: # 18158445. Colorado 811 IT Team also began phone communications with Level 3/Century Link NOC at the same time. Communications Administrator placed notices on social media directing users to Ticket Express and WTE for Emergency and Damage requests. At 4:35 PM Colorado 811 engaged ATT Call Assurance to route calls to 800-922-1987 to a company cell phone with a pre-recorded message directing users to online services. At 5:35 PM MST Century Link techs reported that they were on their way to the Network Operations Center to replace a faulty card. At 6:00 PM a pop-up message was placed on our website to process Emergency Damage tickets on Ticket Express. IT Team members continued to provide after-hours support alongside members of Operations and Member Relations. At 7:00 PM MST Century Link/ Level 3 techs reported that they had successfully re-seated a network card at a Century Link/Level 3 facility. At 8:00 PM MST services were fully restored, and after-hours Damage Prevention Agents resumed normal business duties.
Operational Impact
Internal Impact – The Service Level and Average Hold Time were not impacted due to the outage. Due to weather and the morning volume of that incident, the impact to ticket volume was not measurable.
External Impact – Excavators were able to process Locate Requests and Damage Notifications through online applications.
Response – Cause and Resolution
Incident Resolution Date 02/19/2020
Root Cause Faulty network card at Level3/Century Link facility.