Network Outage Report 02/19/2020

Incident Number  20190219 

Incident Type  MPLS Circuit Outage 

Internal Ticket Number  CO811ITSD-1331 

Description of Incident

At approximately 2:30 PM MST on Thursday February 19th 2020, Colorado 811 lost MPLS connectivity on our Level 3/Century Link managed circuit BBMD4611.  At 2:47 PM MST Systems Administrator created CenturyLink Ticket: # 18158445. Colorado 811 IT Team also began phone communications with Level 3/Century Link NOC at the same time.  Communications Administrator placed notices on social media directing users to Ticket Express and WTE for Emergency and Damage requests.  At 4:35 PM Colorado 811 engaged ATT Call Assurance to route calls to 800-922-1987 to a company cell phone with a pre-recorded message directing users to online services.  At 5:35 PM MST Century Link techs reported that they were on their way to the Network Operations Center to replace a faulty card.  At 6:00 PM a pop-up message was placed on our website to process Emergency Damage tickets on Ticket Express.  IT Team members continued to provide after-hours support alongside members of Operations and Member Relations.  At 7:00 PM MST Century Link/ Level 3 techs reported that they had successfully re-seated a network card at a Century Link/Level 3 facility.  At 8:00 PM MST services were fully restored, and after-hours Damage Prevention Agents resumed normal business duties. 

Operational Impact

Internal Impact – The Service Level and Average Hold Time were not impacted due to the outage. Due to weather and the morning volume of that incident, the impact to ticket volume was not measurable. 

External Impact – Excavators were able to process Locate Requests and Damage Notifications through online applications. 

Response – Cause and Resolution

Incident Resolution Date  02/19/2020 

Root Cause  Faulty network card at Level3/Century Link facility.