WHAT CAUSED THE OUTAGE?
* Telstra’s July 8 outage began at 2:50am following a planned NTP (Network Time Protocol) timing chassis change, according to an independent investigation into what triggered the network failure
* When the chassis came back online, its internal GPS card had reset and began sending an incorrect date – 2006 – into the network
* To prevent the issue, the GPS card required a firmware update, stipulated in bulletins sent by the supplier
* While the chassis change triggered the outage, steps leading up to it go as far back as 2020
WHAT WERE THE IMPACTS OF THE OUTAGE?
* At its peak, the outage impacted 45 per cent of all Telstra calls and data sessions,
* Some users were unable to make or receive voice calls including Emergency Calls, send or receive messages, or access mobile data
* Telstra undertook more than 630 welfare checks on customers prevented from having triple-zero calls connected
WHAT WENT WRONG AT TELSTRA?
* Alarms in Telstra’s system were not recognised as signifying potential broader network issues.
*Telstra failed to recognise and prioritise network time protocols (NTP)
* The department responsible for the failure point had not sufficiently modernised and maintained adequate support for NTP in part due to budgetary constraints
* A lack of competency, curiosity and technical expertise meant problems arose, but no one investigated them
* Two engineers who were involved in the planned change were on mandatory standdown when the outage was noticed
* Despite the change associated with the outage being seen as high risk, no pre-checks of equipment and software were undertaken
*After the change, no testing was done to check key functions were working correctly
KEY QUOTES
* “It’s not a matter of not having sufficient engineering and network experience. It’s that we did not prioritise this particular capability high enough.” – Telstra chief executive Vicki Brady
“It’s not about not enough resources. It’s not about not enough money. It’s about the prioritisation.” – Vicki Brady
Australian Associated Press is the beating heart of Australian news. AAP is Australia’s only independent national newswire and has been delivering accurate, reliable and fast news content to the media industry, government and corporate sector for 85 years. We keep Australia informed.
