IT Brief Australia logo
Technology news for Australia's largest enterprises
Story image

Cloudflare outage in 19 data centers worldwide due to own error

By Jessie Chiang
Wed 22 Jun 2022

Content delivery network Cloudflare says its outage for 19 of its data centers yesterday was because of a change in a long-running project to increase resilience in its busiest locations.

Many sites went down as a result, including Discord and Shopify. The company says even though these locations, which include Mumbai, Osaka, Singapore, Sydney, and Tokyo, are only 4% of its total network, the outage impacted 50% of total requests. 

Cloudflare says the first outage started at 06:27 UTC on June 21. At 06:58 UTC, the company brought the first data center back online, and by 07:42 UTC, all data centers were online and working correctly.

In a statement on its blog, Cloudflare says, "We are very sorry for this outage. This was our error and not the result of an attack or malicious activity."

The company says over the last 18 months, it has been working on converting all of its busiest locations to a more flexible and resilient architecture. During this time, Cloudflare converted 19 of its data centers to this architecture, internally called Multi-Colo PoP (MCP).

Cloudflare says this new architecture provides significant reliability improvements, allowing it to run maintenance in those locations without disrupting customer traffic. But those locations also carry a substantial proportion of the Cloudflare traffic, so any problem there can have a wide impact.

In order to be reachable on the internet, networks like Cloudflare make use of a protocol called BGP. As part of this protocol, operators define policies that decide which prefixes (a collection of adjacent IP addresses) are advertised to peers (the other networks they connect to) or accepted from peers.

Cloudflare says these policies have individual components, which are evaluated sequentially. The result is that any given prefixes will either be advertised or not advertised. A policy change can mean a previously advertised prefix is no longer advertised, known as being "withdrawn", and those IP addresses will no longer be reachable on the internet.

The company says while deploying a change to its prefix advertisement policies, a re-ordering of terms caused it to withdraw a critical subset of prefixes.

Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. However, the company says it has backup procedures for handling such an event and used them to take control of the affected locations.

But F5 Labs threat research lead David Warburton says this should be a reminder of the dangers of the centralisation of large cloud solution providers.

"In a traditional internet app deployment model, an outage of a server or misconfigured application might take out a single website," he says.

"But similar problems with a cloud solution provider can end up taking out all of their customers, resulting in not one website being taken offline, but hundreds or thousands. The impact can potentially affect organisations' digital experiences, revenues and reputations." 

Warburton says cloud solution providers provide immeasurable benefits to their users, but the re-centralisation of the internet through these cloud solutions is now causing the very problems the original design of the internet was intended to avoid through redundancy. 

ServerChoice commercial director Adam Bradshaw says outages can cause businesses serious reputational and financial damage. 

"For a smaller or medium-sized business, the risk associated with such failures in service can be critical. Even short outages in the digital economy could cause damage," he says.

"Diversifying an IT environment reduces the likelihood of third-party outages negatively impacting a business. Owning the hardware gives an organisation more control, with IT components such as colocation services providing an organisation with a backup when cloud services fail."

Related stories
Top stories
Story image
Microsoft
Elders signs five-year agreement with Microsoft to boost innovation
Australian agribusiness Elders has signed a five-year agreement with Microsoft that looks to transform its customer experience, efficiency and sustainability outcomes.
Story image
Ransomware
Examining the future of ransomware threats with Vectra’s CTO
As customers' valuable data move to the cloud, so will ransomware. What is the current landscape and what do we need to know?
Story image
Apple
Your tools, your choice: why allow employees to choose their own devices?
Jamf Australia says giving your team the freedom to work with their digital device of choice could help to attract and retain top talent in a tight labour market.
Story image
AGVs
Bridgestone Australia uses Dematic's AGVs to optimise warehouse operations
Bridgestone Australia has deployed Dematic's Automated Guided Vehicle solution across its new Melbourne warehouse in Truganina.
Story image
Sustainability
Honeywell named Frankston facility services provider
Honeywell has been named the joint facility services provider for Frankston Hospital’s AU$1.1 billion redevelopment.
Story image
Tech job moves
Tech job moves - Bitdefender, Cohesity, Fortinet & MODIFI
We round up all job appointments from June 27-30, 2022, in one place to keep you updated with the latest from across the tech industries.
Story image
Aspire
NEC expands Open RAN ecosystem with Aspire Tech acquisition
With its agreement to acquire Aspire Technology, NEC Corporation has further increased its capacity to deliver End-to-End Open RAN ecosystems.
Story image
Security Information and Event Management (SIEM)
LogRhythm updates SIEM Platform with latest innovations
LogRhythm has announced the launch of version 7.9 of the LogRhythm SIEM Platform and updates to LogRhythm NDR and LogRhythm UEBA.
Story image
Document Management
NZ's FileInvite raises $10M in latest investment round
FileInvite has raised $10 million in Series A investment to fast-forward the extinction of email for requesting and collecting documents online.
Story image
Cybersecurity
Without trust, your security team is dead in the water
The rise of cyberattacks has increased the need for sound security that works across any type of business, but with any change, buy-in is essential. Airwallex explains why.
Story image
Documentation
Adobe study finds lack of digital trust and utilisation in Australian Government agencies
New research commissioned by Adobe has revealed a significant lack of digital trust within Australian Government departments, along with the continued underutilisation of key digital processes.
Story image
Payroll
How New South Wales state departments achieved cloud migration success
State departments in New South Wales are heading to the cloud to achieve better workflow solutions, and one company is paving the way for their success.
Story image
Metaverse
How the metaverse will change the future of the supply chain
The metaverse is set to significantly change the way we live and work, so what problems can it solve in supply chain management?
Story image
Data Protection
Five signs your business is ready to move to the cloud
Many organisations are thinking about moving to the cloud. But what are the signs you are ready, and what are the reasons to move?
Story image
CSG
To win at 5G, telcos must tame their quoting chaos
The catalogs of CSP (communication service providers) market offerings are set to explode as new digital services emerge, powered by B2B2X business models.
PwC
WSLHD and PwC’s Consulting Business came together to solve through the challenges of COVID-19. A model of care was developed to the NSW Health Agency for Clinical Innovation guidelines with new technology platforms and an entirely new workforce.
Link image
Story image
Cybersecurity
Tech and data’s role in the changing face of compliance
Accenture's study found that 93% of respondents agree or strongly agree new technologies such as AI and cloud make compliance easier.
Story image
IDTechEx
The next stage for 5G in thermal materials - IDTechEx
IDTechEx says higher frequency deployments, such as mmWave devices and very different station types such as small cells, present their own technological evolution and, with it, thermal challenges. 
Story image
Remote Working
RDP attacks on the rise, Kaspersky experts offer advice
"Given that remote work is here to stay, we urge companies to seriously look into securing their remote and hybrid workforce to protect their data."
Productivity
Discover the 5 ways your ERP may be letting you down. Is your current system outdated, difficult to manage, and costing you a fortune?
Link image
Story image
Low-code
Appian unveils low-code certification program in Australia
Appian has announced a program to provide the next generation of low-code developers with access to education on the subject and certification to foster career opportunities.
Story image
Adobe
Marketplacer and Adobe accelerate partnership for enhanced commerce solutions
Marketplacer has accelerated its partnership with Adobe in order to further enhance the global commerce marketplace.
Story image
Airwallex
How Airwallex helps businesses achieve globalisation success
As markets continue to shift, businesses need to be able to provide the same quality of service for customers regardless of where they are located around the world.
Story image
Wiise
Four things wholesale distributors need to consider for FY2023
In a post-pandemic world, there are many things for a distribution business to juggle. ERP solutions company Wiise narrows down what companies should focus on.
Story image
Collaboration
Enterprise service management: the importance of a one-stop shop
In an online world, employees and end-users want one place to go for all their questions and requests. Intranet technology and self-service portals are useful tools that help serve this purpose.
Story image
Accounting
Four factors to consider when choosing the right job accounting solution
Progressive job-based businesses can achieve success by strengthening their ability to quantify every cost attributable to the delivery of an outcome for a customer.
Story image
Management
MYOB snaps up Sydney-based management software specialists
MYOB has announced the acquisition of Sydney-based business management software and support specialists, GT Business Solutions.
Story image
Compliance
SentinelOne integrates with Torq to empower security teams
"With Torq, security teams can extend the power of SentinelOne to systems across the organisation to benefit from a proactive security posture.”
Story image
Artificial Intelligence
Accenture shares the benefits of supply chain visibility
It's clear that gaining better visibility into the supply chain will help organisations avoid excess costs, inefficiencies, and complexity to ultimately improve their bottom line.
Story image
Ransomware
Businesses unprepared to defend against ransomware attacks
Ransomware attacks continue to impact organisations worldwide with high costs, but businesses are still largely unprepared.
Supply chain
Discover the 4 critical priorities for wholesale distribution businesses in FY23. Are you worried about how supply chain issues may affect your business in 2023?
Link image
Digital Transformation
Discover the 5 signs your business is ready for a cloud-based ERP. Is your business being left behind as more of your competitors switch to the cloud?
Link image
Story image
Infrastructure
Video: 10 Minute IT Jams - An update from Paessler
Sebastian Krüger joins us today to discuss how unified infrastructure monitoring enables MSPs to seamlessly deliver services to their clients.
Story image
Enterprise Resource Planning / ERP
Five ways your ERP is letting you down and why it's time for a change
Wiise explains while moving to a new system may seem daunting, the truth is that legacy systems could be holding your business back.
PwC
PwC's Consulting Business and PwC's Indigenous Consulting are proud to play an important role in helping Australian Indigenous Mentoring Experience build IMAGI-NATION, a free online university for marginalised communities around the world.
Link image
Story image
Artificial Intelligence
Decision Inc. partners with provenio.ai to expand offering
Decision Inc. Australia has partnered with provenio.ai to expand its offering to clients in the retail, FMCG, manufacturing, supply chain and logistics sectors.
Story image
API
Industry-first comprehensive risk-based API security enhances protection
Application Programming Interfaces (APIs) have become a crucial part of operating web and mobile application businesses and are causing significant economic growth in the digital sector.
Story image
Artificial Intelligence
Siemens expands NVIDIA partnership for industrial metaverse
Siemens is expanding its partnership with NVIDIA to enable the industrial metaverse and increase the use of AI-driven digital twin technology.
Story image
Training
Barracuda customises training to fit ACSC Essential Eight
Barracuda has announced that its Security Awareness Training now provides a customised training curriculum in line with the Australian Cyber Security Centre’s (ACSC) Essential Eight.
Story image
Artificial Intelligence
Salesforce announces new innovations for financial services
Salesforce has launched expanded financial services that offer more targeted and trusted automation to help teams unlock insights, deliver better customer service, and drive operational efficiencies.
Story image
Media
Registrations for the W.Media Sydney Cloud and Datacenter Convention 2022 now open
Are you a C-Level executive looking to enhance your knowledge in the cloud and data center space in order to get the best results for your company?
Project management
Discover the 4 crucial factors for choosing the right job-costing solution. Is your team struggling to cost jobs and keep projects running on budget?
Link image
Story image
Artificial Intelligence
Eight top DevSecOps trends to support IT innovation in 2022
The use of DevSecOps practices is growing, as it is increasingly seen as the best way to produce high-quality and secure code. So what are the current trends?
Story image
Samsung
Monitors are an excellent incentive for getting employees back
The pandemic has taught us that hybrid working is a lot easier than we would’ve thought, so how can the office be made to feel as comfortable as home? The answer could be staring you in the face right now.