IT Brief Australia logo
Technology news for Australia's largest enterprises
Story image

Why effective data warehousing is critical in today’s data-driven economy

By Contributor
Tue 17 Nov 2020
FYI, this story is more than a year old

Article by SnapLogic field CTO Brad Drysdale.

Data is the lifeblood of modern business, and the flow and storage of this data is essential to ensuring companies can make critical decisions with the right intelligence. 

Data warehousing is a big part of modern data storage practices, as it enables organisations to bring all of the data they have into one place. By rationalising information in a single data warehouse, businesses can get a much more holistic view of their data estate, which in turn allows them to accelerate internal processes, deliver better analytics, and ultimately save time and money. 

Research from Forrester has shown organisations are investing more of their capital in data warehousing and data lakes than ever before. Yet, even with the increased spend, the process of moving data into a data warehouse or data lake can be plagued with pitfalls and hurdles.

The challenge facing businesses

Data can be challenging to manage, so it’s rare to come across an organisation that doesn’t have data that is missing, duplicated, or stored somewhere unexpected. 

It’s therefore unsurprising that, according to our own research, 83% of IT decision-makers (ITDMs) are not completely satisfied with the performance and output of their data management and data warehousing solutions. 

For many, this dissatisfaction is amplified by legacy technology restricting data movement and increasingly complex data types and formats, as well as the growing number of applications and data sources that organisations are using today. With the average organisation hosting 115 distinct applications or data sources and only half of these (51%) adequately integrated, the data warehousing problem is plain to see. 

Ultimately, the first step in tackling these challenges is to acknowledge the problem and understand how the current data landscape looks. By accepting their data landscape state of play, businesses can then start to build out an action plan to streamline and accelerate change — this should take into account an organisation’s tech stack, in-house skillset, and the future needs of the business.

The role of automation

From our research, 42% of ITDMs stated that they had data management and warehousing processes that could be automated, but which were currently being done manually. This is shocking when considering the clear ability of automation to save IT teams both time and money when managing data and data warehouses. 

Automation can accelerate all stages of data management and data warehousing, including data collection, integration, preparation, storage, sharing, and analysis. This can lead to improved data quality through consistent handling processes and reduced errors, while increasing IT productivity as they have more time to focus on business-critical tasks, ultimately reducing costs and driving higher ROI. 

By introducing automation to this aspect of the organisation, employee time can be redeployed to speed up innovation and deliver better business value. 

Should businesses increase their data management and warehousing budgets?

Despite the uncertain times, our research showed that many IT leaders are boosting their data budgets, with 73% of ITDMs in the study saying they had increased their budgets for data projects by up to 50% in the past year.

However, increased spend doesn’t necessarily mean increased ROI – output is also reliant on the insights that can be extracted from the data, whether the data is flawed, and if the data can be delivered to the right place in time. Ultimately organisations need to take a highly critical look at their data management processes before making those budgetary decisions. 

Deploying successful data warehousing initiatives

Below are seven simple steps all businesses should think about when trying to make their data warehousing projects a success:

Establish a data culture

Teach everyone in the company, from the intern to the CEO, the value of data and what the organisation hopes to achieve. This needs to be hardwired into the way every employee thinks, informing every decision that is made.

Define the data objectives

Building on the previous step, understanding what the organisation wants from data and being clear to the rest of the organisation about goals and desired outcomes is essential. 

Understanding the objectives will help with building the right tech stack, and ultimately support the organisation in creating an effective data strategy.

Assess the existing technology toolbox

Before creating and loading new data warehouses, take stock of the current tools used across the organisation, both on-premises and in the cloud. Work out what needs replacing, updating, or consolidating. 

Define data sources

Take time to identify the different applications and data sources the business is using and where and when they are being used. This will give the organisation a head start on knowing which workflows to streamline, which processes to automate, and which data siloes to integrate.

Integrate, automate, and load the warehouse

Next, it’s time for extracting, transforming and loading (ETL), or extracting, loading and transforming (ELT) from the data sources, systems, and application into the data warehouse. An essential sub-step here is picking a modern, flexible, self-service integration solution which can connect both applications and data from on-premise, cloud, or hybrid environments.

Report and analyse

Rationalise the new data infrastructure by bringing it together in an easy to use interface — this will help the IT team make edits and adjustments as well as report back to management on the data warehousing initiative. The easier data warehousing is to understand, the more people there will be who will be able to contribute to data projects.

Monitor, measure, iterate

Monitoring and editing is the final step. With everything in place, continuous improvement should be straightforward, allowing for the adjustment of data warehousing practices to fit the requirements of the business at the time.

Related stories
Top stories
Story image
Hawaiki Cable
BW Digital completes acquisition of Hawaiki Submarine Cable
BW Digital has completed its full acquisition of Hawaiki Submarine Cable, with all applicable regulatory filings and approvals now received.
Story image
Adyen expands partnership with Afterpay as BNPL payments increase
Adyen has expanded its partnership with AfterPay allowing more of Adyen’s merchants in more countries worldwide to use the BNPL provider.
Story image
Ivanti and Lookout bring zero trust security to hybrid work
Ivanti and Lookout have joined forces to help organisations accelerate cloud adoption and mature their zero trust security posture in the everywhere workplace.
Story image
Atturra partners with Focus HQ to support Aus organisations
Atturra has executed a partnership agreement with Focus HQ, to resell and support the company's Australian developed SaaS-based portfolio management platform.
Story image
Power / Energy
SmartCIC report reveals top five 5G carriers in the world
The Global Cellular Performance Survey also found that 5G networks are delivering high download speeds but lagging in upload speeds.
Story image
Hands-on review: STM laptop bags
The advent of hybrid working has meant we need laptop bags. We got our hands on two of the most popular laptop bags from STM.
Story image
Employee Retention
Company values increasingly important in post-pandemic world
New research released today reveals that company values are the new currency when it comes to employee retention in a post-covid economy.
Story image
HubSpot launches 'The Great Upskill' week to inspire learning
Brands across APAC including Google ANZ, MessageMedia, Meltwater, Seismic and Aircall, will give their employees a full workday to dedicate to upskilling.
Story image
Avaya expands Microsoft partnership to deliver OneCloud on Azure
The joint technology and go to market agreement will help customers accelerate their digital transformation initiatives in the cloud.
Story image
A10 Networks finds over 15 million DDoS weapons in 2021
A10 Networks notes that in the 2H 2021 reporting period, its security research team tracked more than 15.4 million Distributed Denial-of-Service (DDoS) weapons.
Story image
Application Security
What are the DDoS attack trend predictions for 2022?
Mitigation and recovery are vital to ensuring brand reputation remains solid in the face of a Distributed Denial of Service (DDoS) attack and that business growth and innovation can continue.
Story image
Ingram Micro Cloud adds Bitdefender solutions to marketplace
Ingram Micro Cloud has announced the expanded availability of Bitdefender solutions on the Ingram Micro Cloud Marketplace.
Story image
Power / Energy
Keysight Technologies introduces new next-gen DPT solution
Keysight Technologies has announced its new next-generation Double-Pulse Tester (DPT) with the PD1550A Advanced Dynamic Power Device Analyser.
Story image
Kodari Securities (KOSEC)
NFT trends and opportunities: expert reveals all
The NFT market is growing at an exponential rate, with unprecedented liquidity. Here we explore how businesses can profit.
Story image
Tech job moves
Tech job moves - Datacom, Micro Focus, SnapLogic and VMware
We round up all job appointments from May 6-12, 2022, in one place to keep you updated with the latest from across the tech industries.
Story image
CyberArk launches $30M investment fund to advance security
CyberArk has announced the launch of CyberArk Ventures, a $30 million global investment fund dedicated to advancing the next generation of security disruptors.
Story image
Artificial Intelligence
ForgeRock releases Autonomous Access solution powered by AI
ForgeRock has officially introduced ForgeRock Autonomous Access, a new solution that uses AI to prevent identity-based cyber attacks and fraud.
Story image
Avaya OneCloud sees 118% ARR growth for second quarter 2022
Avaya Holdings has reported $750 million annual recurring revenue (ARR) for its OneCloud offering, up 21% sequentially and 118% from the same period last year.
Story image
Tech innovation crucial to growth, but barriers remain
Businesses in the A/NZ region believe tech innovation is crucial to drive future growth, but 76% say they’re being held back by complex data architecture.
Find out how a behavioural analytics-driven approach can transform security operations with the new Exabeam commissioned Forrester study.
Link image
Story image
Remote Working
How organisations can meet employees' changing expectations
The global employment market has shifted dramatically in favour of employees, sparking the so-called great resignation, in which people are leaving unsatisfying roles in search of greener pastures.
Story image
HackerOne launches Attack Resistance Management solution
HackerOne has launched Attack Resistance Management - a new category of security solution that targets the root causes of the attack resistance gap. 
Story image
Microsoft unveils adaptive accessories for disability access
Microsoft is introducing an expansive Inclusive Tech Lab to give people with disabilities greater access to technology through new software features and adaptive accessories.
Story image
HINDSITE wins Aerospace Xelerated Pitch Challenge with solution to support Boeing
Brisbane-based startup HINDSITE was the winner of the first ever Pitch Challenge organised by Aerospace Xelerated in partnership with Queensland XR Hub. 
Story image
Grasping the opportunity to rethink the metrics of a sustainable data centre
A data centre traditionally has two distinct operations teams: the Facility Operations team, and the IT Operations team. Collaboration between them is the key to defining, measuring, and delivering long-term efficiency and sustainability improvements.
Story image
Could your Excel practices be harming your business?
While Excel has been the de-facto standard for budgeting, planning, and forecasting, is it alone, enough to support organisations in the global marketplace that’s facing rapid changes due to digital transformation?
Threat actors are exploiting weaknesses in interconnected IT/OT ecosystems. Darktrace illuminates your entire business and takes targeted action to stop emerging attacks.
Link image
Story image
Video: 10 Minute IT Jams - An update from IronNet
Michael Ehrlich joins us today to discuss the history of IronNet and the crucial role the company plays in the cyber defence space.
Story image
Digital Transformation
Unlocking the next digital frontier for educational institutions
Understanding where to invest in technology can be challenging for education institutions, especially after the COVID-19 disruptions.
Story image
Energy storage demand momentum continues, says BYD
BYD has announced an expansion of its production capacities and will deliver 250,000 units of its energy storage system, BYD Battery-Box Premium.
Story image
Palo Alto Networks says ZTNA 1.0 not secure enough
Palo Alto Networks is urging the industry to move to Zero Trust Network Access 2.0 because previous versions have major gaps in security protection.
Story image
BeyondTrust integrates Password Safe solution with SailPoint
BeyondTrust has announced the integration of BeyondTrust Password Safe with SailPoint identity security offerings.
Story image
Zerto unveils updates to ransomware recovery capabilities
"Organisations face increased risks from the volume and sophistication of ransomware attacks prevalent today."
Story image
Cybersecurity starts with education
In 2021, 80% of Australian organisations responding to the Sophos State of Ransomware study reported being hit by ransomware. 
For every 10PB of storage run on HyperDrive vs. comparable alternatives, an estimated 6,656 tonnes of CO₂ are saved by reduced energy consumption alone over its lifespan. That’s the equivalent of taking nearly 1,500 cars off the road for a year.
Link image
Story image
Data Center
Preventing downtime costs and damage with Distributed Infrastructure Management
Distributed Infrastructure Management (DIM) can often be a lifeline for many enterprises that work with highly critical ICT infrastructure and power sources.
Story image
Google reveals new safety and security measures for users
Google's new measures include automatic two step verification, virtual cards and making it easier to remove contact information on Google Search results.
Story image
Sift shares crucial advice for preventing serious ATO breaches
Are you or your business struggling with Account Takeover Fraud (ATO)? One of the latest ebooks from Sift can provide readers with the tools and expertise to help launch them into the new era of account security.
Story image
Power at the edge: the role of data centers in sustainability
The Singaporean moratorium on new data center projects was recently lifted, with one of the conditions being an increased focus on power efficiency and sustainability.
Story image
Employee Experience
Zendesk launches customer service and employee experience offering
"Zendesk is helping businesses embrace this new generation of conversational customer relationship management and turn customer service into growth.”
Story image
Prophecy International migrates VMware environment with Oracle Cloud Solution
The Adelaide-based global provider is using the solution to eliminate the need to re-write applications, therefore allowing the company to enhance its business operations.
Story image
ThoughtLab reveals 10 best practices for cybersecurity in 2022
The benchmarking study reveals best practices that can reduce the probability of a material breach and the time it takes to find and respond to those that happen.
Story image
Telstra and Silver Trak Digital delivers 5G to the cinemas
Telstra and Silver Trak Digital say they've launched Australia's fastest and most secure delivery of content over 5G for cinemas.
Story image
Artificial Intelligence
Updates from Google Workspace set to ease hybrid working troubles
Google Workspace has announced a variety of new features which will utilise Google AI capabilities to help make hybrid working situations more efficient and effective.