itb-au logo
Story image

The winning formula for successful DataOps

28 Jul 2020

Article by Qlik APAC vice president of data integration, Byron Yeung.

When the COVID-19 crisis hit earlier this year, organisations were forced to quickly reinvent how they operate. Today, many businesses have accelerated their digital transformation and quickly embraced new technologies.

As a result, IT demand has grown across areas such as cloud and data analytics and reduced across legacy categories. This makes it an ideal time for IT teams to streamline the development and delivery of data to better support business operations.

Enter DataOps. Similar to DevOps, DataOps is an emerging set of practices, processes and technologies for building and enhancing data analytics pipelines to better meet the needs of the business.

The method promises improved productivity, streamlined and automated processes, increased data output and greater collaboration across teams. Ultimately, DataOps accelerates time to insight and solves the many challenges associated with data availability. So, what does it look like in practice?

Establishing the right foundation

While data managers, architects and engineers are embracing new cloud and data lake initiatives to provide a more scalable and agile infrastructure, they need to avoid the mistakes of first-generation big data projects that fail to deliver real business value and become large landing zones for corporate data.

Agile organisations require modern data architectures that allow for the quick use of the ever-expanding volumes of data. These infrastructures – based on hybrid and multi-cloud for greater efficiency – provide enterprises with the flexibility they need to compete more effectively, improve customer satisfaction and increase operational efficiencies.

When the DataOps methodology is part of these architectures, companies are empowered to support real-time data analytics and collaborative data management approaches while easing the many frustrations associated with access to analytics-ready data.

Adding the key ingredients for DataOps success

Before shifting to modern analytics and data management platforms in the cloud, organisations should first evaluate their legacy integration technology to ensure it can support the key DataOps principles that will accelerate time to insight.

Here are four considerations to ensure DataOps success:

Maintain continuous integration 

Continuous integration is foundational to modern data platforms and to meet the needs of real-time analytics. It requires individuals to think differently about integration, it can no longer be a batch process that only happens at certain points in time and impact transactional systems.

Applying technologies such as change data capture (CDC) provides a non-invasive method to catch data and metadata changes from transactional systems, relational databases and applications and stream them in real-time to the analytic platform.

Ensure solutions are universally applicable

Seek solutions that support a broad variety of sources and targets and span multiple integration use cases. For example, you should only capture changed data and route that data where and when it’s needed: to a data lake; or a data warehouse; or replicate it to another database; etc. This provides greater efficiency and fewer moving parts.

Automate your processes

It’s always a challenge to find the right people and skill sets to meet the everchanging technology landscape. Automation becomes essential to the success of a DataOps initiative, from the generation of change data streams to delivery to refinement and finally the creation of analytics-ready data sets.

This becomes particularly important for data lakes where heterogeneous data is coming in a variety of formats and is continuously updated with new change files.

Apply an agile approach 

Being able to embrace and adopt new technologies and implement new data pipelines is vital due to the continuous change in data formats, platforms, technology and data loads. An enterprise must be able to quickly pivot or it can be left behind the competition.

Flexible integration solutions allow for IT to simply change a source or target without disrupting the entire infrastructure – providing an agile, modern infrastructure that future proofs an enterprise.

With COVID-19 forcing organisations to embrace transformation in order to continue operating, having the right information and analysis could mean the difference between leading the market or ceasing to exist.

This is further shown in a new global study conducted by IDC, where a clear association was identified amongst companies that were more capable of turning data into insights for decision-making. In fact, the Australian business leaders surveyed pointed to a significant gain to the bottom line from investing in a strong data pipeline.

DataOps is improving the way that data is being made available across the enterprise and is the key to better business insights and a competitive edge. 

Link image
Legacy systems left in the dust as enterprises turn to the three Cs of modern work
With fundamental changes in the nature of the modern workplace, cloud is the obvious choice for providing a flexible work environment.Watch Now
Story image
Why IT and HR must work together to help businesses weather the storm
Employers are striving to balance team productivity, security and employee engagement. If remote work is the new norm, it’s impossible to ignore the challenging nature of the situation, writes Gigamon manager for A/NZ George Tsoukas.More
Link image
The importance of data resilience in the current cybersecurity climate
Protecting an organisation's data is one of the most crucial functions of any CISO. Strategies should be in place where data is stored securely and cost-effectively.More
Story image
SOC as a Service: Fortinet’s answer to today’s network challenges
Jon McGettigan, Fortinet A/NZ Regional Director, explains how SOC as a Service can back up your current SOC team, fast-track deployments and ensure regulatory compliance.More
Story image
Nokia to migrate all on-prem IT infrastructure to Google Cloud
Nokia’s infrastructure and applications will operate in the public cloud or in a software-as-a-service model from now, and the company expects the extensive migration to take between 18 and 24 months to complete.More
Story image
Security and operations collaboration key to success post COVID-19
“We are in an ultra-hybrid world with multi-everything, and in order to successfully navigate this landscape, ITOps, DevOps, and SecOps teams need to more closely align."More