Quest study finds hidden tax on ad-hoc data delivery

Quest Software has published a study on data delivery practices among large organisations in Australia and Singapore. The survey found widespread reliance on single-purpose data work despite broad adoption of data products.

The research surveyed 154 chief data and analytics officers and senior data leaders at organisations with more than 500 employees and annual revenue above USD $100 million. It found that 97 per cent of respondents said their organisation delivers data products, but only 25 per cent said they run a structured data products programme with standardised, repeatable workflows.

Across the sample, three-quarters of organisations said they operate in an unstructured, ad-hoc way, delivering single-use data products rather than reusable assets. According to Quest, this creates a "hidden tax" on data teams, with staff spending much of their time rebuilding work for individual projects instead of reusing approved, trusted data products.

That strain on capacity was a recurring finding. Among organisations without structured data programmes, 45 per cent said between a quarter and a half of their data team's effort goes to non-reusable projects, while a further 17 per cent said more than half of team capacity is taken up by one-off work.

The survey also found that 61 per cent of respondents said around half of their projects are built for a single purpose only. Another 32 per cent said this happens frequently across many projects.

Many respondents linked more structured delivery to faster execution. Some 62 per cent said delivery would be faster if projects were built as reusable data products, while 37 per cent expected improvements of 40 to 50 per cent.

Susan Laine, chief data technologist at Quest Software, said boards and senior executives are pushing ahead with artificial intelligence projects even though many organisations still lack strong data foundations.

"C-level leadership wants to move faster in the age of AI, and boards are backing AI initiatives, but the underlying data foundations in many organisations have not caught up," Laine said.

"Data teams are delivering impressive outcomes, yet a significant share of their effort is still tied up in individual, one-off data projects that will never be reused. The result is a hidden tax on capacity at exactly the time they need it most."

AI pressure

The findings point to a gap between AI plans and current operating models. Forty-four per cent of respondents said they have active AI or generative AI projects under way that depend on foundational data preparation, while 45 per cent said they are experimenting with pilots that are not yet in production.

Even so, most organisations still work through ad-hoc or mixed models that combine structured and unstructured practices. The report suggests these arrangements can slow data delivery just as demand for trusted data rises.

Compliance is another factor shaping those decisions. The study found compliance can absorb up to 30 per cent of IT team budgets, leaving less available for development work.

While many respondents said they were confident about meeting compliance requirements, confidence was weaker when it came to demonstrating that compliance. Quest linked the gap to a lack of auditable processes, clear data lineage and documentation that can stand up to scrutiny.

Governance reviews and approvals were cited by 59 per cent of ad-hoc operators as the main bottleneck slowing data project delivery. That share was notably higher than among organisations with more structured data programmes.

Change barriers

The obstacles to moving away from one-off work were not chiefly financial, according to the survey. Respondents ranked skills, culture and competing priorities above budget as the main barriers to structured delivery, pointing to a management and organisational challenge as much as a technical one.

Laine outlined how Quest defines a structured data product.

"A data product is a complete package: the data itself, the logic behind it, the quality indicators, the business context and the guardrails that define how it should be used," Laine said.

"Once that exists, new use cases move faster because the hard work has already been done in a repeatable way."

Practitioners interviewed for the study described the operational problems that can emerge when teams create data outputs for one department without shared enterprise definitions. Brendan Mathias, director of analytics and data science at Cochlear, said local solutions can still lead to disagreement and duplication when used across a wider organisation.

"I have experienced a couple of domain-centric data products which satisfy operational reporting needs. For example, sales and marketing vary in attribution rules for a sales order," Mathias said.

"Within the domain, the attribution rules are agreed and reporting is trusted, but in enterprise decision forums there can be disagreement on the reported numbers because there is no shared definition. This can lead to duplicate work as organisations move to build enterprise data products with aligned attribution rules across domains."

Rakesh Menon, data evangelist at Maybank Singapore, also pointed to staffing and judgment as central to any shift in operating model.

"Organisations need an experienced data team able to recognise the need for structured data approaches, with the knowledge to know when to use them appropriately and how to establish and develop component technologies," Menon said.

Laine said organisations must decide whether to continue with fragmented data work or redesign data management practices around reusability and governance.

"Organisations can continue absorbing the hidden tax of one-off work, duplicated effort and fragmented governance, or use 'AI for AI' to reimagine the data management best practices needed to make structured, reusable delivery the norm."