Addressing the 4 key challenges of designing a data and AI strategy

According to a report by The Tech Council of Australia and Microsoft, AI could add $115 billion to the Australian economy annually by 2030. Yet, business leaders who are looking to adopt a data-first strategy for developing new products or improved experiences are facing significant challenges. First is the increasing pressure and demands from customers to invest in new technology, such as artificial intelligence (AI), to better understand their needs and provide a more personalised service. Secondly, the requirement to satisfy stricter and increasingly more complex data security, data governance and regulatory requirements at the same time.

To navigate these challenges, there are a number of critical steps a data-driven business must take. At the centre and as a necessary first step, they need to choose a modern data and AI cloud platform with governance, security and compliance baked into its very foundation. By leveraging such a data platform, businesses can focus on their core competencies, accelerate insights from their data and create new end-user experiences. 

The compliance challenge

Increasing regulatory requirements are becoming front and centre for any modern, data-first business strategy. This is true for all businesses, including those looking to scale within a single geography or regulated industry, and even more so for global businesses looking to operate and scale across different geographies and diverse industries. Regions such as the U.S., Europe and Asia have introduced numerous regulations as it pertains to data management and data governance that organisations must adhere to. 

For example, in the US, there is no singular law that covers the privacy of all types of data. Instead, it has a mix of laws that are designed to target specific states or specific types of data. The California Consumer Privacy Act (CCPA) gives California residents increased transparency and control over how businesses collect and use their data, while the Gramm-Leach Baily Act (GLBA) covers consumer financial products and requires companies to explain how they share data. In the EU, businesses must adhere to the GDPR, and in Asia, organisations have to comply with the Information Technology Act. 

An integrated, consistent data governance strategy

In order to navigate continuously evolving regulatory requirements, organisations must implement a data governance framework that empowers them to discover, understand and protect their heterogeneous data while leveraging it securely to collaborate internally and externally. 

An effective, all-encompassing data governance strategy will enable organisations to store and manage personally identifiable information (PII) and other sensitive data securely while monitoring and protecting that data in near real-time. This includes modifications as well as new incoming sensitive data without the need to manually intervene and adjust existing secure workflows. 

The emergence of modern data applications paired with the need to enable global collaboration poses an additional complex challenge to governance and security. A modern data platform allows its users to seamlessly integrate, apply and enforce the aforementioned core security and governance platform capabilities. It accelerates modern and global data application development and enables application builders to focus on their core competencies and monetise opportunities with peace of mind.

Future-proofing a data strategy

A scalable and efficient data governance strategy must also be forward-looking. Technology is advancing rapidly, making it more challenging for organisations to keep pace with security and governance advancements. Therefore, businesses need to think about the foundations and frameworks that will apply to technology in the years to come. Take generative AI and the emerging large language models (LLMs) as recent popular examples. Over the past few years, AI has swiftly become a crucial aspect of modern life, transforming the way we live, work and interact with each other, with many believing it will be one of the most profound technology shifts seen in our lifetimes. Organisations need to stay nimble, with a security and governance framework that can easily adapt to such innovations. 

As more businesses leverage LLMs, the models will adopt more sensitive and private data to learn from. The laws and regulations around data use and governance are expected to keep changing, and companies need to be in a position to respond with ease and at scale via proper automated security and compliance workflows. Increased automation around security and compliance reduces the likelihood of causing disruption to existing processes, products, and experiences, while the absence of manual and error-prone intervention decreases the risk of security and compliance violations. 

Scaling compliance across data teams

To scale a data strategy, businesses must have the right teams in place who work together effectively. Typically, there are domain experts in security, governance and compliance working with IT, and data teams who are responsible for modernising the tech stack. Often, these groups are separate, with a different level of understanding around security and compliance, which can often result in friction with data modernisation initiatives. For example, data platform teams may be keen to adopt the latest technologies to keep pace with the competition or build new products and experiences, but the compliance team might be reluctant due to potential violations of existing regulatory requirements. 

We are seeing a trend where forward-looking, data-driven organisations ensure that both these teams work very closely together so that they can share an understanding of these challenges and arrive at the right solution in order to scale the strategy with minimal disruption. Data stewards and compliance officers are deeply embedded within data platform teams and closely collaborate with data architects.   

Bringing it all together - a modern data platform secure and compliant by design

A scalable data governance framework alongside automated workflows and collaboration requires the right underlying data platform with the following key characteristics: 

• multi-cloud enabling organisations to apply security, governance and compliance consistently across different technology stacks and services; 
• high degree of automation, allowing users to easily scale and reduce the likelihood and exposure to security and compliance violations; 
• ensuring secure and compliant collaboration on heterogeneous data and modern data applications. 

A modern data platform must offer a first-class search and discovery experience, including the ability to manage unorganised data independent of its data format, such as structured or unstructured. It needs to scale with data volumes and data changes by ensuring automated classification of all data assets. This should feed into scalable and consistent data access policies the data steward can define and enforce. These are all core security and governance platform capabilities.