With AI having entered the consultation room, governance matters more than ever

Artificial intelligence is no longer a future consideration for healthcare. Across Australia, AI is already moving from pilots into clinical practice – from ambient AI scribes that reduce documentation burden to tools that support clinical decision-making and surface population health risks.

Yet many organisations are still navigating how to deploy these technologies safely at scale. Recent Australian research found that while 60% of healthcare organisations are piloting AI technologies, only 12% have deployed them across multiple care or administrative functions.

AI is a Medical Leadership Challenge

There is a misconception that AI is primarily a technology challenge. In my experience, it is fundamentally a clinical leadership challenge.

Imagine a rural hospital where an AI tool alerts clinicians to sepsis hours before it becomes clinically apparent. The first time it fires, the patient improves. The second time, it is wrong and the patient deteriorates. Who is accountable – the vendor, the clinician or the organisation that deploys it? if responsibility cannot be clearly traced, the issue is not the technology; it is the governance around it.

Through my engagement with healthcare leaders across Australia, I have seen a common theme emerge: accountability becomes harder to define as AI moves into clinical practice.

At a recent Royal Australasian College of Medical Administrators (RACMA) event on AI and Digital Leadership, I spoke about three critical distances where accountability can disappear: between data and decisions, where bias or poor-quality data can influence outcomes; from vendor to governance, where real-world use can drift beyond a tool's intended purpose; and between a tool to clinical judgement, where over-reliance on AI can erode critical thinking.

If those three distances are closed, the one question is answered.

The principles of clinical governance have not changed. We still need good data, capable people, clear accountability, defined escalation pathways and ongoing oversight.

What has changed is the speed, scale and complexity with which decisions can now be influenced. AI amplifies the consequences when governance fundamentals are not strong enough.

Building Trust Through Governance

Trust in AI is not created by algorithms alone. It is earned through strong governance, transparency and accountability.

AI learns from the systems and data we provide. When data is incomplete or unrepresentative, algorithms can reproduce existing biases at scale, which means that data quality is not simply an IT issue; it is a clinical governance and, ultimately, a clinical safety issue.

Healthcare has spent decades developing pharmacovigilance to monitor medicines after they enter clinical practice. We will need the same discipline for AI. More than 10,000 AI-related incidents have been recorded globally since 2014. Governance cannot stop at implementation. It must be continuous.

The Opportunity is Significant. So is the Responsibility

AI has the potential to improve outcomes, reduce inequity, support overstretched clinicians and help health and care systems meet growing demand. But technology alone will never deliver these benefits. Success depends on the governance, culture and accountability that surround it.

On AI Appreciation Day, we should celebrate the progress artificial intelligence is making in health and care settings. But we should also recognise the leadership work happening behind the scenes - by clinicians, governance professionals, digital health leaders and health organisations - to ensure these technologies are safe and implemented responsibly.

Ultimately, the future of AI in healthcare will be shaped by the quality of the leadership, governance and clinical judgement that surround it, not by the technology itself.

The technology is ready. Our responsibility is to ensure our governance is, too.