I’ll put my hand up: I’m in that group sometimes. There have been moments where I’ve had a clear read on something, and AI has come back with something that contradicts it. That hesitation is worth paying attention to. It’s your gut telling you something might be off. And the instinct to push back is the right one. It’s exactly what you’d do with any analyst who handed you a number that didn’t feel right.

Here’s what makes AI different from a human analyst: it doesn’t signal uncertainty. A human might caveat a number, look slightly uncomfortable when you push on it, or admit they’re not sure. AI stays just as confident regardless of what’s underneath the answer. There are no cues to read. A wrong answer and a right answer look identical on a slide.

The SAP-sponsored Wakefield Research survey shows that 44% of 300 US executives surveyed (at companies with over a billion dollars in annual revenue) would trust generative AI to override a decision they were planning to make. Leaders everywhere are trusting output they have no reliable way to verify.

That’s the actual problem.

Why AI gets things wrong — and what a semantic layer has to do with it

AI doesn’t make things up randomly. When it gets something wrong, there’s usually a structural reason.

One of the most common is a missing or poorly built semantic layer.

Your business has rules. Revenue doesn’t just mean money coming in. It might mean recognised revenue, contracted revenue, or whatever your finance team uses to close the month. A “customer” might mean an active account, or anyone who’s ever transacted, or only those who transacted in the last 12 months. These definitions live somewhere: in dashboards, in databases, in people’s heads.

A semantic layer takes all of those definitions and stores them in one place, so that when AI queries your data, it’s using your rules rather than guessing at them. Think of it as a translation layer built specifically for your organisation.

When that layer is well-built, AI accuracy improves significantly. When it’s missing or out of date, the AI produces answers that are technically coherent but factually wrong for your business context. It sounds right. It looks right. It isn’t right.

And building it isn’t a one-off activity. It needs to be maintained as your business changes, and it needs a feedback loop: when your gut tells you something’s off, that gap needs to go back to whoever owns the semantic layer so it can be addressed.

Three things worth doing now

Most conversations about AI governance focus on policy documents and risk frameworks. This is more practical than that.

First — ask whoever manages your data systems one question. What does the semantic layer look like, and who owns it? You don’t need to understand the technical answer. You need to hear whether there is one. The quality of that answer tells you a lot.

Second — apply the same scepticism to sources and references that you’d apply to numbers. AI cites things with the same confidence it gives you a revenue figure, whether those sources exist or not. In 2025, the AFR reported that Deloitte delivered an A$440,000 report to the Australian government’s Department of Employment and Workplace Relations. It was later found that the report had contained fabricated academic citations, references to non-existent legislation, and a quote attributed to a federal court judge that never existed. Deloitte repaid $98,000 of its fee. The report had been written using Azure OpenAI.

Now admittedly, I also use AI to help articulate my own thinking, including for this podcast. But anything that goes out and will influence a decision gets verified first.

Third — give yourself permission to push back. If an AI output doesn’t match your read of the business, that instinct is worth something. Feed it back to your team so they can address it. That’s how the system gets more accurate over time. Trust in AI is built the same way gut instinct is — through accumulated experience, feedback, and iteration. Not overnight.

The executives who use AI well aren’t the ones who trust it most. They’re the ones who stay in the loop long enough to know when to push.

Want to understand where AI governance is already a live issue in your business?

We help Australian organisations map their current AI usage, identify the highest-exposure workflows, and build governance that fits how people actually work — not how policy documents assume they do. See our services for more information or get in touch!