Well Written Is Not Enough: Why AI Output Must Be Checked
The unchecked version looked better.
That was the uncomfortable part.
Yesterday, we ran an AI Leadership Session with the executive, senior leaders and managers of a Primary Health Network, as part of their Data & AI Capability Program, From Strategy to Practice. The task was simple. Leaders were given two versions of the same one-page briefing.
One had been properly checked.
The other was unchecked AI output.
The unchecked version was smoother. It read well. It had a confident tone, tidy structure and clean phrasing. At a glance, it looked like the kind of document a busy leader might approve and move on from.
But it had problems.
A percentage was wrong. A source was made up. The recommendation was vague. It missed the local detail that mattered most.
The group picked it quickly, which was reassuring. But the more useful part came after that. The discussion moved from “Can we spot the bad one?” to “What should we ask before signing off any AI-assisted work?”
That is where AI governance becomes real.
Not in a policy document sitting on an intranet. Not in a broad statement about responsible use. It becomes real in the moment someone is about to rely on an AI-assisted briefing, report, email, analysis or recommendation.
Well written is not the same as checked.
Polished writing can hide weak thinking
AI tools are very good at producing text that sounds complete. They can create a neat paragraph, a confident summary and a recommendation that appears sensible. That fluency is useful, but it can also create risk.
A polished document can make a weak argument look mature.
In the leadership session, the unchecked briefing had four problems that matter in almost any organisation, and matter even more in health.
It included a wrong percentage. That kind of error can shape the way a reader understands scale, urgency or performance. A small numerical error may seem harmless, but numbers often carry authority. People make decisions around them.
It referred to a made-up source. This is one of the known risks of generative AI. A model can produce a reference that looks plausible but does not exist, or it can attribute a claim to the wrong source. If no one checks it, the false source becomes a credibility problem.
It gave a vague recommendation. This is common in unchecked AI output. The language can sound safe and reasonable, while avoiding the hard work of choosing a path, naming trade-offs or giving enough detail for action.
It missed local context. For a Primary Health Network, local detail is not decoration. It shapes whether a recommendation is practical, fair and relevant. A generic briefing may mention access, workforce, demand or patient experience, but still miss the local realities that change what should happen next.
That is the core issue. AI can produce a document that looks ready before the thinking is ready.
This is why leaders need to separate style from substance. A briefing can be clear and still be wrong. It can be concise and still be shallow. It can be persuasive and still be unsupported.
A useful test is to ask what would remain if the fluent writing were stripped away.
Is there evidence? Is there judgement? Is there local knowledge? Is there a clear line between what is known, what is assumed and what still needs checking?
If the answer is no, the document is not ready.
The sign-off questions that matter
The most valuable outcome from the session was not the ability to identify the weaker briefing. It was the agreement on the questions leaders should ask before signing off AI-assisted work.
These questions are practical because they do not require everyone to become a technical expert. They make responsibility visible.
What sources is this based on?
Which parts did AI draft, and what did you change?
How did you check the numbers?
Has it dealt with the complexity, or only the surface?
Could you explain this without the AI output in front of you?
Each question does a different job.
What sources is this based on?
This question forces the work back to evidence. If a briefing makes a claim, the source should be clear. If the source cannot be found, it should not be used. If the evidence is uncertain, that uncertainty should be stated.
This is especially important when AI tools summarise or combine material. A summary may sound authoritative, but the original material still matters. Leaders need to know whether the work draws on approved internal documents, current guidance, local data, public reports or something less reliable.
Which parts did AI draft, and what did you change?
This question moves the conversation away from vague disclosure and towards practical transparency.
It is not enough to say, “AI was used.” The more useful question is how it was used. Did it draft the whole document? Did it summarise notes? Did it help reword a recommendation? Did it generate options? Did it assist with structure?
The answer helps reviewers understand where to focus their checks.
If AI drafted the evidence summary, sources need close attention. If AI helped write the recommendation, the logic and local fit need close attention. If AI only helped tidy language, the risk is different.
How did you check the numbers?
Numbers need their own check. Not a general sense-check. A real check.
That means going back to the original source, recalculating where needed and confirming that percentages, dates, totals and comparisons are accurate. It also means checking whether the number is being used in a fair way.
A figure can be technically correct and still misleading if it lacks context.
Has it dealt with the complexity, or only the surface?
This question protects against generic answers.
AI tools often produce broad, sensible-sounding statements. In health and social care, broad statements are rarely enough. Patient needs, workforce limits, funding rules, geography, community trust, lived experience, data quality and clinical risk can all change the answer.
A briefing that does not deal with complexity may still be useful as an early draft. It should not be treated as finished advice.
Could you explain this without the AI output in front of you?
This may be the strongest leadership question of all.
If the person signing off, presenting or recommending the work cannot explain the reasoning without reading the AI-generated text, they do not yet own the work.
That does not mean they need to memorise every line. It means they should understand the evidence, the judgement and the limits well enough to stand behind it.
AI can help draft the work, but it cannot own it. Whoever signs off is responsible for the content.
No AI use comes without conditions
One of the clearest points made by leaders in the session was that AI use should not be treated as either allowed or banned in a blanket way.
AI use comes with conditions.
At a minimum, two conditions should apply.
A person reviews the output.
AI use is declared.
Those two conditions sound simple, but they change the culture around AI. They make it harder for AI-assisted work to slip through as if it were fully human-produced and fully checked. They also make it easier for people to use AI responsibly, because the expectation is clear.
Review does not mean a quick read for tone. It means checking the parts that could cause harm if wrong.
For a short internal update, that might mean checking names, dates, numbers and links.
For a board paper, it might mean checking evidence, assumptions, risks, options and recommendations.
For work involving patient data, service decisions or clinical context, the standard must be higher again.
Declaration also needs to be practical. If the process is too heavy, people will avoid it or ignore it. If it is too vague, it will not help anyone.
A useful declaration might answer three questions:
What tool or type of tool was used?
What task did it support?
What human checking was completed?
This is not about shaming people for using AI. In many cases, AI use will be sensible. It can help draft, summarise, compare, reword and structure. It can save time on administrative work and reduce the blank-page problem.
The point is that AI-assisted work should be traceable enough to review.
The right pace is not one pace
Another important point from the session was about pace.
Organisations often look for one AI position. Move fast, or slow down. Encourage use, or restrict it. Open access, or lock it down.
That kind of single answer rarely fits the work.
The better approach is to match the pace to the risk.
Moving faster on low-risk administrative work can make sense. Drafting a meeting summary from approved notes, rewording a plain-language email, creating a checklist or turning a rough outline into a cleaner first draft may be reasonable, if the output is reviewed.
Taking more care wherever patient data is involved also makes sense. The same applies to work that affects service planning, funding decisions, public communication, clinical pathways or community trust.
This is not a contradiction. It is good governance.
A Primary Health Network does many kinds of work. Some tasks carry low risk. Others carry serious consequences if the information is wrong, incomplete or used in the wrong setting.
A single rule for all AI use may feel neat, but it can produce poor results. If the rule is too strict, people may miss useful ways to reduce administrative load. If the rule is too loose, people may use AI in settings that need stronger controls.
A better model is to define categories of use.
Low-risk use may allow more freedom, with light review and simple declaration.
Moderate-risk use may need clearer checking, source review and manager sign-off.
High-risk use may need strict controls, approved tools, privacy review, subject matter checking and clear accountability.
The categories do not need to be complicated. They need to be understandable enough that people can use them in real work.
This also supports data literacy. People begin to ask better questions, not just about the AI tool, but about the task itself.
What data is being used? Is it sensitive? Is it current? Is it complete? Who could be affected if the output is wrong? What decision might this shape?
Those questions build capability across the organisation.
Accountability cannot be outsourced to a tool
The most dangerous belief about AI-assisted work is that the tool somehow shares responsibility.
It does not.
A tool does not attend the meeting to defend the recommendation. It does not answer questions from a board, a commissioner, a clinician, a community partner or a patient group. It does not carry the duty to be accurate, fair and careful.
People do.
That is why sign-off matters. Signing off AI-assisted work means taking responsibility for the content, not just approving the format.
This can feel uncomfortable because AI changes the production process. A person may not have written every sentence. They may have prompted, edited and assembled the work. They may have used AI to summarise a long document or draft an initial recommendation.
But if their name goes on it, or if they approve it, they need to understand it.
That means organisations should avoid treating AI governance as only an IT issue. The technology matters, but the approval habits matter just as much.
Good AI governance shows up in everyday behaviours:
People declare AI use without fear.
Reviewers ask about sources and checks.
Teams distinguish between drafting and decision-making.
Leaders set different standards for different levels of risk.
Sign-off means real ownership.
This is where leadership has a practical role. Leaders do not need to know every technical detail of how a model works. They do need to set the expectation that polished output is not enough.
They need to model the questions.
They need to make checking normal.
They need to create room for people to say, “This reads well, but I have not verified it yet.”
That sentence should be seen as responsible, not slow.
The real test is not whether AI was used
The real test is whether the work is reliable enough for its purpose.
AI-assisted writing can be useful. It can help people start faster, organise messy material and communicate more clearly. In stretched health and community settings, those benefits are not trivial.
But usefulness does not remove responsibility.
The lesson from the leadership session was simple and practical. Do not judge AI output by how polished it looks. Judge it by how well it has been checked.
Before signing off, ask what it is based on. Ask what AI drafted. Ask what changed. Ask how the numbers were checked. Ask whether the local complexity is there. Ask whether the person responsible can explain it without the AI text in front of them.
A good AI culture is not built by banning every use or trusting every output. It is built by matching pace to risk, declaring use, reviewing carefully and keeping ownership with people.
AI can draft the work.
People have to own it.
Comments