AI hallucination guardrails – what we built into our VA workflow

Every AI-augmented VA agency that says “we’ve never shipped a hallucination” is lying or hasn’t been doing this long enough. We have shipped them. We have caught them. We have built guardrails because of them.

Here are the four guardrails we put in place and the cases that slipped through anyway.

What we mean by hallucination

Two flavours:

Factual hallucination. The AI states a number, source, date, or fact that is not in the prompt and is wrong. Example: “Australian Consumer Law section 42 protects refund rights” (it doesn’t; the relevant section is 18 for misleading conduct and 54 for consumer guarantees).

Tone hallucination. The AI states something true but in a tone that doesn’t match the brand. Example: an empathetic refund response that says “we totally understand and absolutely want to make this right!!” when our brand voice is “we’re sorry, here’s the fix, this is what we’ll do differently”. Less obviously wrong but cumulatively erodes trust.

In the customer-facing work we see, tone hallucinations are the more frequent of the two, and they are easier to miss because nothing in them is checkably false. We over-invested in fact-check guardrails initially and under-invested in tone guardrails.

Guardrail 1: source verification at the prompt level

Every prompt that asks for a factual output includes the constraint that any “fact” must come from the prompt itself.

Bad prompt:

Write a customer reply explaining our refund policy.

Better prompt:

Write a customer reply explaining our refund policy.

Our refund policy:
- 30-day no-questions-asked window for change-of-mind returns
- Faulty goods refunded under Australian Consumer Law
- Refunds processed within 5 business days
- Customer pays return shipping for change-of-mind

DO NOT cite any other policy. If the customer asks about something not
covered here, output [ESCALATE] instead of inventing the answer.

The bad prompt gets the AI inventing 30-day vs 14-day vs 60-day windows depending on its mood. The better prompt grounds the response.

This is the easiest guardrail to teach and the one with the biggest single payoff.

Guardrail 2: claim-level review at the edit step

Before any AI output ships, the VA marks every specific claim and verifies it.

A “claim” is anything that could be fact-checked: a number, a date, a regulation reference, a named person, a source attribution, a price, a deadline, a feature claim.

The verification loop:

1. VA reads the AI output and highlights every claim
2. For each, VA either:
   • Confirms the claim came from the prompt (no further check needed)
   • Verifies the claim against the source of truth (Xero, Shopify, your SOP, etc)
   • Marks it [VERIFY] and escalates if it can’t be confirmed

This takes 30-90 seconds per output. It is the step that catches most factual slips before they leave the VA’s hands.

The training rule: if you’re not certain a claim is real, don’t ship it. Mark it [VERIFY] and ask.

Guardrail 3: customer-facing edit floor

For any customer-facing copy (emails, social, support replies, marketing), the VA must visibly edit the AI output. Minimum threshold: at least 2 sentences different from the AI version.

Rationale: rubber-stamping AI output is the #1 failure mode (see Training a new VA on AI in their first week). The edit floor forces the VA to engage with the output rather than copy-paste.

What “edited” looks like:

• Replaced cliched openers (“I hope this email finds you well”) with specific openers
• Tightened over-apologetic phrases (“we’re so terribly sorry”) to measured ones (“we apologise”)
• Added a specific detail from the customer’s history (their name, their previous order, their location)
• Removed any sentence that adds no information

If the final output is 100 per cent identical to AI, the VA hasn’t done their job, and it goes back for a redo.

Guardrail 4: the day-5 competency check

Every new VA passes a recognition drill on day 5 of onboarding before they go live (see Training a new VA on AI in their first week).

The drill: we show them deliberately broken AI outputs containing the three failure modes (confident invention, tone drift, generic phrasing). They must identify each failure and fix.

Pass: they go live with weekly spot-checks for the first 30 days.

Fail: they get a second week of supervised practice. We don’t push a VA to handle AI-augmented customer-facing work until they pass.

Some new VAs need the second week. If a VA still can’t pass after it, we replace them rather than put them live on customer-facing work.

What still slips through

Four composite cases, built from the failure patterns we train against rather than a log of any one client’s incidents:

Case 1: Fabricated study citation in a blog draft. The AI cites “a 2023 University of Melbourne study showing 78 per cent of small businesses…” The study does not exist. The VA doesn’t verify because the AI cited a confidence-inspiring source. It gets caught at the client’s editorial review before publication. The fix: every external citation needs a URL or a [VERIFY] tag before the draft moves on.

Case 2: Wrong product weight in a Shopify product description. The AI invents a “250g” weight where the actual product is 175g. The VA misses it because the spec sheet wasn’t in the prompt, and the customer notices on receipt, which usually ends in a refunded shipping fee. The fix: the spec sheet goes into every product-description prompt, and weights are tagged with [SPEC] to flag that they need verification.

Case 3: Tone drift on a complaint resolution. The AI’s reply is technically correct but uses the phrase “totally understand”. The customer is already inflamed and reads it as condescending, and the business has to apologise separately. The fix: any reply to a complaint flagged above 5/10 severity goes to the business owner for review before sending.

Case 4: Date hallucination on a calendar invite. The AI writes an invite for “Tuesday 14 May” when 14 May is actually a Wednesday. The VA is on autopilot and doesn’t double-check, and the customer turns up on the wrong day. The fix: dates come from the calendar tool or the system clock, never from AI free-text.

Each case prompted a workflow change. The guardrails are not “set and forget” – they evolve with new failure modes.

The hardest case

The one we still can’t fully prevent: when the VA’s verification is itself wrong.

Example: AI cites “ATO Determination 2024/3”. VA googles, sees results, accepts as valid. The first Google result is itself AI-generated content citing the same fabricated determination. The hallucination has propagated through the broader web.

Mitigation: for legal, tax, and regulatory references specifically, we require verification against the primary source (ato.gov.au, legislation.gov.au, ahpra.gov.au, mara.gov.au) – not against secondary search results.

For everything else, the residual risk is real and manageable. We don’t publish a count of propagated hallucinations, because a tally from an agency our size would say more about how hard anyone looked than about the risk. What matters is that the primary-source rule catches this class of error when it is followed.

What we tell clients

Honesty over optics:

• AI is used in a large share of VA tasks, and we tell you up front which ones.
• Hallucinations have got past VAs, including ours. Anyone who claims otherwise isn’t looking.
• Each failure pattern taught us a guardrail, and the guardrails keep evolving.
• We will not promise zero hallucinations. We will promise faster detection and faster correction than any non-AI workflow.

Clients respond well to this. Vibes-y “we have it under control” loses business.

What’s next

For the wider context, see Why we don’t replace VAs with AI.

For the training plan that builds in these guardrails, see Training a new VA on AI in their first week.

For the SOP format that supports verifiable outputs, see Writing SOPs your VA AND Claude can both follow.

If you want a placement that uses these guardrails by default, book a discovery call.