You test an AI agent the same way you'd onboard a new hire before letting them near a client. Real scenarios, no audience, and a long hard look at what they say when they don't know the answer. Two things break almost every agent, missing context and no constraints, and both of those get fixed before launch, not after.
That's the short version. Here's the process I actually run across the 45 AI employees working inside my business.
Why do AI agents hallucinate and go off-brand in the first place?
My first support agent invented a refund policy. Not a small one. A 45 day money back guarantee I have never offered in my life.
That was not a model problem. The model had no source of truth, so it built one out of the shape of the question. That is exactly what a new hire does on day two when nobody handed them the rules.
What's becoming clearer to me is that almost every failure people blame on the model is a context failure. The agent was never told the refund window. The agent was never told who we are. The agent was never told what to do when it doesn't know. Context engineering for founders is just this: deciding what the agent sees, what it never sees, and what it does in the gap between the two.
There's a second thing, and it's the part people miss. A chatbot that makes something up is annoying. An agent that makes something up is serious, because it makes something up and then sends the email. Or books the meeting. Or moves the money. The tools turn a bad sentence into a real event.
Because of that, the testing has to happen with the tools switched on. Testing the words alone tells you almost nothing.
What does a passing test actually look like?
Everyone who builds AI agents for business hits the same wall around month three. The agent sounded great in the demo and then said something to a real client that you'd never say.
My answer is a scenario test. Twenty real conversations from the last 90 days, pulled straight out of my inbox and my DMs, including the ugly ones. For each scenario I write down two things: what a great human on my team would have said, and what off-brand looks like, so I can recognize it when it shows up.
Then I run the agent against all twenty and grade three categories.
Facts. Does it know the price, the delivery time, the refund window, the actual name of the thing we sell? Every wrong fact is a missing document or a stale one.
Boundaries. What does it do when someone asks something outside its lane? The right answer is usually "let me get a human," and most agents have never been told that's an option.
Voice. Does it sound like me or like a press release? If it opens with "Thank you for reaching out to us regarding your inquiry," it failed, and it failed loudly.
I keep all of it in Airtable, one row per scenario, with columns for the input, the expected behavior, the actual output, and pass or fail. You can absolutely do this in a Google Sheet. If you want the runs automated, n8n or Make will fire each scenario through the agent and drop the transcripts back into the table. At real volume, LangSmith and Braintrust both do proper evaluations, but you do not need them to start. The file structure behind my own AI agent operating system is laid out in detail over on the playbook page if you want to copy it.
How do I build test scenarios if I'm not technical?
You already have the material. You just never thought of it as material.
Go back 90 days and grab twenty messages. Not the pleasant ones. The billing complaint. The person who asked a question your website doesn't answer. The one who wanted a discount. The one who was angry about something that was genuinely our fault.
Write them into a sheet. One row each.
Then run every scenario three times, not once. Agents are non-deterministic, which is a fancy way of saying the same input can produce a different answer on Tuesday. If it passes once and fails twice, it failed.
The whole point of no code AI agents is that this part is available to you without hiring a developer. Claude Projects and ChatGPT Projects both let you drop in a context file and hold a conversation against it in about ten minutes. If you're already in n8n, you can wire the sheet to the agent and watch twenty transcripts come back while you make coffee.
One addition: keep the failing outputs. Those are your regression set, and they're worth more than the passes.
How do I keep an agent on brand across a whole agentic workflow?
Brand is not a vibe. Brand is a list of constraints, and constraints belong in the context file.
I give every agent a short brand document. What we say. What we never say. How we open. How we close. What we do when we're wrong. Mine is under 400 words, because a long brand doc gets ignored the same way a long onboarding doc gets ignored.
One concrete line in mine: never say "I apologize for the inconvenience." That sentence makes a person feel like a ticket number, and it shows up in default model responses everywhere.
Then there's the handoff rule, which did more for me than any prompt rewrite. If the agent is unsure, or the person is upset, or the request touches money or a legal question, it stops and tags a human. The agent's job in that moment is not to be helpful. It's to be quiet and hand it off.
That single rule is why I let my agents talk to clients at all.
Worth naming too: the pass criteria change once an agentic workflow has more than one step. In a single reply you're grading tone. In a workflow where the agent reads an email, drafts a reply, updates a record in Airtable, and posts to Slack, you're grading whether it did the right thing at each step. Same scenario sheet, more columns.
How often do I need to re-test?
Every time you touch the context file. Not every time you touch the prompt, every time you touch anything the agent reads.
This is the boring part, and the boring part is the point. I keep a regression list of the ten scenarios that have broken before, and those ten run first after every change. I also read five real conversations a week myself.
That last one isn't optional for me. No eval set catches the thing you didn't think to write down. Reading does.
FAQ
Do I need to tell the agent not to hallucinate? You can, but it doesn't do much. The real fix is making the information available and telling the agent what to do when it isn't. "If you don't know, say you'll check and tag a human" works far better than "do not hallucinate."
Do I need to be technical to do this? No. A sheet, a chat project, and twenty real messages get you 80% of the way there. Tools like n8n, Make, and Airtable handle the rest when you want it running on its own.
How do I know when it's good enough to go live? When it passes 90% of twenty scenarios across three runs each, and the failures all leaned in the safe direction, meaning it asked for help instead of guessing. Then watch the first two weeks closely and add every miss to the regression list.
I wrote the full playbook for this. How to Build Your Own AI Agent Operating System walks you through the exact architecture I use, step by step. You can get it at a.mastermindshq.business/ai-os-book.