You audit by exception, not by volume. Your agent writes a structured record of every single run, and each run gets checked against a short list of rules: did it spend money, did it touch a client, did it do something you cannot undo, did it come back unsure of itself. You read the handful that trip a rule and you leave the other 300 alone.
I run 45 AI employees. If I read their output, that is the only job I would have.
Why does re-reading every output fail as an audit?
Because output is the artifact, not the behavior.
You reading a finished email tells you the email is fine. It does not tell you the agent rewrote it four times, pulled a stale price list from a folder you forgot existed, and almost sent it to the wrong Sarah. All of that lives in the run, not in the result. Reading results is proofreading. It is not supervision.
The other problem is attention. On Monday you read all 40 outputs. By Thursday you are skimming. By the following Monday you read none and you tell yourself you are trusting the system. That is not trust. That is neglect with better branding. I have watched smart founders do this with their first hire, human or otherwise, and the crash always looks the same.
AI for founders and operators gets sold as speed. The genuinely hard part is supervision that does not eat the speed back.
What has to be in an AI agent audit log for it to be worth reading?
One row per run, in a real database. I use Supabase, which is Postgres with a decent dashboard. Airtable works when you are small. What matters is the fields.
- run_id, timestamp, agent name
- the task as given, in plain language
- the context it actually pulled, with source names. That field is context engineering for founders, and almost nobody logs it
- every tool call in order, with arguments
- tokens in, tokens out, cost in dollars
- retries and errors
- a confidence number the agent is required to emit
- internal or external destination
- reversible or irreversible
- a diff of records or files changed
- one line written by the agent: what I did and why
That last line is the most useful thing in the table. Make it mandatory. An agent that cannot state what it did in one sentence does not understand what it did.
If you are running an agentic workflow in n8n or Make, add one node at the end that writes the row. If you are in code, OpenTelemetry or LangSmith will capture traces, but keep your own table too, because you need the log joined to your customers and your invoices, and their dashboards will not do that. The audit log is the black box of the whole AI agent operating system. If your system does not have one, it is a demo.
What rules decide which runs I actually read?
Four rules cover almost everything.
Money. Any run that spends or refunds above a threshold. Mine is $50 for new vendors and zero tolerance for refunds.
Irreversibility. Sends, deletes, overwrites, payments, calendar invites. Anything a second email cannot fix.
Confidence. Below 0.7, I read it. The number is self-reported and it is wrong sometimes, but it is directionally honest enough to sort a queue.
Shape. Any run with two or more retries, or any run that touched a file outside its assigned folder. That is usually where the real story is hiding.
Write the rules once as a Postgres view or a filter branch, and have it drop flagged runs into a queue with three tiers. Red, read now. Yellow, read tonight. Green, sample only. That is the whole audit. You are not reviewing work. You are reviewing the runs that broke something.
How do I get this down to four minutes a morning?
Summarize the flagged runs with a model and send it to yourself.
At 7am a scheduled job pulls yesterday's red and yellow rows, hands them to a model with a tight prompt, and DMs me on Slack. One line per run: what it did, why it flagged, what it recommends. Links to the raw row underneath. I read it with coffee. Most mornings it is empty, which is the point.
For anything irreversible, the agent does not act. It posts the intended action to a channel and waits. I hit approve and it proceeds. This is what people mean by human in the loop AI agents, and it is less about control than about sleep. My wife has stopped asking why I check my phone at breakfast, so the system is working.
The audit log is what makes the approval queue useful. Without it you are approving a sentence. With it you are approving a run.
How do I keep the log honest?
Sample the green runs. Five percent, random, every week.
An agent that logs badly will pass your rules by accident, and you will not find out until something breaks in front of a client. Five percent is cheap, and it keeps everyone, including the model, in the habit of writing things down properly.
Do one replay a month. Take a past run, feed the same context back in, see if you get roughly the same behavior. If the answer drifts wildly, your prompts or your context have rotted.
Then read your own rules every quarter. Mine have changed four times this year. The first version flagged nothing because I set every threshold too high, and I spent three weeks feeling great about an audit that was not auditing.
FAQ
How long should I keep AI agent audit logs?
A year of rows in Supabase costs almost nothing, so I keep everything and archive anything older than 90 days to cold storage. The reason is not compliance. It is so you can answer "has it ever done this before" in thirty seconds.
Do I need a dedicated observability tool?
No. A Postgres table and a filter will carry you a long way. Add LangSmith or Helicone when you are debugging prompts, not when you are auditing behavior. Those are different jobs, and companies keep buying one for the other.
What is the one field people forget?
The context that was pulled. Everyone logs the output and the tool calls. Almost nobody logs which documents or records the agent read to get there, and that is where the bad answers come from nine times out of ten.
I wrote the full playbook for this. How to Build Your Own AI Agent Operating System walks you through the exact architecture I use, step by step, including the log schema and the rules that sort the queue. You can get it at a.mastermindshq.business/ai-os-book.