AI in a production workflow

I gave a webinar for the VAIT community this week on how my team at Envato runs AI inside a production workflow. The slides are on this site with the speaker notes intact (press S), and there is a PDF that prints the notes after each slide. This is the version for people who would rather read than click through 57 slides. I have kept the shape of the talk but loosened the sentences, because nobody talks the way slides look.

The audience was engineers early to mid career who already use AI tools and want to know what it looks like inside a team that ships to production and has auditors. Not a demo. One real piece of work from start to finish, then the plumbing that made it possible, then the parts that are still not working.

Two numbers

Four in ten of the pull requests we merged last month were opened by autonomous agents. Not autocomplete. An agent picks up a ticket, works in its own environment and opens the PR. 227 of 527 in September. Since we switched them on in March, 1,069 of 3,311. It was 7 percent in March and peaked at 46 in August.

Zero of those were merged by an agent. Every one was read, understood and merged by a person.

A scope line before anyone over-reads that: the numbers are from one repository, the main monolith behind the new Envato App. Envato has 585 repositories and my domain owns 23. The pattern runs across the domain, but the counting was done on one repo.

And a scope line on the zero: it is zero since April. For five weeks in March and April our AI reviewer could approve PRs, under a human’s login. We took that away. Not because it was wrong, but because we could not yet prove it was right, and in a controlled pipeline those are the same thing.

The second number is not a gap we are trying to close. It is the design. Agents write. People own.

Why the rules are as strict as they are

Our parent company is listed in the US, and the app takes money, so we sit under SOX, the Sarbanes-Oxley Act. The bit that reaches engineers is section 404: a listed company has to prove it has controls over anything that touches financial reporting, and auditors test those controls every year.

In practice an auditor asks three things about any change on a path that affects what a customer is charged, what they are entitled to, or whether they get what they paid for. Who made it. Who approved it. Were those two different people, with evidence.

So when we ask “should the AI reviewer be allowed to approve”, that is not a convenience question. The approver has to be an identity the auditors can trace, acting on criteria they can read, with evidence they can check. It also means the honesty later in this post is not optional. If a control is not working yet, the audit trail has to say so.

Part 1: search for Yemen, get Romania

Earlier this year a customer searched our stock video for a country and got footage from a different country. Search “yemen” and the top rows were maps, country outlines, a flag, a 3D render of the border, and when you scrolled, somewhere else entirely. The real footage was there. It was buried under things that merely contained the word.

I picked this piece of work for the talk because it was one of our key search improvements for the quarter and because it runs end to end: a complaint, a model retrained on real queries, a new service in front of search, a production rollout and a measurable result. Two engineers built all of it in a few days. You can see every step in one sitting, which is rare for work that size.

Step 1: ideation

It started in a backlog review, then three short huddles over a week with two teammates. We are a remote, async-first company, so the writing is the work. Every meeting ends up as a note with decisions and actions. AI does the first pass from the transcript and pulls in context from older notes, tickets and Slack threads so I am not starting cold. The useful part is not the summary. It is being able to ask “what did we decide about location detection in March” and get an answer with a source.

The output of that week was one written state-of-play document. AI drafted it. I rewrote every judgement call in it.

The decision that shaped everything after it came out of a huddle, not a model: precision over recall. Our existing entity model was very good at finding places and very bad at not finding them. It flagged “UTC” and “Piazza” as locations. Route those to a different search path and you make search worse for everyone. So we agreed we would rather miss a location than wrongly call something a location. That is a product call. No agent made it.

Step 2: the spike

Two days to compare the existing model against a compact fine-tuned classifier that answers one question: does this query contain a place, yes or no. One of my teammates trained it. We tested on 20 thousand real queries, then 100 thousand. Precision went from around 30 percent to over 85, with similar latency. For the curious, the models are public: an entity model in the GLiNER family versus a MiniLM binary classifier.

Where AI sat: it pair-wrote the notebook and the evaluation harness, and it wrote the queries that sampled real searches from our data warehouse. Hours instead of days. Where people sat: we picked the metric and we set the time box. Two days to compare, two weeks to decide. Agents do not time-box themselves.

One more thing from the spike changed how we work. We used to judge search quality by eyeballing grids of results, which caps you at about twenty queries before your attention goes. Instead we used a large language model as a relevance judge over a few hundred sampled queries, and two of us could evaluate a whole search profile in an afternoon. The model was also useful as a sceptic. An earlier precision number for a bigger model had itself been judged by a model, and it asked the question we should have asked: is that number a red herring? It was worth checking.

Step 3: Jira

Three cards: train and deploy the classifier, fix the worst country and city searches, make the decision. Our stories follow a simple shape. Context, done when, resources. Clear intent, not step-by-step instructions.

We have a Jira skill for this. I give it a sentence and the spike notes, it reads the codebase, drafts the card with acceptance criteria and scope fences, and I edit it. This matters more than it used to. A cloud agent will not stop to ask a clarifying question. A thin ticket gives you a weak PR. The card serves the next engineer and the next agent equally, and the bottleneck has moved from writing code to writing work.

Step 4: build

Building happened in two places. Locally, paired with an agent, I did the search routing logic and the search profile changes. Plan first, then code, then tests, and before I touched code a second model reviewed the plan. In the cloud, I delegated the well-specified pieces straight from the cards: tests, wiring, a config change. A card gets a label, an agent boots an isolated environment with the full app running, does the work and opens a PR with screenshots. It reads the same repo conventions my local session does. I reviewed those PRs like I would review anyone’s. The relevance judgement calls never left my machine.

What we ended up with is not a small change. Every query now goes through a query interpretation service before it is searched. That service is the classifier from the spike, retrained on a hundred thousand real queries from our logs. The search API routes on its answer: a yes goes to a new location profile, a hybrid of lexical and vector search with a Location filter, and a no goes to the default vector profile exactly as before. The classifier is in the BERT family and runs in milliseconds, and because it replaced an entity-extraction step and a language-detection step in the old pipeline, end-to-end latency did not go up.

Underneath that sit two loops. Retraining pulls real queries from the logs, which is how the model keeps up with what people actually search for. The evaluation set, a few hundred sampled queries judged by a language model, is replayed against the live API, so the same set that won the spike is the regression check in production. Routing rate and latency go to a dashboard, and I watched that during the rollout.

Step 5: review

Review is where the zero comes from. Commits and PR descriptions come from skills, so they are consistent: problem, approach, evidence. For the big PR a skill generated a one-page visual walkthrough so a reviewer could get the shape of the change in two minutes.

Then the reviewers. An AI reviewer runs in CI within minutes and posts inline findings with a severity. A second model gives an independent opinion. And then a teammate reads it, reasons about it and approves. The AI review is a required check and the PR cannot edit its own reviewer, which I come back to below. But the approval is human, every time. Four reviewers, one approver.

Step 6: ship

CI to staging, then production, behind a flag with a staged rollout. A person pressed the button. AI drafted the log queries and the dashboard: routing rate, latency percentiles, how many queries the classifier flagged. I decided whether the rollout continued.

Then we searched for Yemen again. Same query, same catalogue. Towns, wadis, the dragon blood trees of Socotra, camels, people. The maps and flags are gone from the top rows, and there is a Location filter in the toolbar now.

The table I wanted people to remember

Stage	Person	AI
Ideation	direction	summaries, context
Spike	metric, time box	notebooks, data, judging
Jira	intent	drafting the card
Build	design, delegation	plans, code, tests
Review	approval	findings, walkthroughs
Ship	go or no-go	queries, dashboards

Read the middle column top to bottom. The accountability never moved. The right column is wide, because AI was in every step, but every row ends with a person.

A year ago that project is a quarter of work for a small team, with the model retraining alone eating most of a sprint. The difference is not that the two engineers got smarter. AI was in every step, and the people spent their time on the decisions instead of the typing. That is the honest version of the productivity claim: one project, one team, but a real one with a before and after you can look at.

Part 2: what’s underneath

None of that works by accident. Ten things have to be true in the repository and in the team.

Config is code

The AI configuration lives in the repo and goes through pull request review like any other code. One AGENTS.md at the root, following the open convention, so Claude Code, Cursor, Codex and Copilot all read it. Detailed guidelines live in a folder next to it. Skills follow the Agent Skills open standard, with one source folder and symlinks into each tool’s directory. We use three tools on purpose, partly because they have different strengths and partly so our workflow is not hard-wired to one vendor.

Someone on the RSVP form asked how to keep model results consistent between dev and production. Half the answer is here. We do not try to make everyone’s model identical. We make the context identical and version it. Each config file carries a signature string and a script checks it, so “did the agent load the rules” is a test rather than a hope. Behind the rules sit 208 architecture decision records the agents can cite, so they get the why as well as the what.

Load the minimum

Context is finite and attention degrades, for models as well as people. The failure mode we avoid is a three-thousand-line instructions file the model half reads. Ours has four layers. Always-on is about 250 lines: who we are and the rules that never bend. Open a Ruby file and the Ruby rules load. Mention translations and the i18n skill loads. Need a deep dive on a Rails engine and a specialist takes it in a separate context and comes back with a result, not a transcript.

A skill is a procedure. A specialist is a colleague.

A skill is a Markdown file with a description of when it applies and a procedure. Our commit skill knows our seven rules for commit messages and the pre-commit checks. Our PR feedback skill fetches every review comment, classifies it and builds a task list. We have about 28 of them, and half are about the pull request: writing it, describing it, reviewing it, responding to review. That is where the habits were weakest and where codifying them paid most.

A specialist is a sub-agent with the full guideline for its domain embedded and its own tools. Rails engines, React routes, testing, CI pipelines. We also have a QA chain of five specialists that drives a real browser against the running app: one writes a feature definition from the live page, one designs tests, one runs them, one goes off-script looking for what the plan missed, and one turns the plan into a spec we keep. Each one hands a file to the next, so I can stop after any step and read what it produced before the next one starts. The exploratory tester is the one that earns its keep.

The mental model that helped the team: a skill is something I do, a specialist is someone I ask.

If you only write one skill, start with the thing you have corrected the agent on twice. Write the description as “when to use this”, with an example call. Allow-list the tools it needs and nothing more. Try it in a fresh session before you commit it, because it has to work without the conversation that made it obvious to you.

Hooks, not hope

A rule in a Markdown file is a suggestion. A hook is enforcement. Pre-commit runs formatting and linting, and it also fails the commit if a code comment or test name contains a ticket key, because a ticket key is not a reason and the ticket will be closed by the time someone reads the code. Pre-push refuses to push to any remote outside the company, with no bypass flag. My favourite is the stop hook: when an agent finishes, a script checks whether the branch has meaningful changes and, if you have opted in, kicks off an independent review by a different model before you have even looked.

Every one of these exists because of something an agent actually did. The ticket-key check exists because agents kept writing them into comments. The engine-boundary check got stricter because agents walked around the module structure. The push guard exists because a private codebase and an eager agent is a bad combination.

Seven months of our team channel contain no praise for any hook and two complaints about ones that broke. That is the job. I keep a tally of what each one has caught, and that tally is the argument for keeping them. The lessons, if you want them: fail open, finish in seconds, never call the network, opt in when it spends money, one script wired into every tool’s hook file, and test on the laptop you ship from as well as in CI. The stop hook shipped broken the first time. Making it opt-in was the fix.

The other RSVP question was how to avoid tech debt when AI takes away the struggle of working out how. This is half of my answer. Debt comes from decisions and engineers still make them. What changed is volume: more plausible code per hour. So the guard rails went from advice to automation.

Spec first

For anything bigger than a bug fix we use spec-driven development. A product manager’s Confluence page becomes a set of engineering files in the repo: requirements with testable acceptance criteria, a design with a decision log, stories, and tasks sliced so each PR is reviewable in one sitting. Every phase has a human gate, and the approval is recorded with a name and a time. The agent records the approval. It cannot invent one.

When not to use it: small fixes, config changes, spikes. Vibe-code the spike. Use the spec to productionise it.

Where it stands: three epics have used it. The practitioner retro called reviewing the generated artefacts laborious, walls of text. Our engineering manager’s verdict was that it is not useful where there is ambiguity, discovery or unknowns, and we set no goals for it this quarter. It fits small, well-specified work. This is the other half of the tech debt answer. The thinking did not disappear. It moved upstream. You argue with the plan before any code exists, a second model argues back, and your name goes on the approval.

Review the plan before the code

The cheapest quality win we have found. The agent checks its plan against how the codebase already does things, runs its own review from six angles (security, performance, testing, operability, architecture, product), and then a different model reviews it and they go back and forth until it approves. Different models have different blind spots. The bug in the plan costs almost nothing to fix. The same bug in production costs an incident. Our tooling lead’s rule: never use the review model as the coding model. Keep it adversarial.

Thin ticket, weak PR

The cloud agents from the story. A Jira card or a Slack message triggers one. It boots an isolated machine with Postgres, Redis, the Rails app, Storybook and a headless browser, reads the same configuration as a local session, does the work and opens a PR with screenshots.

The lesson after a few months is that input quality beats model capability. The agent will not pause to ask a question, so everything it needs has to be on the card. Signed-in flows are still hard for it, and PR descriptions are weaker when the harness opens the PR rather than the agent.

Its share of our merged PRs went from 7 percent in March to 43 in September. Our review load did not go down. It went up, and it moved to the leads. We wrote that down too.

The part people underestimate is the boot. A cloud agent is only as good as the machine it wakes up on, and that machine has to be ready without anyone watching. We learned to split “install once” from “start every time”, to fingerprint the migrations so a stale snapshot catches itself up, and to write the environment’s quirks into a guideline the agent loads only when it is running in the cloud. None of that is AI work. It is plumbing.

In July we tried to move the agent to a repository-defined environment so it would be reproducible from the repo. The headless browser would not start in it, the browser tests broke the same evening, and we reverted to the managed snapshot that night. The bigger operational cost has been the vendor’s metering: four spend-cap outages and one API break in seven months. The agent runs on a third party’s meter.

The PR can’t edit its own reviewer

An AI reviewer is only trustworthy if the author of a PR cannot influence the review of their own diff. So the prompt, the schema, the scripts and the shared guidelines live in a separate repository that PR authors cannot write to, pinned to a tagged release. The consumer repo’s guidelines are restored from the base commit before the model runs, so a PR cannot change them either. A PR that tries to weaken a shared rule is itself a top-severity finding.

It took three decisions over four months. March: a reviewer action inside the repo with structured output, because the vendor’s connector could only react with emoji. May: extracted to a shared workflow, with a bot identity so an approval can never be mistaken for a human’s. Our wrapper is 52 lines of configuration, down from roughly 600 lines of review machinery. June: on every PR.

It posts inline findings with three severities and a structured verdict, and every finding cites the rule behind it. On code paths with financial impact, uncovered changes are automatically blocking, which is what makes it acceptable to our auditors. It does not approve. Findings only.

Who reviews the reviewer?

If a bot is going to gate merges, something has to grade the bot. We built a ground-truth dataset: real PRs that were later reverted versus clean merges, plus planted scenarios like a pricing bug, a missing authorisation check and a prompt-tamper attempt. A cron job every Monday replays it through the reviewer and posts the score with confidence intervals. When a human overrules the bot, an issue form records the direction and the reason. Monthly snapshots, a quarterly evidence package into a tamper-evident repository, and missing weeks marked missing rather than hidden. Four passing weeks would unlock approval rights.

Now the honest state, which I said before anyone asked. The dataset landed mid-September. No eval week has passed its thresholds yet, so the four-week clock has not started. Two exceptions have ever been logged, both in the first fortnight. The mechanism exists. The habit does not.

Tokens are cheap. Attention isn’t.

We are on enterprise plans for Claude, Cursor and Codex, per seat, and the limits are generous enough that nobody rations tokens for daily work. The visible cost is cloud agent runs: isolated machines plus frontier models. The way I would frame it: a wasted agent run costs a few dollars. A badly reviewed PR costs an incident. So we spend on slicing, walkthroughs and review tooling rather than on cheaper models.

The cost that actually bites is attention. Our engineering manager posted last month about engineers having to manage agents without being taught how, and told people it is fine to let agents idle and to cap how many streams they run.

Prompts rot like docs

Skills and prompts rot the same way documentation does. Two things keep ours fresh. They live next to the code they describe, so drift shows up in the same PR. And people use them every day, so a broken skill gets noticed within hours. We have a skill for writing skills and one for scaffolding specialists, and shared skills ship through an internal plugin marketplace maintained by a community of practice. For the review prompt that gates merges, every material change has to pass a batch evaluation first. We treat it with the rigour of authentication code.

What the team said

Five quotes from seven months of our team channel, by role and month of the rollout. I read them out rather than paraphrasing.

“Reviews are very much still required, perhaps even slightly more so.” (lead engineer, month one)

“How does it notice the rule when reviewing but not when generating?” (principal engineer, month four)

“I only use it when the requirement is super clear.” (senior engineer, month five)

“It’s not useful if there’s ambiguity, discovery or unknowns.” (engineering manager, on spec-driven development, month five)

“Engineers now have to manage agents, but haven’t been taught the coping mechanisms.” (engineering manager, month seven)

The last one is the gap I think about most.

What is not done

The evals have not passed a week, so the reviewer still cannot approve. The overrule log exists and almost nobody uses it. Review load is up and it sits on the leads. One engineer owns the plumbing, which is a bus-factor problem I have not solved. The agent runs on a vendor’s metering, which is where most of our outages came from. And there has been one production incident attributed to an agent PR in seven months: an admin lockout, reverted the same day, which exposed a latent auth gap we did not know we had.

The failure I usually admit to, the July environment, lasted one evening. The one I should admit to is that the eval meant to earn the reviewer’s trust has not produced a passing week yet.

Start Monday

None of this needs an enterprise licence.

• Write the spec before the prompt: context, done when, resources.
• Read every line you ship, and do not approve what you cannot explain.
• Put your conventions in a file in the repo. An AGENTS.md is an afternoon’s work.
• Slice work so a person can review it in one sitting.
• Hooks, not hope.
• Get a second opinion, from another model and then from a person.

If I had to pick one, it is the first. Writing clear work is now the highest-leverage skill in engineering, and it is the one we teach least.

For the people who asked the RSVP questions: version your rules, and feed the same file to your reviewer that you feed to your coding agent. Engineers still make the decisions. We just made the decisions visible.

Four in ten written by agents. Ten in ten understood by people.