The app an AI builds is the app in its training data

The standard advice for AI-assisted development is simple: let the agent write the code, then review it. We follow a stricter version of that ourselves and wrote it up as controlled AI coding. But this build taught us where that advice runs out. Review catches the mistakes you already know to look for. It does nothing about the mistakes that look like correct code.

Some context, kept short. We run a software-export business in India, and export invoicing follows rules that generic tools don’t model. Exports of services are zero-rated under Section 16 of the IGST Act, payment must arrive in convertible foreign exchange, the bank issues a FIRC (Foreign Inward Remittance Certificate) as proof, each remittance carries an RBI purpose code, and invoice numbers run gapless per financial year, which in India starts on April 1. This is our working understanding, not tax advice; confirm specifics with your chartered accountant.

Because no off-the-shelf tool ties a FIRC to the invoice it settles, we built our own system, with Claude Code doing most of the typing. The agent was fast, confident, and produced clean PHP. It also produced two failures that never threw an error, never failed a test we had at the time, and would have passed any code review that judged the code as code.

Incident one: the agent built a void feature and made it law

Every accounting tutorial, every open-source invoice app, every SaaS product the model has ever seen handles a wrong invoice the same way: void it, keep the number, never delete. So the agent built exactly that. Not a suggestion, a complete feature: a Void button, a confirm dialog, a reinstate action, and a comment declaring the behavior as rule “§1.10” of its own spec. From invoice-view.php at commit 99d6730:

/* Tax-invoice document (§2). Born paid — status is `issued` or `void`,
 * voided, never deleted (§1.10). */
if ($action === 'void' && $inv['status'] === 'issued') {
    $DB->prepare("UPDATE invoices SET status='void' WHERE id=?")->execute([$id]);
    flash('Invoice voided. The number is retained (never reused).');
}

As generic accounting software, this is correct. As software for an Indian exporter, it is a liability generator. In our reading of GST, an issued tax invoice creates a tax liability whether or not you later mark it void. A wrongly issued invoice has to be removed and its proforma reverted, so the liability never exists on paper. We caught it in the functional-gap review, and the fix went in as commit c9f4107, whose message states the reasoning better than any paragraph I could write here:

“invoice-view: replace Void with ‘Undo conversion’ — deletes a wrongly converted invoice, reverts its proforma to Paid, frees the FY number (an invoice that exists is a GST liability, so wrong ones are removed).”

That last part deserves a beat. The agent didn’t hedge. It codified the pattern as canonical, complete with a section number, and every later piece of generated code would have treated void as settled behavior. A wrong assumption an agent writes into its own spec compounds with every subsequent prompt.

Incident two: numbering that was right everywhere except India

The first version of next_invoice_number(), from the initial commit f6b7863, made two default choices in one function:

$st = $db->prepare('SELECT prefix, next_seq FROM companies WHERE id = ? FOR UPDATE');
// ...
return sprintf('%s-%s-%03d', $row['prefix'], date('Y', strtotime($issueDate)), $seq);

One global counter per company, and date('Y'): the calendar year. Reasonable everywhere the model’s training data comes from. Wrong in India, where invoice sequences reset on April 1 for the new financial year, and where our proforma and its resulting tax invoice must share a sequence number so the books line up. The catch, again, came from a review question, not from the code: “every business year, April 1st, we change the invoice numbers.” That single sentence forced a rebuild into per-company, per-document-type, per-FY gapless sequences.

Then the second layer, and this one is the more interesting failure. The rebuilt version used INSERT IGNORE plus SELECT ... FOR UPDATE plus UPDATE, a sequence that deadlocks under InnoDB when two transactions create invoices at the same moment. No human review caught this. It was caught by a concurrency test that existed for one reason: someone asked “what if a team member creates an invoice on their machine at the same time?” The question generated the test; the test caught what eyes could not. The fix was a single atomic statement.

That is the arc worth remembering: naive, then domain-corrected, then concurrency-corrected, each step with a diff. And the mechanism differed. The FY miss was caught by knowledge. The deadlock was caught by a test that a question produced. If your process only has “review the diff,” you get the first catch and miss the second.

The mistake that never happened

I expected a third incident. Agentic tools routinely model payment as a boolean, an is_paid flag on the invoice row, because that’s what tutorial invoice apps do. It never happened here. The very first schema, commit f39c0b2, already had a payments table with amount, currency, and FIRC linkage as a first-class record.

Why? Because the FIRC requirement was in the spec before the agent generated a line of code. When the spec says every foreign payment must carry its remittance certificate and purpose code, a boolean cannot satisfy it, so the wrong shape never gets generated. Nobody had to catch this mistake, because the constraint made it impossible to make.

Three rows, one pattern. The only mistake that cost nothing was the one the spec prevented.

The MCP connector: agentic defaults trust too much

The system has an MCP connector, so an accountant can open Claude Desktop and type “create and send an invoice to Acme Ltd for the March retainer” without ever logging into the app. The agentic default for this kind of connector is full write access, because that’s the frictionless demo. We went the other way. The connector is a thin layer over the app’s own token-authenticated API: writes are draft-first and require confirmation, every AI-initiated change is audit-logged, and the token is role-scoped so the AI cannot see or touch more than that accountant could. It’s stateless, single-request JSON, which lines up with where the 2026-07-28 MCP specification took the protocol. The AI holds no business logic. The invoice system stays the single source of truth, which matters for the same reason everything above matters: an agent that improvises accounting logic is an agent that improvises liabilities.

Review catches what you know. Specs prevent what you don’t.

Agentic tools never say “that’s unwise.” A human developer with export clients would have asked about the financial year before writing a numbering function. The agent shipped date('Y') without a flicker of doubt, and it would ship the void feature again tomorrow. We’ve written before about keeping AI-generated code clean and about why vibe coding needs guardrails before production, but this build sharpened the point: cleanliness was never the problem. Every one of these mistakes was clean.

So put the domain in writing before the agent starts. For anything with statutory weight, our pre-build spec now answers:

• Which jurisdiction’s rules govern this system, and which documents are statutory records?
• What is the full lifecycle of each document, including what may be voided, deleted, reverted, or never touched?
• What are the numbering rules: sequence scope, gaplessness, and which fiscal calendar they follow?
• What proof documents (FIRC, purpose codes, certificates) must exist as first-class records, not fields?
• What happens when two people do the same thing at the same time, and which test proves it?
• If an AI connector exists, what is its write model: scope, confirmation, and audit trail?

Every “what if” question in that list is a test waiting to be written. Ask it early and the agent generates the right shape from the first commit, the way the payments table proved.

The bottom line, and the code

The agent was worth it. It typed the app in a fraction of the time it would have taken us by hand, and both incidents were cheaper to fix than a single GST notice would have been to receive. But the value of the build lived in the two moments a human said “not in India” and the one moment a spec made the question unnecessary. Speed came from the agent. Correctness came from the domain.

We’re releasing the system under MIT for other Indian software exporters and freelancers with the same compliance shape: a single-business kit that self-installs on ordinary PHP and MySQL shared hosting, with your data in your own database. It is deliberately narrow. Built around Indian GST and export-of-services rules, it is not a fit for domestic billing at other tax rates or for non-Indian firms without changes. The repository is on GitHub; if you find a rule we got wrong, an issue telling us so is the most useful contribution you can make.

The post Agentic Coding Mistakes: Lessons From a Real App build first appeared on Macronimous Blog.

It usually goes like this. A developer points Cursor or Copilot at a gnarly legacy file and asks it to clean things up. The model does what it’s good at and collapses a nest of conditionals into something half the length and twice as readable. Tests pass locally. The diff looks clean. It ships. A week later, a quiet edge case that one of those “redundant” branches used to catch shows up in production, on a client site nobody has opened in years.

There’s a hundred-year-old name for the mistake. Chesterton’s Fence goes like this: you find a fence across a road with no obvious purpose. The reckless reformer says, “I don’t see the use of this, clear it away.” The wiser one answers: go find out why it’s there first, then you may remove it. Every codebase older than its current team is full of fences. The validation that looks paranoid. The handler for a case that “can’t happen.” The query that dodges an obvious index for reasons lost to time. We are now handing those codebases to tools that are fast, confident, and carry zero memory of why anything was built.

The clean diff is the trap, not the safeguard

The standard advice sounds responsible: let AI move fast, then review the diff. Clean change, green tests, merge. The problem is that a diff shows you what changed, not what the change knew. A deleted block leaves no trace of the bug it was preventing. The tests pass because nobody ever wrote one for that edge case, which is precisely why there was a guard sitting there instead of a test. So you get a clean diff, a green build, and a landmine reburied for next quarter. The review ritual feels like diligence while it quietly rubber-stamps the deletion of context.

None of this is an argument against AI in the workflow. We run Claude Code, Copilot, and Cursor across React and PHP every day, and the speed is real. It’s an argument against confusing velocity with progress when you’re standing in code you didn’t write. The discipline of writing clean code with AI starts with knowing when not to let it touch something.

Why this bites agencies hardest

If you run an offshore or white-label shop, the exposure is sharper than it is for a product team that has lived on one codebase for years.

• You inherit. Most of what we touch was built by someone else, often years ago, for a client whose original developer is long gone. The fence and the person who built it both left before we arrived.
• Juniors lean on it hardest. A mid-level developer using AI to cover gaps is making the right move, until the model suggests something elegant and wrong and they don’t yet have the scars to tell the difference. In a pull request, elegant-and-correct and elegant-and-wrong look identical.
• The data risk is its own category. Pasting a client’s source, an API key, or a slice of proprietary logic into a public model is a one-keystroke NDA breach. That isn’t a code-quality problem. It’s a get-fired problem.

This is the same pattern we flagged in hidden technical debt: the costliest problems are the ones nobody can see in a quick review.

Not every old line is a fence

Here’s where the lazy reading of Chesterton, “never delete old code,” is as wrong as the reckless one. Plenty of old code is genuinely dead: commented-out blocks, an abandoned feature flag, a function nothing calls anymore. That is not a fence. It’s litter, and AI is excellent at hauling it away. Let it. The whole skill is telling the two apart.

When in doubt, treat it as a fence. The cost of investigating a line that turns out to be litter is two minutes. The cost of deleting a fence is a production incident on a client’s site.

Make the human own the commit

The rule we land on isn’t “trust the AI” or “distrust the AI.” It’s a short set of habits that cost time on purpose. The human owns the commit: “the AI wrote it” explains nothing, and if you push code, you defend every line in review. The AI builds inside your guardrails, not its own: state the stack versions, conventions, and component structure explicitly, every time, or it will invent a perfectly reasonable pattern that contradicts the three already in the project. And before you let a tool rewrite or remove anything, you find out why it’s there. We’ve written up the full workflow as Controlled AI Coding, but the pre-flight before touching existing code fits on one card:

• Run git blame on the block and read the commit message
• Check the original issue or ticket for the why, not just the what
• Confirm whether a guard is covering an edge case the diff would silently remove
• Make sure the change matches existing stack versions and conventions
• Verify no client source, credentials, or keys are going into the prompt

This is not about slowing everyone down. It’s about spending two minutes on the handful of lines that are actually fences, and letting AI fly on everything else. The same caution applies when the speed feels intoxicating; the high of vibe coding is exactly when fences get cleared without anyone noticing.

The one thing AI can’t bring to the review

AI doesn’t tear down fences because it’s reckless. It tears them down because it has no memory and no stake. It was never paged at 2 a.m. by the bug that guard was written to stop, so it feels no caution. Which means the caution is now entirely ours.

Reading that map, and knowing which fences to keep, is quietly becoming one of the more valuable things a developer can do, right at the moment the tools are most eager to skip it. Treat it the way Martin Fowler frames technical debt: something you take on deliberately and pay down on purpose, not something an assistant runs up for you while you watch the diff scroll past.

The post The Code Your AI Wants to Delete Is Load-Bearing first appeared on Macronimous Blog.