What happens when AI coding agents build without product context

For a stretch of their experiment, a team at OpenAI spent every Friday cleaning up after Codex. That’s 20% of the week spent untangling patterns the agent had copied across the repo, and this was OpenAI, working with its own agent.

The bigger picture looks stranger. A Harvard study followed 718 companies as they rolled out coding agents, and engineers there started writing about 30% more code. The amount of finished, deployed work didn’t measurably move.

I wanted to know where all that extra code went, so I went through this year’s research on it, and the trail leads somewhere pretty boring, to a ticket.

The ticket never says the important part

Every ticket is missing things, because whoever wrote it already knew them. They knew which accounts kept bringing this up on calls, roughly what it’s worth at renewal, that the last attempt got rolled back, and that enterprise customers want an audit trail on anything touching permissions. None of it went into the ticket, because a human teammate would have asked or already known.

An agent running on its own doesn’t ask. The researchers behind a benchmark called SWE-RPG gave Claude Code, Codex and OpenCode 163 real tasks from 31 repositories, and wrote down what a correct fix actually had to cover, including everything the issue text left out. The agents solved 31.5% on average, and the biggest single reason they failed was missing those unwritten requirements, which sank somewhere between a quarter and almost half of all runs depending on the setup.

Take a ticket that says “we need to update permissions for multiple users at once.” An agent working from that line builds a multi-select list, a bulk edit button, an endpoint and tests for exactly what it was told, and all of it passes. The person who filed it knew a similar feature had already broken permission inheritance once, and that the biggest accounts need a log of every access change, so what they actually needed was delegated admin rights with limits. Both versions pass CI, and only the person who filed the ticket can tell you which one is the feature.

Then the code sits in review

This is where a lot of the extra 30% goes. In the Harvard data, once companies adopted agents, pull requests took 49% longer to get merged and the share sent back with changes requested nearly doubled. The share of people doing reviews went up about 14% to cope, and merges still got slower.

Faros, which tracks 22,000 developers, sees the same queue from inside the pipeline. Pull requests got 71.8% bigger and merges with no review at all rose 76.3%, which looks to me like people skipping the queue to keep work moving. (Harvard didn’t see PRs grow, so take the size number with some salt. The skipped reviews are harder to explain away.)

Some of the code never gets reviewed in the first place. Researchers at Drexel went through 33,596 pull requests opened by coding agents on GitHub and hand-checked 600 that got rejected. Broken code was the minority. The biggest group, 38%, sat untouched until someone closed them, and another 23% duplicated work that already existed in a different PR, which is an agent with no idea what the rest of the team was doing.

My guess is that a lot of that review time is someone senior reconstructing what the ticket left out, one pull request at a time.

The expensive part never shows up on an engineering dashboard

Review time is the cost engineers can see, and I think the bigger one sits on the product side, where almost nothing measures it. None of the 2026 studies I read tracked what agent-built features do to adoption, retention or revenue, so I went back to older numbers on how often features work at all.

Ronny Kohavi ran experimentation at Microsoft, and back in 2013 he wrote that fewer than a third of the ideas they tested moved the metric they were built to move, and on a mature product like Bing it was even fewer. That was when building a feature took weeks. Agents made building cheap and left those odds exactly where they were, so now it’s easy to ship a lot of things that were never going to move a number.

And shipped features don’t go away. Each one has to survive every API change, carries its own permission rules and docs, generates support tickets and gets dragged through the next migration, whether ten customers use it or none do. If nobody agreed on a success metric before the agent started, there’s nothing to check afterwards, and on a throughput dashboard a feature that changed nothing looks exactly like one that worked. We wrote about what this does to the people involved in what happened to the PM and developer handoff.

Every context-free change becomes the next agent’s context

An agent’s context is mostly the codebase it’s working in, so every change it merges becomes an example for the next run to copy.

That’s exactly what happened at OpenAI. Codex copied whatever patterns were already in the repo, the uneven ones included, until the team was spending its Fridays on cleanup. They got out of it by writing a short list of mechanical rules into the repository and running background jobs that hunt for deviations and open small refactoring PRs. The researchers behind SlopCodeBench saw the same thing in a lab, where 15 agents extending their own code across 196 checkpoints kept getting messier, and none of them finished a single problem end to end.

In real repositories, GitClear looked at 623 million code changes and found duplicated code blocks up 81% since 2023, while changes to code older than a year shrank from 1.7% of all changes to 0.46%. That second number worries me more, because old code is where the unwritten rules live, like a billing exception for one big account or an API response some partner integration still depends on.

OpenAI at least started from an empty repo. A fifteen-year-old codebase with its rules in postmortems and in the heads of whoever’s been there longest starts much further behind, and in a Carnegie Mellon study of 807 projects that adopted Cursor, the complexity that piled up afterwards was a big reason those projects slowed down a few months later.

Each pass around the loop starts from a slightly worse codebase, which is why this tends to look fine in the first sprint and expensive a few months later.

What the agent needs before it writes a line

Everything the agent was missing in this story already exists somewhere in the company. It’s in sales calls and support tickets, in the CRM and the product analytics, in old postmortems and in a few senior people’s heads, which are all places the agent never reads. One senior engineer interviewed for the Harvard study said that in most places they’d worked, “the real bottleneck has been communication between business and product teams,” and coding agents didn’t touch that bottleneck.

Before a nontrivial feature, I’d want the agent holding six things:

1. The problem in the customer’s words, who’s affected, and why it matters now.
2. The calls and tickets behind it, and how often it keeps coming up.
3. Which accounts care, how much revenue rides on it, and the KPI that should move.
4. What was tried before and why it got rejected.
5. The boundaries the code can’t cross, from banned patterns to legacy contracts.
6. A definition of done that includes the number someone will check after release.

Even with all of that, I’d make the agent write a plan first and show which evidence it’s leaning on, and anything touching architecture or risky business rules goes to a person before code exists. OpenAI’s agent-first team kept humans on exactly this part, prioritizing work, turning user feedback into acceptance criteria and checking outcomes, and DORA’s 2025 report found AI helps most in teams that stay focused on their users. So when people talk about the AI development lifecycle, I picture it starting at the customer signal and ending at the measured outcome, with the code somewhere in the middle.

Neta Barkay, Head of Data & AI at Bagel AI, explains why an LLM that can summarize 10,000 tickets still can’t tell you what to build in Your AI sounds like it knows what’s broken. It doesn’t.

Where Bagel AI fits

Full disclosure, this is the problem we work on. Bagel AI is a product brain. It reads sales calls, support tickets, chats and threads next to usage, ARR and whatever KPIs a team cares about, collapses the duplicates into single problems, and serves the resulting product decisions through the Bagel AI MCP to Claude Code, Cursor, Codex or any agent that speaks MCP. Each decision links back to the calls and accounts behind it, so the agent gets the reason along with the request.

Our own numbers show why that matters for agents specifically. A month of our inbound calls, tickets and messages came to 12.3 million tokens, and 904,000 of them carried real product signal, about 12x less for an agent to read (full breakdown in the Data Room). After a decision ships, Bagel AI tracks it against the outcome it was supposed to produce and feeds that back into the next one. And if you’re wondering why not just connect the agent to every raw source, Anthropic tried that with its own data and landed at 21% accuracy. Our MCP explainer covers the rest.

Try it on your own backlog

None of these studies proves that product context fixes the problem, and I’d rather you test it than take my word for it. The experiment is cheap. Pick a handful of similar feature requests and have a PM write down the real problem behind each one before any code exists. Run the same agent twice per request, once with only the ticket and the repo and once with the customer evidence, the business context and the history added, then have someone who doesn’t know which run is which score both against the written problem.

Keep extending the same features for a few rounds, because SlopCodeBench found the damage builds with every extension and better prompts only improved the starting point. Review data will show you the code side within a few rounds. For the product side, write down the number each feature is supposed to move before you start, because that’s the one thing you can’t reconstruct later.

The numbers behind this piece

For anyone who wants to check my work, these are the studies I leaned on, newest first.