Can AI prioritize your product backlog? What the 2026 tests show

Triage was a manageable tax when feedback arrived in dozens a week. It arrives in thousands now, across sales calls, support tickets, chats and threads, and a lot of teams still sort it by hand or by counting how many times something was mentioned.

So the pitch for AI prioritization is easy to buy. In September a startup called TypeSafe AI released Jev, a model built to make exactly this kind of call, in under a second, for a fraction of a cent. If any model was going to take triage off a PM’s plate, it was going to be one like this.

The team behind Mastra, a TypeScript framework for building AI agents, tried it on their own GitHub backlog. The result was bad, and Mastra’s explanation for it applies to most product backlogs.

What is AI backlog prioritization?

AI backlog prioritization means using a model to rank or label backlog items, such as feature requests, bugs and customer feedback, by urgency or value. In practice it covers two jobs. The first is sorting: grouping duplicates, tagging themes and routing items to the right team. The second is deciding which items matter most, and that’s the job teams increasingly hand to AI.

Models are already good at the first job. The second is harder to check, and the guides that rank for this topic rarely try. The top result I read for “AI backlog management” lists what AI can do, like segmenting customers and forecasting demand, without a single accuracy number.

The stakes are high because human prioritization isn’t very accurate either. Ronny Kohavi’s 2012 summary of online experiments found that 60 to 90% of ideas don’t improve the metric they were designed to improve, depending on the domain. Pendo’s 2019 Feature Adoption Report, based on 615 product subscriptions, found that 80% of features in the average product are rarely or never used.

A lot of that waste starts with how requests get counted. Ranking by mention count measures who was still around to be counted, which we wrote about in September. An AI that learns to rank the same way only gets there faster.

AI matched a human on about 10 of 78 issue priorities

Jev is a decision model, which means it doesn’t write anything. You give it some input and a question with a fixed set of answers, and it returns one answer with a probability attached. Input costs $0.042 per million tokens and output is free.

In late September, Mastra tested it on 78 of their open-source GitHub issues. Jev had to mark each one urgent, high or low, and it only saw the issue’s title and body. Roughly 10 of the 78 answers matched the priority a Mastra teammate would have given.

That’s about 13%. A random pick between three labels would land around a third of the time. The comparison was against one person’s judgment, and 78 issues is a small sample, so I wouldn’t lean on the exact number, but a result that far below chance is hard to explain away.

Mastra explained why. Priority depends on who is affected, how bad the failure is, and what the team plans to build next. None of that is written in an issue’s title and body, so the model was asked a business question and handed a ticket.

A bigger model had the same problem

Anthropic’s data team published a very similar result in June with Claude. Answering analytics questions against company data, Claude was accurate no more than 21% of the time on their evals.

They tried the obvious fix first and gave it raw access to thousands of past queries. Accuracy moved by less than a point. What worked was adding skills that told Claude which sources to trust and in what order, and that took accuracy above 95% in aggregate. We covered that result in detail when it came out.

TypeSafe’s own documentation for Jev points the same way. It says accuracy drops as irrelevant material in the input grows, and it tells developers to filter in code and send only the fields a question needs. It also says Jev is “not a calculator”, so any weighting of revenue or scores belongs in code.

Across two models and two tasks, piling on more data barely moved accuracy, and choosing the right data before the model saw it is what changed the result.

The evidence that decides priority lives outside the ticket

Whether an issue is urgent depends on facts about customers and money, and an issue tracker holds very few of them. This is roughly where each one lives at a typical hyper-growth, SME or enterprise company:

A model that sees only the ticket is working from the first row at best. That’s the situation Mastra put Jev in, and the result looked like it.

[Embed: backlog-reranker.html] Caption: Try it on an example backlog. Eight requests start out ranked by mention count. Merge duplicates by account, add revenue, usage and roadmap fit, and all three of the top spots change. The data is illustrative.

The obvious move is to paste all of it into the prompt, and both Anthropic’s results and TypeSafe’s documentation say that doesn’t work. Raw customer data is mostly noise for any one decision. In one month of our own inputs at Bagel AI, 12.3 million tokens of raw feedback held 904,000 tokens of actual product signal, more than 12x less to read, which we wrote about earlier today.

For the cost side of this, What token-maxxing misses about the cost of product work models decision yield and the cost of each successful outcome.

So the work that decides whether AI prioritization is any good happens before the model is called. Somebody, or something, has to match each request to the right customer, collapse duplicates into one count, attach revenue and usage, and hand the model a short, scoped summary. Ranking math, like weighting by revenue or a north-star metric, should run in code on top of that.

What AI prioritization needs to work

Put the 2026 results together and AI prioritization works when five things happen in order:

1. Every signal is matched to a customer. A request that shows up in a sales call, a support ticket and a chat thread from the same account counts once, tied to that account.
2. The evidence is cut down to what the decision needs: affected accounts, the revenue behind them, deduplicated frequency, usage and roadmap fit, short enough for the model to read in one pass.
3. The model answers a bounded question, and code does the math. “Is this blocking a renewal?” is a good question for a model. Weighting that answer by revenue or a north-star metric is a job for code.
4. Confidence decides what gets automated. Running Dolphins’ open benchmark showed that the same 0.9 confidence cutoff gave 98.9% accuracy on one task and 81.6% on another, so thresholds have to be set per decision, on your own labeled data. Confident answers flow through and the rest go to a person.
5. Shipped decisions get checked against the outcome. With 60 to 90% of ideas failing to move their target metric, a prioritization system that never looks back can’t tell whether it’s getting better.

Where Bagel AI fits

Bagel AI is a product brain. It runs as an autonomous decision layer underneath the product org and covers that whole sequence. It resolves feedback from calls, tickets, chats and threads to one canonical customer, attaches revenue and usage, and ranks opportunities by the KPI you choose, whether that’s ARR, NPS, active users or a custom formula. Each opportunity becomes a scoped, dev-ready artifact that PMs and coding agents like Claude Code, Cursor and Codex can pull through MCP.

After a feature ships, Bagel AI measures whether it moved the metric it was picked for. Most prioritization tools stop before that step, and it’s the one that tells you whether the ranking was right.

How to test AI prioritization on your own backlog

You don’t have to take anyone’s word for this. A small version of Mastra’s test, with one change, will tell you how much evidence matters on your own data:

1. Pull a few hundred closed issues whose priority was reviewed by someone other than the person who filed them.
2. Ask a model for a priority using only the title and body. Record each answer and its confidence.
3. Run the same issues again with a short evidence block added: affected accounts, the revenue behind them, how often the request came up across channels, and whether it fits the current roadmap.
4. Pick a confidence threshold on one slice of the issues and score the results on a different slice, so the threshold isn’t tuned to the answers.
5. Compare agreement with the reviewed priority, how many truly urgent issues each run caught, how many answers cleared the threshold, and the cost, including the work of assembling the evidence.

The gap between runs 2 and 3 is roughly what the missing context costs you. If it’s large, the model was never the bottleneck. If it’s small, your labels or your data deserve a look before any AI goes near the roadmap.

Models that make these calls will keep getting cheaper. The issues in your tracker will keep arriving with a title and a body and nothing about who’s waiting on them, until something upstream fills that in.

The numbers behind this piece