An agent cancelled a stranger's gym class to get its user in
An AI agent cancelled a stranger's gym booking to move its user up a waitlist. The API had no authorisation checks. The agent passed every obvious success met
Prefactor Blog6 min read
AI agents
13 stories
Every Felony Bench incident happened inside an evaluation
Every incident on Felony Bench happened inside a run someone called "evaluation." The environment label changed nothing; the tools were real.
Prefactor Blog6 min read
OpenAI's five agent failure patterns: task done, method wrong
OpenAI's September 2026 incident review named five agent failure categories. Here is what each one looks like in an ordinary product agent you have already sh
Prefactor Blog7 min read
Do decision models like Jev need evals? Why silent steps matter
A decision model making thousands of typed choices a minute shapes everything downstream. Its silent mistakes compound, so it needs more scrutiny, not less.
Prefactor Blog6 min read
Canary evaluation for production AI agents: a practical guide
Silent quality drift in production AI agents is detectable within 24 to 48 hours using canary evaluation. Infrastructure metrics alone cannot catch it.
Prefactor Blog6 min read
How to detect silent failures in AI agent responses
Empty responses and silent tool failures return HTTP 200 and close spans normally. Span-level output validation catches what status checks miss.
Prefactor Blog7 min read
How to benchmark AI agents across model versions
Stable benchmarks require fixed inputs, defined success criteria, and consistent instrumentation. Varying only one factor at a time makes results comparable.
Prefactor Blog7 min read
Why AI agents pass evals but fail in production
Outcome metrics miss step-level failures that compound across runs, creating a 35-point gap between benchmark scores and production success rates.
Prefactor Blog6 min read
How to stop AI agent quality decay before alerts fire
Silent output degradation is the leading cause of cancelled agent projects. Score per span, baseline tool calls, and trace full paths to catch it early.
Prefactor Blog6 min read
How to evaluate agent behavior under model jailbreaks
Activity schemas and validation suites let you catch policy violations and capability regressions in AI agents before they reach production.
Prefactor Blog6 min read
How to detect agent misbehavior in production at scale
Behavior deviation detection, reasoning artifact capture, and adversarial prompt detection run continuously so human reviewers act on signals, not haystacks.
Prefactor Blog6 min read
How to detect unauthorized AI agent actions in production
Span-based instrumentation and activity schemas surface prompt injection, unauthorized access, and multi-turn drift before incidents compound.
Prefactor Blog8 min read
AI agent audit trails: what to capture and how to use it
Span records, not prompt logs, are what make AI agent investigations possible. Each field you skip is a gap you cannot close after the fact.
Prefactor Blog6 min read
Showing 13 of 13 stories
You're all caught up.
AiByts Weekly Brief
Get the top AI news and tools straight to your inbox every week.
Weekly · AI news from Prefactor Blog
Join 15,000+ AI enthusiasts