Does Jev model routing need evaluation? The unmeasured failure
A Jev router that sends a hard request to a cheap model logs a clean 200 and a cost saving. The wrong answer lands on the customer, not in your logs.
Prefactor Blog6 min read
evaluation
38 stories
Can Jev typed outputs still be wrong? Type safety vs correctness
Jev guarantees a valid instance of the requested type, not a correct decision. Typed outputs still require behaviour evals to catch semantic errors.
Prefactor Blog6 min readHow UK AISI and EvalEval Are Making Benchmark Results Reproducible
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Hugging Face Blog4 min readWhat is Jev? Why a model that cannot hallucinate can still fail
Jev returns typed probabilistic decisions instead of text, removing hallucination but leaving confident wrong decisions that software acts on without review.
Prefactor Blog6 min readCanary evaluation for production AI agents: a practical guide
Silent quality drift in production AI agents is detectable within 24 to 48 hours using canary evaluation. Infrastructure metrics alone cannot catch it.
Prefactor Blog6 min readHow to detect silent failures in AI agent responses
Empty responses and silent tool failures return HTTP 200 and close spans normally. Span-level output validation catches what status checks miss.
Prefactor Blog7 min readHow to benchmark AI agents across model versions
Stable benchmarks require fixed inputs, defined success criteria, and consistent instrumentation. Varying only one factor at a time makes results comparable.
Prefactor Blog7 min readWhy AI agents pass evals but fail in production
Outcome metrics miss step-level failures that compound across runs, creating a 35-point gap between benchmark scores and production success rates.
Prefactor Blog6 min readHow cascading failure compounds across multi-step agent workflows
A cascade starts when one step returns a plausible wrong output and later steps build on it. Boundary validation and checkpoints contain it; retries do not.
Prefactor Blog8 min readStep-level evaluation in production agents: what to measure
Final-output benchmarks hide compounding failures. Instrumenting each step shows where accuracy drops and why workflows fail at scale.
Prefactor Blog6 min readHow to stop AI agent quality decay before alerts fire
Silent output degradation is the leading cause of cancelled agent projects. Score per span, baseline tool calls, and trace full paths to catch it early.
Prefactor Blog6 min readWhy 85% step accuracy collapses to 20% end-to-end success
At 85% per-step accuracy, a ten-step agent completes the whole task about one time in five. Final-output scoring cannot see this; trajectory evaluation can.
Prefactor Blog8 min readStep-level agent evaluation: measure every intermediate step
At 95% per-step accuracy, a 10-step agent succeeds only 59% of the time. Step-level spans show which step fails and why.
Prefactor Blog6 min readHow to evaluate agent behavior under model jailbreaks
Activity schemas and validation suites let you catch policy violations and capability regressions in AI agents before they reach production.
Prefactor Blog6 min readWhy multi-step agents fail in production but pass benchmarks
Step repetitions cause 17% of multi-agent failures and reasoning-to-action mismatches cause 14%, both undetected by final-output evaluation.
Prefactor Blog6 min readHow to detect agent misbehavior in production at scale
Behavior deviation detection, reasoning artifact capture, and adversarial prompt detection run continuously so human reviewers act on signals, not haystacks.
Prefactor Blog6 min readHow to detect unauthorized AI agent actions in production
Span-based instrumentation and activity schemas surface prompt injection, unauthorized access, and multi-turn drift before incidents compound.
Prefactor Blog8 min readAI agent audit trails: what to capture and how to use it
Span records, not prompt logs, are what make AI agent investigations possible. Each field you skip is a gap you cannot close after the fact.
Prefactor Blog6 min read
Showing 36 of 38 stories
Load more storiesAiByts Weekly Brief
Get the top AI news and tools straight to your inbox every week.
Weekly · AI news
Join 15,000+ AI enthusiasts