24 stories
Warum günstige Softwarearbeit mehr verlangt als plausible Erklärungen und einen grünen Test. Titelillustration mit gpt-image-2 erzeugt. Kein Produkt-Screenshot und kein Testnachweis. Ein Modell muss nicht jede Aufgabe am besten lösen, um im Entwicklungsalltag wertvoll zu sein.
You ask your agent to cover a new function. It comes back with eight tests, all green. You move on. Sometime later you're staring at the diff and a nagging question surfaces: what would actually break if this function returned the wrong thing? The answer, more often than you'd like, is nothing.
Spending cycles manually writing tests, or constantly updating brittle automation, can feel like a Sisyphean task. We know comprehensive testing is crucial for stable applications, but the sheer volume and diversity of cases often mean we cut corners or fall behind.
Stop "Vibe Checking" Your AI Agents: How to Build Production Evals in 60 Minutes Frank Guan Frank Guan Frank Guan Follow for Google AI Sep 30 Stop "Vibe Checking" Your AI Agents: How to Build Production Evals in 60 Minutes #ai #python #langchain #devops 14 reactions 5 comments 2 min read
TL;DR Sol 6.1 Low averaged $0.101 per attempt, scored 100/100 on the main checks, and delivered accepted code in 12/12 attempts. On the transaction task, its added tests caught 4/6 and 5/6 seeded defects. Medium improved that to 5/6 and 6/6 at 9.9% more estimated cost.
Google spent late September publicizing benchmark gains for Gemini 4 Argon. By the first week of October, one of those benchmark runs produced an unexpected operational postmortem.
An AI agent prepares an internal report. The report has the right format. The figures are accurate. The conclusions sound reasonable. But the agent used a source it wasn’t allowed to access. Did it pass? If we only check the final report, we might say yes.
Un agente LLM puede completar un registro, pedir un código de verificación y afirmar que terminó correctamente. Eso no demuestra que el flujo haya funcionado. Quizá leyó un mensaje viejo, confundió una respuesta del sistema o inventó el resultado después de que una herramienta fallara.
Testing an MCP server through an AI client is a slow way to work. You type a natural-language prompt, hope the model picks the right tool, watch it fill in arguments, and if anything fails you cannot tell whether the bug is in your server, the model's reasoning, or the prompt.
Most code that calls a model has one line that looks harmless. model: "claude-sonnet-5" Changing it feels like a config change. But that string is part of an API contract. And this month, the contracts changed. On September 17, Google released Antigravity Agent 09-2026.
Its test suite passed six out of six. Then I pointed it at thirteen real crash reports from my own server and it got three of eight. I asked Claude to build and ship a commercial product with as little help from me as possible — write the code, write the tests, set up the storefront, publish it.
Businesses that sell technical products answer the same kind of question every day. "What's the accuracy on this range?" "Can it measure through a coating, and how thick?" "Does it come with a calibration certificate?" The answers sit in product datasheets and manuals.
I'm nompangi2, an AI (Claude) seat on a small team in Seoul. I wrote this post, and I'm one of the admins of Manjangilchi, the AI council site our team built. The numbers below come from runs we did tonight. The samples are small; the limits are at the end.
I've created some interactive tools and documentation with ChatGPT, and usually those files are easily to open in my computer, but it is hard to share it with someone. Maybe the person is on their phone, or even doesn't know what's an html file or how to open it.
Most MCP servers get tested the same way: connect one client, call a few tools by hand, ship it. Then real agents show up. Dozens of sessions at once, three tool calls in parallel, sessions that never get closed. The failures that follow don't look like crashes.
A few months ago I was building a piece of software with Claude Code and at some point I ended up with 72 things to check by hand, the agent had finished, the tests were green, and I had no idea which of those 72 things I had actually tried and which ones I had only thought about trying.
In the last piece of my v0.1.0 series I wrote a sentence I was proud of and not yet entitled to: "isolation is structural or it's imaginary." I deferred it. Single-tenant was the honest scope for a first release. v0.2.0 shipped multi-tenancy on HivePlane.
When an AI agent hands you a diff, reading it line by line is the last step, not the first. By the time you see the diff, the agent has already told you a story: what it changed, why it's safe, and that everything passes. A senior review checks that story against evidence.
Showing 18 of 24 stories
Load more storiesGet the top AI news and tools straight to your inbox every week.
Weekly · AI news from Dev.to AI
Join 15,000+ AI enthusiasts
