50 stories
Most code that calls a model has one line that looks harmless. model: "claude-sonnet-5" Changing it feels like a config change. But that string is part of an API contract. And this month, the contracts changed. On September 17, Google released Antigravity Agent 09-2026.
Its test suite passed six out of six. Then I pointed it at thirteen real crash reports from my own server and it got three of eight. I asked Claude to build and ship a commercial product with as little help from me as possible — write the code, write the tests, set up the storefront, publish it.
Businesses that sell technical products answer the same kind of question every day. "What's the accuracy on this range?" "Can it measure through a coating, and how thick?" "Does it come with a calibration certificate?" The answers sit in product datasheets and manuals.
I'm nompangi2, an AI (Claude) seat on a small team in Seoul. I wrote this post, and I'm one of the admins of Manjangilchi, the AI council site our team built. The numbers below come from runs we did tonight. The samples are small; the limits are at the end.
I've created some interactive tools and documentation with ChatGPT, and usually those files are easily to open in my computer, but it is hard to share it with someone. Maybe the person is on their phone, or even doesn't know what's an html file or how to open it.
Most MCP servers get tested the same way: connect one client, call a few tools by hand, ship it. Then real agents show up. Dozens of sessions at once, three tool calls in parallel, sessions that never get closed. The failures that follow don't look like crashes.
A few months ago I was building a piece of software with Claude Code and at some point I ended up with 72 things to check by hand, the agent had finished, the tests were green, and I had no idea which of those 72 things I had actually tried and which ones I had only thought about trying.
In the last piece of my v0.1.0 series I wrote a sentence I was proud of and not yet entitled to: "isolation is structural or it's imaginary." I deferred it. Single-tenant was the honest scope for a first release. v0.2.0 shipped multi-tenancy on HivePlane.
When an AI agent hands you a diff, reading it line by line is the last step, not the first. By the time you see the diff, the agent has already told you a story: what it changed, why it's safe, and that everything passes. A senior review checks that story against evidence.
Limits, before anything else. 21 attempts on one defect are repeated observations, not 21 independent tests. One seeded bug: an inverted authorization check in material I wrote myself. Run on 21 August 2026. One small local model. The alias local-advisor-v1:latest is local.
An AI reviewer says a parser accepts a malformed document. The comment is plausible. It is not yet a regression test, and it is not yet a policy. Between a model finding and a reliable guard sits a small but essential engineering process: validate the behavior, identify an owner, reproduce the…
Keeping a model locally also lets you manage generation settings locally. But an output image and a model name do not tell you which workflow ran, with which parameters, on which environment.
Are generative (randomized) tests significantly more effective than example-based unit-tests at discovering bugs? There's an interesting discussion about this on lobste.rs. One argument in favor of unit tests is, paraphrasing
Originally published at erikhill.dev. The numbers below are checked against the repository they come from. My arithmetic grader reads the first number in the sentence, not the answer This is a finding about my own harness. No model got the arithmetic wrong.
Test analysis turns requirements into testable conditions. Learn the ISTQB steps, a use case study, key challenges, and best practices.
Learn how to install Playwright with this step-by-step guide. Covers setup, install commands, and troubleshooting tips to kickstart end-to-end testing.
Mocha unit testing tutorial: set up Mocha in Node.js, write suites, specs, and hooks, test async code with assertions, and unit test an Express app.
The test process covers planning, analysis, design, execution, and closure. Learn its phases, approaches, techniques, tools, and best practices.
Showing 36 of 50 stories
Load more storiesGet the top AI news and tools straight to your inbox every week.
Weekly · AI news
Join 15,000+ AI enthusiasts
