Why the LLM in my automation agent never clicks anything
I'm a computer engineering student in Manaus, Brazil, and for my research project I've been building ANCHOR, a web automation agent: you describe a task in plain language ("Register Maria Silva, IT department, contractor, accept the terms and save"), and it runs the task in the browser.
This article details the development of ANCHOR, a web automation agent that leverages large language models (LLMs) for task planning while using a deterministic engine for execution, addressing limitations in traditional RPA and LLM-based automation.
What happened
The author developed ANCHOR, an automation tool that uses LLMs to generate task plans (e.g., 'fill Full name', 'click Save registration') while a non-AI engine executes actions. This approach addresses issues with traditional RPA's rigidity and LLMs' tendency to produce false successes. The article presents benchmark results comparing ANCHOR to other methods.
Why it matters
For AI developers and automation professionals, this approach demonstrates a practical way to combine LLM capabilities with deterministic execution, creating more robust automation solutions that adapt to UI changes without constant reconfiguration.
- Separating AI planning from execution improves automation reliability
- Continuous benchmarking reveals hidden flaws in AI systems
- Security considerations are critical when integrating LLMs with sensitive workflows
The bigger picture
The design could influence future automation tools by providing a framework for reliable AI-assisted workflows, reducing dependency on fixed selectors while maintaining control over critical actions.
ANCHOR demonstrates a viable approach to web automation by combining LLM planning with deterministic execution, offering improved resilience to UI changes and better error handling than pure LLM or traditional RPA methods.
Technical details
- Uses Qwen 3.5 models (4B/9B) via Ollama for planning
- Deterministic engine with heuristic element matching (text, labels, roles)
- Benchmarking with 5 levels of UI modifications (renamed IDs, hidden content, etc.)
- Password security measures: ANCHOR never types passwords or processes them in requests
FAQ
How does ANCHOR differ from traditional RPA?
Traditional RPA uses fixed selectors that break with UI changes, while ANCHOR uses LLMs to generate plans and a deterministic engine to execute them, adapting to page changes through user feedback.
Why does ANCHOR avoid letting the LLM perform clicks directly?
LLMs often produce 'false successes' where they claim tasks are done but don't actually complete them. ANCHOR's deterministic engine ensures actions are verified, reducing this risk.
How does ANCHOR handle security concerns?
ANCHOR never processes passwords in AI requests, blocks tasks containing passwords, and requires manual login for sensitive operations, maintaining security while using LLMs for planning.
This summary and analysis were generated with AI from the original article and may contain errors. Read the original for the full story.
Read the full story
Originally published by Dev.to AI on Oct 10, 2026.
Comments
Sign in to join the discussion.