Spending Limits for LangGraph Agents That a Prompt Injection Can't Talk Past

What happened

The article explains that spending limits written into a LangGraph agent's system prompt are not reliable controls. Any text the agent reads, including adversarial inputs, can override these limits. The solution is to move the enforcement of spending limits to the tool boundary, where a server or policy engine checks the request against real budgets and rules before any payment is executed. This approach is demonstrated using Pink Agentic AI Payments and a sandbox environment.

Why it matters

This is important for AI developers and security professionals because it highlights the risks of relying solely on system prompts for enforcing rules in AI agents. Moving enforcement to the tool boundary provides a more secure and reliable method of controlling agent behavior, especially in scenarios involving financial transactions.

• Spending limits in system prompts are not reliable controls for AI agents.
• Enforce spending limits at the tool boundary using a server or policy engine.
• Use sandbox environments to test and validate the effectiveness of these controls.
• Consider combining tool-boundary checks with human-in-the-loop processes for added security.
• Avoid relying solely on the model's ability to interpret and enforce rules.

The bigger picture

The practical impact of this approach is that it reduces the risk of adversarial inputs bypassing spending limits, thereby improving the security and reliability of AI agents in financial contexts. It also provides a clear framework for implementing and testing such controls in development environments.

The article concludes that moving the enforcement of spending limits to the tool boundary, rather than relying on system prompts, is a more secure and effective approach. This method ensures that adversarial inputs cannot override the rules, and it provides a clear framework for implementing and testing such controls in AI agent systems.

Technical details

• LangGraph agents use system prompts to define spending limits, but these are treated as suggestions by the model.
• The tool boundary enforces rules by checking requests against real budgets and rules before any action is taken.
• Pink Agentic AI Payments is a tool that enforces these checks by evaluating each payment request against a set of predefined rules.
• The article provides a code example using LangGraph and a sandbox environment to demonstrate the issue and solution.

FAQ