Urban Institute Urges Guardrails as Governments Experiment With Agentic AI

State and local governments are beginning to experiment with artificial intelligence (AI) systems capable of taking actions on their own. According to new guidance issued by the Urban Institute, making sure governance structures in place before those systems become embedded in public services is an essential first step.

The institute’s new agentic AI playbook is designed to help state and local officials prepare for and maintain responsible deployments as governments look to the technology to cut costs, streamline operations and improve services. Unlike conventional generative AI, agentic systems can go beyond producing content to independently plan and act toward a goal.

Agentic AI “often has some autonomy to make future decisions or to think about future decisions and plan then take action,” Graham MacDonald, the Urban Institute’s chief information officer and vice president of technology and data science, told Route Fifty.

That added autonomy is already being tested in government.

Virginia launched a pilot under former Gov. Glenn Youngkin that used agentic AI to review regulatory and guidance documents for redundant or burdensome language. According to Virginia’s Office of Regulatory Management, the initiative identified more than $1.4 billion in annual savings and helped reduce permit and license processing times by nearly 80%.

Utah’s Office of Artificial Intelligence Policy, meanwhile, partnered with health platform Doctronic on a pilot examining whether autonomous AI could assist with routine prescription renewals. Preliminary results showed the system recommended renewal in 72% of cases, with physicians agreeing with 91% of those recommendations. Boston has also integrated agentic AI into its open-data portal, using an Anthropic model context protocol server to analyze municipal data and help inform policy queries.

Those examples illustrate both the potential and the governance problem identified by the Urban Institute. Agentic AI tools can be created and deployed by employees outside traditional IT departments.

“What’s different about the agentic generative AI is that the IT person is not the only person who is empowered to build it,” MacDonald said. That democratization can accelerate innovation but also create opportunities for errors, workflow disruptions and harms, particularly when agents are incorporated into sensitive functions such as benefits administration.

The playbook therefore recommends that agencies document basic information before deployment, including an agent’s intended purpose, target population, measurable outcomes and estimated costs. Governments should separately define accuracy and fairness rather than relying on generic performance measures. Accuracy, for example, should account for the completeness, clarity and reliability of an agent’s output under varying conditions.

The level of oversight should also reflect how an agent will be used. Internal systems can initially operate at smaller scales and may require less stringent fairness and bias controls, MacDonald said. Constituent-facing systems require greater scrutiny, including fairness metrics designed around the particular population and government service involved.

For an agent helping residents apply for disability benefits, for instance, officials could test whether recommendations differ across populations and measure how often the system provides correct answers to diverse users.

The Urban Institute also recommends formalizing accountability rather than treating agentic AI solely as an IT function. Its framework identifies seven roles: accountable owner, evaluation lead, security lead, transparency lead, responsible AI lead, contributor and independent review. Their responsibilities can include maintaining audit trails, conducting evaluations, gathering stakeholder feedback and monitoring harms.

“I think it’s incumbent on everyone in their organizations to establish a good process,” MacDonald said, including “bringing more people into the process” to establish common goals, metrics and standards.