Insight · Checklist
AI Agent Governance Checklist
How to use this checklist
Agents differ from single-step AI in that they plan and act across systems, reading data, calling tools, writing records, and looping until a goal is met. That autonomy creates failure modes absent from prompt-and-reply interactions. This checklist covers eight governance phases before any agent reaches production: from scope and authority limits through to human accountability. Items marked BLOCKING should stop deployment; ADVISORY items can be accepted as known debt with a written reason. Forty-seven checks in total.
77-word direct answer
Key takeaways
- An agent's scope must be bounded before it is built, not discovered after the first incident.
- BLOCKING items should stop a deployment. ADVISORY items can be accepted as documented debt, but record the acceptance, who made it, and what would change it.
- Agents inherit whatever permissions they are given. Least-privilege scoping and explicit revocability are not optional.
- The difference between an agent doing something wrong and an agent doing something catastrophically wrong is usually blast-radius limits set at design time.
- Human accountability means a named person: not a team, not a tool, not “the AI team.”
The eight phases
Work in order. Phases one and two block the most deployments. Phase seven is ongoing and should be revisited whenever the agent's scope or volume changes.
| Phase | Question it settles | BLOCKING items |
|---|---|---|
| 1 · Scope and authority | What is the agent permitted to do, and where does its authority end? | 4 |
| 2 · Permissions and account ownership | Whose identity does it act under, and who can revoke it? | 5 |
| 3 · Approval gates and stop conditions | When must it pause for human authorization, and what makes it stop entirely? | 4 |
| 4 · Action logging and reconstructability | Can every action be attributed, reviewed, and replayed? | 4 |
| 5 · Testing and evaluation | Has it been tested against the full range of inputs it will encounter? | 4 |
| 6 · Rollback and blast-radius limits | How much damage can a single bad run cause, and how is it reversed? | 5 |
| 7 · Monitoring in operation | Will you know when it fails, and can you catch drift before it becomes harm? | 4 |
| 8 · Human accountability | Is there a named person accountable for what the agent does? | 2 |
32 BLOCKING checks and 15 ADVISORY checks, 47 in total.
This is practitioner guidance, not legal advice
This checklist reflects operational practice for deploying agentic AI systems in organizational settings. It does not constitute legal, privacy, cybersecurity, or regulatory compliance advice.
Agents that operate in regulated domains (financial services, healthcare, employment, education) face requirements that go beyond what any practitioner checklist can address. If your agent touches regulated data or decisions, qualified legal and compliance counsel should review the deployment before it goes live.
What an agent may do unattended vs. what requires authorization
Sized by consequence. The right column is not a technical constraint; it is a governance decision that must be made before the agent is built.
| Action type | Consequence level | Permitted unattended? | If not: what is required |
|---|---|---|---|
| Read internal data (non-sensitive) | Low | Yes, with logging | Least-privilege scope; access documented |
| Read confidential or personal data | Moderate | Only with policy authorization | Data-classification review; documented approval; logged access |
| Draft internal content (not sent) | Low | Yes | Human review before any external use |
| Send external communications | Moderate–High | No | Named human approver; outbound must be reviewed before send |
| Write to a record in a system of record | Moderate | Only if idempotent and reversible | Oversight tier set; write path documented; rollback tested |
| Delete or archive records | High | No | Human authorization per action; audit log retained |
| Trigger financial transactions | High | No | Human authorization; monetary threshold; dual approval above limit |
| Modify user permissions or access | High | No | Named security owner; human-in-the-loop at every step |
| Call third-party APIs with side effects | Moderate–High | Only with scoped credentials and rate limits | Rate limits set; failure behavior defined; retry is idempotent |
| Chain sub-agent calls | Variable | Only if each sub-agent passes its own scope check | End-to-end authorization chain documented; combined blast radius assessed |
Consequence levels are starting judgments, not a standard. Your risk posture may promote or demote any row. What is not defensible is leaving it unexamined.
Phase 1 · Scope and authority
Settled before design. Four BLOCKING, two ADVISORY.
BLOCKING · The agent's goal is stated as a bounded task, not an open-ended mandate
“Summarize and route inbound inquiries from the support queue” is a bounded task. “Handle customer support” is a mandate. The agent will find the edges of an unbounded scope in production, and probably not in a way you anticipated.
BLOCKING · Actions the agent may NOT take are listed explicitly
A permit list is not enough. Name what is out of bounds: sending external communications, modifying permissions, accessing files outside the defined directory, calling APIs not on the approved list. The agent should fail loudly when it reaches a boundary, not try to route around it.
BLOCKING · The scope has a named owner who approved it
A person with authority over the domain the agent operates in reviewed and signed off on the defined scope. Not the engineer who built it, and not the person most enthusiastic about AI.
BLOCKING · The definition of task completion is explicit
The agent knows when it is done. An agent with an unclear termination condition will loop, retry, or escalate in ways that compound errors. Write the exit condition before you write the first prompt.
ADVISORY · The agent's scope is documented in a form non-technical stakeholders can read
The people accountable for oversight need to understand what the agent does without reading code or prompts. A plain-language scope statement is worth writing.
ADVISORY · An adjacent second capability is identified, and deliberately excluded for now
Knowing what the agent might be extended to do later shapes how it is built, without over-building for a use case that has not been governed.
Phase 2 · Permissions and account ownership
Start this on day one. Five BLOCKING, two ADVISORY.
BLOCKING · The agent runs under a dedicated service identity, not a person's account
An agent on an individual's credentials inherits everything that person can access, breaks when their role changes, and produces an audit trail that attributes actions to a human rather than to the system. This is the most common permission error on first deployments.
BLOCKING · The service identity has least-privilege scopes
The agent can read and write only what the defined task requires. Document the scope explicitly: not “access to Salesforce” but “read access to the support queue object; write access to the case status field only.”
BLOCKING · Every account the agent uses is owned by the client organization
API keys, platform credentials, and service accounts must sit with the organization, not with a vendor, consultant, or tool provider. An agent that depends on credentials you do not own is a single vendor decision away from going dark.
BLOCKING · Access is revocable in under five minutes, without a support ticket
Someone inside the organization can remove the agent's access immediately and unilaterally. If that is not true, the agent cannot be stopped quickly when something goes wrong. And something will go wrong.
BLOCKING · The person who can approve the access is identified by name
Not the team that theoretically owns it. A named individual who has the authority and the information to approve the specific scopes being requested. Unowned access decisions are the single most common cause of a stalled agent deployment.
ADVISORY · An access review date is scheduled
Scopes widen during troubleshooting and rarely narrow again afterward. A calendar entry for an access review in ninety days catches the drift.
ADVISORY · Credential rotation is possible without agent downtime
You know how to rotate a key and what the agent does during the rotation window. Discovering this process for the first time during an incident is avoidable.
Phase 3 · Approval gates and stop conditions
Sized to consequence. Four BLOCKING, two ADVISORY.
BLOCKING · Actions above a defined consequence threshold require human authorization before execution
Write the threshold explicitly: “Any outbound communication requires approval,” or “Any write to a record flagged as confidential requires named-reviewer sign-off.” Ambiguity at the gate means the gate is not real.
BLOCKING · The agent has a hard stop condition that a human can trigger at any time
A kill switch that requires no technical expertise to operate and that takes effect within one tool call. The person who triggers it should not need to call the engineer who built it.
BLOCKING · The agent detects and halts on ambiguity rather than guessing
When the agent encounters a situation its task definition does not cover, it surfaces the ambiguity to a human rather than picking the closest match and continuing. Silent guessing in agentic systems compounds across steps.
BLOCKING · A maximum step limit is set and enforced
The agent cannot run indefinitely. A bounded loop count or a wall-clock timeout stops a runaway plan before the consequences scale. The limit should be low enough to be meaningful; not 10,000 steps for a task that normally takes 8.
ADVISORY · The approval gate process is documented and tested before launch
Walk through the gate process once with a real case before go-live. Discovering that the named approver does not have access to the review interface during a live run is a common and avoidable problem.
ADVISORY · A re-escalation path exists if the initial approver is unavailable
A named backup. The agent should not stall indefinitely because the approver is on leave.
Phase 4 · Action logging and reconstructability
Four BLOCKING, two ADVISORY.
BLOCKING · Every action the agent takes is logged with enough context to reconstruct it
At minimum: the step number in the plan, the tool or API called, the input passed to it, the output returned, the model and version, and the timestamp. Reconstructability is what makes an incident investigable rather than speculative.
BLOCKING · The log is immutable and stored outside the agent's own write path
An agent that can modify or delete its own logs has no audit trail. Logs go to a system the agent has no write access to.
BLOCKING · The full plan, not just individual tool calls, is captured
Agentic systems produce plans as well as actions. The reasoning chain matters: why the agent chose a particular sequence of steps, not just what it executed. Single-call logging misses this.
BLOCKING · Log retention is bounded and consistent with the data obligations of the content logged
Logs that contain personal data or confidential information are subject to the same obligations as that data. Indefinite retention is not a neutral default.
ADVISORY · Logs are queryable by case or task ID
Not just a raw append log. Being able to retrieve the full trace for a specific task without sifting through an entire day's output makes review practical rather than theoretical.
ADVISORY · A sample of logs is reviewed on a regular cadence
Logging without review is a liability that looks like compliance. Even a weekly ten-minute sample review builds intuition about where the agent is drifting.
Phase 5 · Testing and evaluation
Four BLOCKING, two ADVISORY.
BLOCKING · An evaluation set of real historical cases exists, including adversarial and edge inputs
Thirty to fifty real cases with known good outcomes, deliberately including the awkward ones: the ambiguous request, the record with missing fields, the input that spans multiple domains. Curated pilot examples are not a substitute.
BLOCKING · The agent has been tested for behavior at scope boundaries
What does it do when it encounters a resource outside its permissions? A tool that returns an error? A task that requires an action it is not authorized to take? These edge cases should be tested, not hoped about.
BLOCKING · The agent has been tested for multi-step error propagation
An error in step two that causes silent downstream effects in steps four and seven is the specific failure mode of agentic systems. Test for it explicitly, not just for the accuracy of individual tool calls.
BLOCKING · Failure behavior is defined and tested for each tool the agent calls
What does the agent do when an API is unavailable? When it receives a rate-limit error? When a write fails? These paths should be explicit and tested before launch, not discovered in production.
ADVISORY · The evaluation set is retained and re-runnable against future versions
The evaluation set is not a one-time artifact. It is the regression suite for every future change to the agent's model, prompt, or tool configuration.
ADVISORY · A red-teaming exercise has been conducted before high-consequence deployments
For agents with access to sensitive data or with the ability to send external communications, deliberate adversarial testing, attempting to make the agent exceed its defined scope, is worth the time.
Phase 6 · Rollback and blast-radius limits
The blast-radius questions are the ones teams most regret not answering before launch. Five BLOCKING, one ADVISORY.
BLOCKING · The worst-case impact of a single bad run is defined and bounded
If the agent ran without stopping for its maximum step limit and everything went wrong, what is the largest possible damage? Write this down. If the answer is “we don't know,” that is the finding.
BLOCKING · Writes are reversible, or reversal is explicitly impossible and accepted
Every action that modifies a system of record must either be reversible (with the reversal process documented) or designated as irreversible with written acceptance by the named scope owner. There is no third option.
BLOCKING · A rollback to the prior process is documented and can be executed without the agent's builder
The step-by-step path back to how the work was done before the agent. Written, tested, and operable by a second person.
BLOCKING · Rate limits and concurrency bounds are set on every external call the agent can make
An agent that can spawn unlimited parallel calls or retry without backoff can exhaust API quotas, trigger rate-limit bans, or send the same external message dozens of times. Set explicit limits before launch.
BLOCKING · A financial spend cap is set if the agent can incur costs
Any agent that calls APIs with per-call pricing, sends communications that incur costs, or triggers transactions needs a hard spend cap and an alert before that cap is reached.
ADVISORY · Scope creep in agent goals is defined as a trigger for re-review
An agent approved for one task can drift into adjacent tasks through prompt modifications or expanded tool lists. A written statement of what change would require returning to phase one is worth having.
Phase 7 · Monitoring in operation
Four BLOCKING, two ADVISORY.
BLOCKING · Failures alert a named person, not a shared inbox or a dashboard nobody checks
The alert goes to a specific individual who will act on it. Named backup for when that person is unavailable. A shared channel counts only if someone has explicit ownership of it.
BLOCKING · Silent stops are detected
An agent that simply stops running is invisible, because success was also quiet. Alert on the absence of expected output volume, not only on error messages.
BLOCKING · Behavioral drift is tracked
Agents using large language models are subject to model updates, prompt sensitivity, and data drift. Track override rates, error rates, and output quality on a regular cadence, not just at launch.
BLOCKING · An escalation path for unexpected agent behavior is defined and tested
What does a human do when they see the agent doing something that looks wrong but has not triggered an error? The answer should be written down, not improvised.
ADVISORY · Override rate is tracked as a quality signal
How often does a human reviewer materially change or reject an agent output? A rate that never falls after launch is a signal the agent is not calibrated for production inputs.
ADVISORY · The monitoring configuration is itself reviewed on a cadence
Alert thresholds set at launch often become wrong as volume changes. Schedule a review of monitoring configuration at the same time as the policy review cadence.
Phase 8 · Human accountability
Two BLOCKING, two ADVISORY. Accountability is not a technical property; no amount of logging substitutes for a named person.
BLOCKING · A named individual is accountable for what the agent does
Not a team. Not a tool vendor. Not “the AI.” A person whose role includes taking responsibility for the agent's outputs, including when those outputs harm someone. This person should know they hold this accountability before the agent goes live.
BLOCKING · The accountability structure is documented and visible to affected stakeholders
If the agent touches customers, employees, or other people who may be affected by its outputs, those people should have a path to raise a concern, and that path should lead to the accountable person, not a generic contact form.
ADVISORY · The agent's existence and general function is disclosed to people it interacts with, where relevant
Agents that interact with humans, such as drafting communications on behalf of a person or responding to inquiries, should be disclosed as such. The threshold for when disclosure is required will depend on the context; the default should be toward transparency.
ADVISORY · A deliberate retirement condition is defined
What would cause you to shut this agent down? Low adoption, a model update that changes behavior in an unacceptable way, a change in the scope it was approved for? Knowing the retirement condition in advance makes a future decision faster and more defensible.
First-hand experience
Which phases block the most deployments
Phase two, permissions and account ownership, blocks more agent deployments than any other. The gap between knowing which team owns access and knowing which specific person can approve it is where weeks disappear. Agents that operate across multiple systems compound this: each integration has its own access owner, and those owners rarely know each other.
Phase three, approval gates, is the one most often skipped with the reasoning that “the agent will be supervised closely at first.” That reasoning is usually sincere and usually wrong. Close supervision at launch degrades within days under production workload, and gates that were not built into the agent from the start are not added retroactively.
Phase six, blast radius, is the question that teams most regret not answering in writing before launch. The answer “we don't know the worst-case impact” is itself useful information that should either generate a design constraint or an explicit written acceptance. Finding it out during an incident is a much more expensive way to learn it.
On accepting an ADVISORY item as known debt
ADVISORY items can be deferred. The discipline is recording the deferral: what was accepted, who accepted it, and what would make it worth addressing. A one-line note against the item is enough.
Undocumented debt is indistinguishable from an oversight twelve months later, when no one who was in the room for the decision is still around.
Limitations of this checklist
This checklist is written for organizations deploying agentic AI in internal operational workflows, using commercially available models and platforms, without a dedicated AI safety or platform-engineering team. It does not address training or fine-tuning your own models, deploying agents in real-time or safety-critical control systems, or agentic systems operating under a named regulatory regime that prescribes specific controls.
The BLOCKING-versus-ADVISORY split reflects a judgment about typical organizational risk postures, not a universal standard. Your situation may reasonably promote an ADVISORY item to BLOCKING; a genuinely high-consequence deployment probably should. What is not defensible is skipping an item without noticing.
This checklist does not replace a security review. Agents that call external APIs, access sensitive data, or execute code introduce attack surfaces not covered here; the OWASP LLM Top 10 addresses several of them.
Forty-seven items will feel like a lot for a small deployment. Most are questions with an existing answer that just needs to be stated out loud. The ones that take real time are access provisioning (phase two), blast-radius assessment (phase six), and building a real evaluation set (phase five). All three are cheaper to address before launch than after.
External references
Useful alongside this checklist for the technical and standards dimensions.
- AI Risk Management Framework (AI RMF 1.0)NIST
- OWASP Top 10 for Large Language Model ApplicationsOWASP Foundation
- EU AI Act: Regulation on Artificial IntelligenceEuropean Parliament and Council of the EU
Frequently asked questions
How is this different from the AI systems integration checklist?
The AI systems integration checklist covers single-step integrations: a workflow that calls an AI model once and routes the output. The governance concerns there are mainly about data handling, access, and oversight of individual outputs.
Agentic systems plan and act across multiple steps, tools, and possibly sub-agents. The specific failure modes are different: error propagation across steps, unbounded loops, approval gates that were not designed into the architecture, and blast-radius questions that do not arise in single-call integrations. This checklist addresses those specifically.
Can a simple two-step agent skip most of this?
It depends on what the two steps do. A two-step agent that reads a draft and suggests edits poses similar governance requirements to a single-step integration. A two-step agent that reads a customer record and sends an external email carries the same accountability requirements as a fifty-step one; the harm potential is in the action, not the step count.
Work through the authorization table above and assess consequence level. The phases that almost always apply regardless of complexity are phase two (permissions), phase four (logging), and phase eight (accountability).
What makes an agent deployment “high consequence”?
Any of: the agent can send external communications without per-message human review; it can write to records that affect real people's access, finances, or standing; it can trigger financial transactions; it has access to sensitive or regulated data; or its failure would be visible to people outside the organization.
When any of those apply, all five BLOCKING items in phase six should be treated as non-negotiable, and a red-teaming exercise (phase five, ADVISORY) becomes worth the time.
Does passing this checklist mean the agent is safe?
No. It means the common governance failures have been addressed before launch. Whether the agent is safe depends on its specific capabilities, the data it accesses, the actions it can take, and your organization's risk posture, none of which a generic checklist can fully assess.
Treat a completed checklist as evidence that the governance work was done attentively. Use the responsible AI governance pillar for the broader organizational structure that makes ongoing oversight possible.