How Can You Trust AI Agents With Your Infrastructure?

Product knowledge and local context

Infrastructure expertise lives in the connections between systems.

Engineers learn a platform by reading its documentation, operating it, handling incidents and seeing how it changes over time. The difficult situations we have worked through appeared when a healthy-looking deployment hid a failing dependency, when desired state lived somewhere else, or when another rollout or migration changed the risk of an otherwise routine action.

Product documentation explains how each component is designed to behave. It cannot describe every local exception introduced during an incident, why a controller will undo a manual fix, or which apparently unused resource still carries production traffic. That understanding develops as a team operates its particular environment.

An agent needs access to both. When we think about infrastructure skills, we start with incident reviews, change history, near misses, runbooks, recovery exercises, service ownership, and observed outcomes as well as product documentation. Those sources contain the practical lessons: what matters here, what tends to break, and when the right answer is to stop.

We turn that experience into required evidence, preconditions, constraints, stop conditions, escalation rules, recovery steps and evaluations that replay situations the team has faced.

Several kinds of correct

A technically correct answer can still produce the wrong result.

We have reviewed infrastructure changes that were valid in every narrow technical sense and still wrong for the system around them. Infrastructure advice can be syntactically correct, technically correct, environmentally correct, organisationally correct, and correct for this exact moment. Models are already good at the first two. The remaining three depend on context that is incomplete, scattered, stale, or never written down.

A valid Terraform plan proves that Terraform can produce a set of changes. The team still has to confirm the service owner's intent, the active migration, the account boundary and the time available for recovery. We use the plan as evidence in that decision.

Context from start to finish

Give the agent the path from request to recovery.

On the platforms we operate, context rarely lives in one place. We have had to trace a single change from an issue or alert through Git, CI, Terraform or Helm, Argo CD, admission policy, Kubernetes controllers, AWS resources, and finally the workload and the people depending on it. Every step owns a different part of the truth.

The repository can say one thing, the cluster another, and the cloud control plane a third. During incidents and migrations, the important context may also be in an issue, a temporary override, a runbook note, or the memory of the engineer who made the last safe change.

A repository view can miss drift and live conditions. A cluster view can miss the controller that will reconcile a fix away. An AWS view can miss the GitOps source of truth and create a second owner for the same resource.

Useful context is current and connected. It explains why the work started, where desired state lives, how the change propagates, what else is moving, who owns the decision, which signals prove success and how to get back. The agent should flag any missing piece before it proposes a change.

• Start with the intent: the alert, request, incident or outcome the work is meant to address.
• Locate the source of truth and follow the delivery path through CI, policy, reconciliation and runtime.
• Map upstream and downstream dependencies across identity, network, DNS, compute, storage and data.
• Compare declared configuration with live state, recent changes, active incidents and planned migrations.
• Name the owner, success signals, observation window, stop conditions and recovery path.
• Require the agent to expose missing context and hand the decision back instead of filling the gap.

State and delayed feedback

Infrastructure can report success before the organisation knows the outcome.

Some of the most misleading infrastructure outcomes we have dealt with looked successful at first. An IAM change can break a deployment later when temporary credentials expire. A network rule can look harmless until traffic moves to another Availability Zone. A node-pool change can finish successfully while removing the capacity needed for the next traffic spike.

The API can accept a change straight away while the failure appears later. Rollback often takes longer than the original action, and some changes to data cannot be fully reversed.

We follow the change into the running system and verify the expected behaviour over an agreed observation window. A completed command, merged pull request or green deployment is one checkpoint in that process.

Explore first, execute through controls

Let the agent investigate, then use the normal delivery path.

We have spent years making infrastructure work more repeatable through declarative configuration, version control, automated tests, policy as code, peer review and controlled deployment pipelines. An AI model adds a probabilistic planner to that environment.

The same goal may produce a different plan after a model update, a context change or a slightly different tool response. That variability can be useful during investigation. It is uncomfortable at the execution boundary, where production needs predictable behaviour under known conditions.

The agent can explore and organise evidence. Before execution, its proposal becomes a machine-verifiable diff or plan. Existing controls check scope, policy, tests, ownership and expected effect before anything applies.

Access was designed for people and services

An agent needs a smaller permission boundary than the tools it can call.

When we operate a platform, we work inside an informal web of constraints. We know which account is sensitive, which command needs another reviewer, and when a technically permitted action is still a poor decision. An agent only sees the constraints we make explicit.

A single natural-language request can become a rapid chain of reads, writes, retries and follow-up actions across several systems. If one credential can modify every cluster, query every secret and delete every resource, the practical authority of the system is much larger than the task.

Proposal is the default. Execution is available for action classes the team has already delegated.

• Give the agent a distinct machine identity with short-lived credentials.
• Separate read access, proposal creation and execution permissions.
• Scope authority by account, environment, resource type, action and time window.
• Set action, cost and retry budgets that the model cannot reinterpret.
• Keep immediate revocation and immutable records of attempted actions.

The incident test

Someone still owns every AI-prepared change.

When something breaks, we still have to investigate it, recover it and explain the decision. Ownership cannot become ambiguous because an AI prepared the change. The team needs to know who delegated the action, who reviewed the evidence, who is paged when it fails, and who can revoke the agent's authority.

A useful audit record includes the goal, live evidence, assumptions, proposed plan, policy results, approvals, tool calls, outputs and post-change validation. It should let the incident team reconstruct what happened without relying on a polished explanation generated afterwards.

Controlled delegation

Define the exact task the agent is allowed to perform.

A broad question such as “Do we trust AI with production?” gives us very little to work with. We start with one workflow and name its evidence, authority and blast radius. The history of that workflow tells us when more access is justified. Some actions may always require human approval.

Deleting persistent data, changing organisation-wide identity controls, rotating critical keys or altering a production network boundary should have a much higher bar than restarting a failed stateless workload.

Turn experience into a skill

A useful skill should behave like a tested operating procedure.

The model is one part of the workflow. The skill also carries the team's operating experience and system map: what to observe, where to retrieve it, how the parts depend on one another, which conditions make the action unsafe, when to escalate and how to verify recovery.

Generic benchmarks show how an agent handles a standard task. We also need to know how it behaves in this architecture, with its failure modes and risk tolerance. Our local evaluations come from routine changes, incidents, near misses, recovery exercises and internal runbooks. Some cases require the agent to stop and ask for help.

AWS's aws-bench evaluates agents on reproducible investigation, troubleshooting and infrastructure-creation tasks. Google recommends checking the sequence of actions, including tool selection, invented arguments, plan adherence and consistency. These tests inspect the work that produced the final answer.

• Start with one narrow task that an experienced operator can describe from observation through recovery.
• Capture the required evidence, preconditions, known failure modes, stop conditions, escalation owner and recovery path.
• Document actions that worked elsewhere and remain unsafe in this architecture or at this moment.
• Replay incidents and near misses as evaluations, including cases where the agent must refuse to act.
• Enforce identity, policy, approval and execution boundaries outside the model.
• Feed the verified outcome back into the skill so experience accumulates instead of disappearing after the incident.