An AI agent can propose a Kubernetes change, assemble a Helm values file, and request a deployment before a human has rebuilt the change’s blast radius in their head. That is the promise and the hazard of AI-assisted delivery: the cost of producing a plausible change has fallen faster than the cost of understanding its consequences.
For teams operating on AWS, the tempting response is to give an agent credentials, a terminal, and a carefully worded prompt. It looks productive right up to the moment it confuses ability to invoke an API with authority to make a decision. An agent that can diagnose a broken deployment can also select the wrong cluster, widen an IAM policy, accept untrusted instructions from a ticket, or treat a missing health signal as a reason to try again.
The useful question is not, “Can the agent run kubectl?” It is: does the delivery system make the safe path the easy path?
That is a platform question: scoped intent, constrained identities, versioned changes, policy gates, reconciliation, telemetry, and a clear point at which a human owns an irreversible decision. AWS supplies many of the primitives; the difficult work is composing them into an operating model that remains understandable when the agent is wrong.
Analysis: This is not a case for making delivery slower. It is a case for moving speed into the repeatable, bounded part of the system, while keeping judgment at the consequential boundary.
The anti-pattern: an agent with cluster access
“Give it read access first” is a reasonable starting instinct. “Then let it fix things” is where intent and authority often collapse into one vague permission set.
AWS IAM does not work that way by default. Effective permission is shaped by multiple policy types, and an explicit deny overrides an allow. Permissions boundaries further set a maximum that identity policies cannot exceed. Those mechanics are valuable precisely because they make an identity’s reach a system property rather than a promise in a prompt. AWS IAM policy evaluation
An agent should not need a broad operator role simply because it has a broad question to answer. “Why is checkout timing out?” may justify reading a deployment revision, a bounded log window, and correlated traces. It does not automatically justify changing a production autoscaling policy, reconfiguring an ingress, or reading a secret.
The distinction is between automation and governed autonomy:
- Automation executes a known procedure with bounded inputs and outputs.
- Governed autonomy can choose among approved procedures, but only inside an explicit authority envelope.
That envelope matters because prompt injection is not a theoretical edge case. Repository text, logs, issue descriptions, and retrieved runbooks are all data an agent may consume. They must stay data. A sentence in a log file cannot grant the agent a new capability; a ticket cannot override a production approval rule.
Build a delivery control loop, not an agent demo
GitOps gives the system a crucial separation: it makes desired state inspectable before it is applied. Amazon EKS documentation describes Argo CD as a declarative GitOps delivery mechanism whose application source is the source of truth; it can reconcile the live cluster with that declared state. AWS also advises restricting production syncs and using carefully scoped repository access rather than wildcard permissions. EKS: Continuous Deployment with Argo CD · EKS: Argo CD considerations
That does not make Git a safety proof. A reviewed pull request can still describe an unsafe change. Reconciliation can fail. Reconciliation can succeed while the service degrades. A Git revert cannot reverse a deleted record, an emitted external event, or an incompatible database migration.
It does, however, give an AI-assisted workflow a durable spine: agent-generated changes are proposals, Git holds the declared state, and the reconciler is the actor that converges the runtime toward an accepted revision.
Analysis: The agent belongs at the proposal and evidence-synthesis stages first. It earns more operational freedom through demonstrated behavior, not through a more persuasive prompt.
Four contracts that make autonomy reviewable
A platform becomes governable when its promises become inspectable contracts. Prompts are useful instructions; they are weak controls. The four contracts below should be visible in workflow schemas, IAM roles, policy tests, pull requests, and incident records.
| Contract | Controls | Evidence to retain | Failure mode to design for |
|---|---|---|---|
| Authority | Scoped roles, permission boundaries, environment and tenant constraints, revision-bound approvals | Requester, acting role, policy decision, approved revision | Privilege expansion or approval applied to the wrong change |
| Context | Approved sources, freshness rules, redaction, provenance labels | Source revision, observation time, tool query, withheld sensitive values | Prompt injection, stale evidence, secret leakage |
| Execution | Typed tools, allowlisted resources, budgets, stop conditions, admission checks | Parameters, proposed diff, check results, tool responses | Unbounded mutation, retry loops, destructive rollback |
| Accountability | Correlation IDs, CloudTrail, deployment records, trace links, outcome review | Intent → commit → reconciliation → runtime outcome | An action cannot be reconstructed or learned from |
Authority: the agent’s permissions must be narrower than its vocabulary
An agent can discuss every AWS service in an account without being allowed to touch every service. Give its workflow a role designed for the workflow: specific actions, specific resources, specific conditions, and a maximum permission set enforced through boundaries where appropriate. AWS documents permissions boundaries as a maximum permission envelope, with effective permissions constrained by both the identity policy and the boundary. AWS IAM permissions boundaries
An agent that can propose a policy change should not be able to attach the policy that approves itself. Permission changes, cross-account access, and data-destruction paths deserve a separate workflow and consequential human authority.
Context: evidence is not instruction
Use immutable or versioned inputs where possible: a Git revision, an EKS object version, a time-bounded CloudWatch query, a named runbook revision. Mark retrieved text as untrusted context. Strip secrets before prompts and before durable records.
Analysis: A useful agent response sometimes is: “The evidence is insufficient to act safely.” Treating abstention as a valid result is one of the cheapest ways to reduce confident, invented remediation.
Execution: a typed operation is safer than an open shell
“Propose a resource adjustment for this non-production workload” has a constrained input space. “Keep running commands until latency improves” does not. Prefer tools that accept a deployment name, namespace, change class, and bounded parameters rather than free-form command strings.
EKS documentation itself distinguishes production choices from less consequential environments: for Argo CD, AWS recommends manual sync for production and notes that automatic sync is more common for development and staging. It also warns that pruning can delete resources and should be used with care. EKS: Argo CD considerations
Stop conditions are part of the contract: a policy deny, conflicting evidence, failed reconciliation, missing telemetry, or a repeated failed action should halt automatic mutation and package an escalation instead of triggering a more desperate loop.
An AgentCore Runtime agent should propose a bounded tool call
Amazon Bedrock AgentCore Runtime gives an agent a managed invocation boundary, but it should still return a proposal rather than smuggle an operator shell into a prompt. This intentionally narrow entrypoint only produces a read-only diagnosis request and escalates any production target.
from bedrock_agentcore.runtime import BedrockAgentCoreApp
app = BedrockAgentCoreApp()
@app.entrypoint
def invoke(payload, context):
target = payload["target"]
if target["environment"] == "production":
return {"status": "ESCALATE", "reason": "production requires approval"}
return {
"status": "PROPOSED",
"tool": "eks_diagnose_workload",
"input": {
"cluster": target["cluster"],
"namespace": target["namespace"],
"deployment": target["deployment"],
"max_log_window_minutes": 15,
},
"stop_conditions": ["policy_deny", "missing_telemetry", "ambiguous_target"],
}
The agent can reason about an incident, but its output remains a typed proposal. AgentCore Gateway can expose the approved tool through MCP, and its policy engine can authorize a specific action rather than a wildcard capability. AgentCore Runtime deployment requirements · AgentCore policy scope
Put the production boundary in AgentCore Policy, not the prompt
This Cedar policy pattern permits one Gateway action only for a named preview namespace. The exact gateway ARN and action identifier are account-specific; the point is that a policy decision sees structured input and wins over agent intent.
permit(
principal,
action == AgentCore::Action::"EksDiagnostics___eks_diagnose_workload",
resource == AgentCore::Gateway::"arn:aws:bedrock-agentcore:REGION:ACCOUNT:gateway/platform-tools"
)
when {
context.input.namespace == "checkout-preview" &&
context.input.max_log_window_minutes <= 15
};
forbid(
principal,
action == AgentCore::Action::"EksDiagnostics___eks_diagnose_workload",
resource == AgentCore::Gateway::"arn:aws:bedrock-agentcore:REGION:ACCOUNT:gateway/platform-tools"
)
when { context.input.environment == "production" };
AgentCore policy evaluation is deny-by-default: a request needs a matching permit, while a matching forbid removes access. That is the practical difference between “the model promised not to do it” and “the platform cannot authorize it.” AgentCore guardrail-policy rules
Accountability: explanations are not audit trails
CloudTrail records events in an AWS account and its Event history provides a view of recent management events. That is essential attribution material, but it does not by itself tell an operator whether a deployment was healthy or whether an agent made a sound choice. AWS CloudTrail Event history
Record one operation identifier through the request, agent proposal, pull request, approval, reconcile attempt, deployment revision, and service telemetry. A model narrative can help a human read the record. It must not become the only record.
Correlate an AgentCore invocation with delivery evidence
Pass the same operation identifier into AgentCore Runtime as a session and trace identifier. That makes the agent invocation joinable with a pull request, CloudTrail events, GitOps reconciliation, and the resulting service telemetry.
import boto3
import json
operation_id = "op-2026-10-13-8f31"
client = boto3.client("bedrock-agentcore")
response = client.invoke_agent_runtime(
agentRuntimeArn="arn:aws:bedrock-agentcore:REGION:ACCOUNT:runtime/eks-diagnoser",
runtimeSessionId=operation_id,
traceId=operation_id,
qualifier="DEFAULT",
payload=json.dumps(
{
"operation_id": operation_id,
"target": {"environment": "staging", "cluster": "platform-staging"},
}
),
)
AgentCore Runtime-hosted agents are automatically instrumented with OpenTelemetry. AgentCore Observability sends agent, Gateway, and memory metrics to CloudWatch; use the operation identifier as the correlation key rather than relying on an LLM-generated explanation. AgentCore Observability getting started · AgentCore observability overview
Self-service is a governed interface
Self-service packages the right controls so teams do not need privileged, one-off help for ordinary work.
On EKS, a GitOps path can make declarative sources the deployment interface. AWS describes GitOps sources as Git repositories, Helm registries, or OCI images, with the reconciler applying the declared state to the target cluster and namespace. EKS: Argo CD concepts
For an AI-assisted platform, a golden path should create more than a deployment. It should establish:
- an owner and service identity;
- environment boundaries and promotion rules;
- resource and telemetry expectations;
- secret references rather than secret values;
- a rollback or compensation description;
- an exception path with reason, scope, owner, expiry, and approval.
Ephemeral environments are still environments with cost, data, and identity consequences. Templates should make expiry, cleanup ownership, quotas, and controlled test data part of the default. Otherwise “self-service” becomes a polished route around governance.
Analysis: The success metric is not how many templates exist. It is whether a team can complete a routine, compliant change faster through the path than around it.
Observe the delivery decision, not only the service
AWS provides pieces of an observability fabric. The AWS Distro for OpenTelemetry can send metrics and traces from EKS workloads to options including CloudWatch, Prometheus, and X-Ray. AWS describes ADOT as its supported OpenTelemetry distribution, with the ability to send correlated signals to multiple AWS monitoring services. EKS: ADOT Operator · AWS Distro for OpenTelemetry and X-Ray
But “we have dashboards” is not yet evidence that agentic delivery is controlled. Three observations answer different questions:
Audit tells us who did what. Platform telemetry tells us what happened afterward. Evaluation tests whether the agent chose the right workflow, respected scope, used sufficient evidence, and escalated when it should have. A perfect audit log can describe a harmful action; healthy latency can hide an unauthorized one.
Telemetry gaps must alter behavior. If a workflow promises a post-deployment health observation and the observation is unavailable, the result is unknown. Do not silently translate it into success.
A ninety-day path for a lean team
The first governed workflow should be useful enough to matter and narrow enough to fail safely. A non-production capacity recommendation, a read-only deployment diagnosis, or a templated service bootstrap is a better opening than “autonomously manage production.”
Days 1–30: make proposals attributable
Choose one operation and make it read-only or proposal-only. Define the owner, accepted sources, typed inputs, allowed AWS role, output schema, and stop conditions. Exit when every trial is attributable and out-of-scope requests are rejected rather than improvised.
Days 31–60: connect a change to an outcome
Add a review path, policy checks, a GitOps reconciliation path in a bounded environment, and an operation ID carried into CloudTrail and telemetry. Exercise failures deliberately: denial, drift, failed reconcile, unavailable telemetry, and a malicious instruction embedded in context. Exit when an operator can reconstruct each trial from intent to outcome and each failure triggers its specified stop or escalation.
Days 61–90: expose the golden path, then widen one dimension
Package the workflow as self-service with documentation, ownership, and an exception route. Widen one variable at a time: another operation or another service or another environment. Track correlation coverage, rejected out-of-scope attempts, reconciliation failures, time to escalation, recovery effort, and completion time. If the workflow stays expensive or opaque, improve the contract before widening autonomy.
Analysis: More agent freedom is not the default solution to friction. Often the missing ingredient is a better template, a sharper approval boundary, or evidence that arrives too late.
The platform is the product
AI will make AWS changes easier to produce. It will not make risk ownership disappear. Humans still define acceptable exposure, decide which business semantics are consequential, review exceptions, and approve changes whose effects cannot be safely or automatically unwound.
The lasting advantage will not belong to the team with the most powerful prompt or the agent with the broadest role. It will belong to the team whose platform turns a request into a constrained, inspectable, observable, and recoverable path to production.
The agent is one operator inside that product. The platform is the product.
AWS sources
- AWS IAM permissions boundaries
- Amazon EKS: Continuous Deployment with Argo CD
- Amazon EKS: Argo CD considerations
- Amazon EKS: Argo CD concepts
- AWS CloudTrail: Viewing events
- Amazon EKS: Send metric and trace data with ADOT Operator
- AWS Distro for OpenTelemetry and AWS X-Ray
- Amazon Bedrock AgentCore Runtime
- Amazon Bedrock AgentCore Policy
- Amazon Bedrock AgentCore Observability