Operations

This page explains how to run agent-sidecar day to day: where logs go, how to read the audit trail, how to fix common problems, and how to change charters and certificates. It ends with a production checklist and the full list of known limitations.

Commands on this page use the names in the code: the sidecar is attached, the CLI is attache, and the CA is embassy. Architecture explains the naming. The examples use the demo namespace attache-demo and the kubectl context minikube. Change them for your cluster.

Where logs go

The attached container writes three kinds of lines.

Stream Written by Goes to How to recognise a line
Audit records attached stdout when audit.sink: stdout (the default), or a file when audit.sink: file One JSON object per line. It always starts with {"ts":.
Sidecar logs attached stderr A timestamp, a level such as INFO or WARN, then a message. These lines can contain ANSI colour codes.
Runtime logs attache-runtime, forwarded by attached stderr of attached Every line starts with [runtime] .

The runtime is a child process of the sidecar. The sidecar reads the runtime’s stdout and stderr. It writes every line to its own stderr with the prefix [runtime] . So stdout carries audit records only.

Kubernetes keeps stdout and stderr of a container in one log. So kubectl logs -c attached shows all three streams mixed together. attache logs keeps only the lines that parse as audit records.

flowchart LR
  RT["attache-runtime (child process)"] -->|"model_call records"| SC["attached"]
  RT -->|"stdout and stderr lines"| SC
  SC -->|"audit records, sink stdout"| OUT["stdout"]
  SC -->|"audit records, sink file"| FILE["audit file in the container"]
  SC -->|"sidecar logs and runtime lines"| ERR["stderr"]
  OUT --> KL["kubectl logs -c attached"]
  ERR --> KL
  FILE --> EX["kubectl exec ... cat"]

Log level

The sidecar log level comes from the RUST_LOG environment variable. The image and the demo Deployments set RUST_LOG=info. Set RUST_LOG=debug on the attached container for more detail. The runtime always logs at INFO. It has no setting for this.

Audit sink: stdout or file

There are two sinks: stdout and file. stdout is the default. You set the sink in the charter. This snippet is partial: it shows only the audit block of a charter.

audit:
  sink: file
  file: /var/log/attache/audit.jsonl
  redact: []

What you need to know about the file sink:

Local runs

make local-demo runs the sidecars on your machine. It writes these files:

File Content
.local/logs/<svc>-audit.jsonl stdout of the sidecar: audit records only
.local/logs/<svc>-attached.log stderr of the sidecar: sidecar logs and [runtime] lines
.local/logs/<svc>-app.log output of the demo app 
cat .local/logs/orders-audit.jsonl .local/logs/inventory-audit.jsonl | ./target/debug/attache logs --file -

The audit record

Each record is one JSON object on one line. Only attached writes records. The runtime reports its model_call records to the sidecar over the ReportEvent call, and the sidecar writes them. The field order is fixed, so grep and people see the same layout. The full reference is CONTRACT.md section 6.

Fields

Field Type Meaning
ts string When the record was made, normally when the step finished. UTC, RFC 3339 with milliseconds, for example 2026-10-20T13:05:24.710Z.
kind string One of nine kinds. See Record kinds.
request_id string A UUID. One per top-level invocation. It is the same on every hop.
trace_id string W3C trace id, 32 hex characters.
span_id string The span of this hop, 16 hex characters.
parent_span_id string or null The span that called this one. null for a root span.
service string The service of the sidecar that wrote the record.
namespace string The namespace of that service.
agent string or null The agent this record belongs to. On a callee’s inbound service_call, this is the calling agent from the other pod. null when no agent is known.
caller string local (this pod: the app or a local agent), a SPIFFE id such as spiffe://attache/ns/attache-demo/svc/orders, or unknown (the certificate was refused).
caller_agent string or null The agent that made the call, when an agent made it.
callee string What was called. The format depends on the kind.
direction string inbound, outbound or internal.
input JSON or string What went in, after redaction and truncation.
output JSON, string or null What came out, after redaction and truncation. null when the step was denied or failed.
input_sha256 string or null SHA-256 (hex) of the full original input. null when the input is null.
output_sha256 string or null The same for the output.
tokens object or null {"input": n, "output": n} for model calls, agent calls and invocations.
cost_usd number or null Cost from the runtime’s price table. The mock model costs 0.
latency_ms integer How long the step took.
decision string allow or deny.
rule string or null The rule that decided, for example authz.services.deny-admin.
reason string or null A human-readable reason. Mostly set for denials.
error string or null Set when the step was allowed but failed.

Record kinds

kind direction Written when callee example
agent_invocation inbound An agent is invoked: by the app, by a peer sidecar, or by a local agent tool. Exactly one per invocation. It is written last, after the records of its steps. ops-assistant
model_call outbound Each model request in the agent loop. The runtime reports it. The sidecar writes it. anthropic:claude-opus-5-5 or mock
tool_call internal Each tool call from the runtime: allowed, denied or failed. get_stock (the tool name)
service_call internal (own app), outbound (to a peer), inbound (on the callee) A service tool request was sent to the app or to a peer. The callee writes its own inbound record, also for denials. inventory GET /stock/WIDGET-1
agent_call outbound (remote agent), internal (local agent) An agent tool call was sent. inventory/stock-analyst or order-triage
http_call outbound An http tool request was sent. GET https://example.com/items/1
guardrail internal An input or output guardrail denied the invocation, or pii_redact rewrote the output (then decision: allow and rule: guardrail.pii_redact). the agent name
budget internal max_calls_per_minute or max_cost_usd_per_day denied an invocation. the agent name
authz inbound A mesh call was refused because the client certificate has no spiffe://attache/ URI SAN. The rule is authz.identity.invalid and caller is unknown. the agent, or <service> <METHOD> <path>

Some details:

decision, rule, reason and error

Read the four fields together:

decision error Meaning
deny null Policy stopped the step. rule names the policy and reason says why.
allow set Policy let the step through, but it failed. Examples: peer down, app down, model error, unknown agent.
allow null The step was allowed and worked. rule may name the rule that allowed it, for example authz.services.self-stock, or be null.

Rule names come in families:

Rule Comes from
authz.callers.default No authz.callers rule exists for the caller’s identity.
authz.callers.<service> A rule exists for the identity, but it does not list the agent, or its caller_agents does not include the calling agent. For calls from the app the suffix is local, for example authz.callers.local.
authz.services.<name> The named authz.services rule. With deny, one of its deny patterns matched. With allow, one of its allow patterns matched.
authz.default.deny, authz.default.allow authz.default, because no authz.services pattern matched.
authz.identity.invalid A mesh client certificate without a valid SPIFFE URI SAN.
guardrail.<check> max_input_bytes, input_deny.<name>, output_deny.<name>, pii_redact, allowed_tools, allowed_services, tool_spec, path.
budget.<limit> max_calls_per_minute, max_cost_usd_per_day, max_depth.
model.refusal The model refused the request.
agent.unknown The agent is not in the charter. The record has decision: allow and error set.

Some failures name a rule only in the HTTP response: runtime.unavailable, model.error, budget.max_steps and budget.max_tokens_per_call. Their agent_invocation record has decision: allow, rule: null and the reason in error.

Linking records across pods

Two ids join the records of one call, across all pods:

If your gateway sends the same traceparent for many requests, they share one trace_id. Each still has its own request_id. Use --request to separate them.

sequenceDiagram
  participant App as orders app
  participant O as orders sidecar
  participant I as inventory sidecar
  participant IA as inventory app
  App->>O: POST /v1/agents/ops-assistant/invoke with optional traceparent
  Note over O: new request_id R. trace_id T from traceparent, or new.
  O->>I: mesh call with traceparent (child span of T) and x-attache-request-id R
  Note over I: same R and T, new span_id
  I->>IA: GET /stock/WIDGET-1 with traceparent, x-attache-request-id, x-attache-caller, x-attache-agent
  IA-->>I: 200
  I-->>O: agent output
  O-->>App: body has request_id R and trace_id T, headers traceparent and x-attache-request-id

Tips:

Payload redaction and hashes

The sidecar does these steps, in this order, for every record it writes:

  1. Hash. input_sha256 and output_sha256 are computed over the full original payload, before any redaction or truncation. For a string payload, the hash is SHA-256 of its UTF-8 bytes. For any other JSON value, it is SHA-256 of its compact JSON text. A null payload has a null hash. So you can prove later that a payload you hold is the one that passed, without the log storing it.
  2. audit.redact paths. Each path is a dotted path that starts with input. or output., for example output.body.warehouse_contact. The value there becomes "[redacted]". Arrays are walked. The path input or output alone redacts the whole payload. The charter validator does not check these paths. A path with another start does nothing.
  3. Scrub repeated values. The values found at those paths (strings and numbers of 4 or more characters) are remembered for that request_id for 15 minutes. They are replaced with [redacted] wherever they appear as text in later records of the same request in that sidecar, for example inside a model prompt.
  4. PII. With guardrails.pii_redact: true, emails become [redacted:email] and phone numbers become [redacted:phone]. This applies to every string in input and output.
  5. Truncate. A payload larger than 8 KiB is cut to 8 KiB plus …(truncated, N bytes). Truncation comes after redaction, so a cut never shows part of a redacted value.

This snippet is partial: it shows only the audit block of a charter.

audit:
  sink: stdout
  redact:
    - input.context.customer_email     # agent_invocation input, when the app sends a context
    - output.body.warehouse_contact    # service_call output: {"status": 200, "body": {...}}

Each sidecar applies the charter of its own service. A value that the inventory charter redacts can still appear in the orders log if the orders charter does not redact it.

What redaction does not cover:

pii_redact changes the agent’s final output and the audit records. Nothing else.

Example record

This is the denial written by the inventory sidecar in the local demo. It is one line in the log. It is shown here formatted.

{
  "ts": "2026-10-20T13:05:24.710Z",
  "kind": "service_call",
  "request_id": "b7e5832a-ad31-4f78-9215-8d103cc9ed07",
  "trace_id": "e3af314822426057d5ee3179b51461b4",
  "span_id": "2e77fa4388933dfb",
  "parent_span_id": "d4ab404d3c0097b1",
  "service": "inventory",
  "namespace": "attache-demo",
  "agent": "ops-assistant",
  "caller": "spiffe://attache/ns/attache-demo/svc/orders",
  "caller_agent": "ops-assistant",
  "callee": "inventory POST /admin/reset",
  "direction": "inbound",
  "input": { "method": "POST", "path": "/admin/reset", "body": {} },
  "output": null,
  "input_sha256": "004b59a790029158af09d1ac3ccd89684b533043e26499e4e20ecdc1f18826e8",
  "output_sha256": null,
  "tokens": null,
  "cost_usd": null,
  "latency_ms": 0,
  "decision": "deny",
  "rule": "authz.services.deny-admin",
  "reason": "POST /admin/reset on inventory matches deny pattern \"* /admin/*\"",
  "error": null
}

The orders sidecar wrote a tool_call and a service_call record for the same call. They have the same request_id, the same trace_id and the same rule.

You can check the hash. The input is a JSON object, so the hash is over its compact JSON text:

printf '%s' '{"method":"POST","path":"/admin/reset","body":{}}' | shasum -a 256
# 004b59a790029158af09d1ac3ccd89684b533043e26499e4e20ecdc1f18826e8  -

On Linux, use sha256sum instead of shasum -a 256.

Reading logs

With kubectl

# Everything the sidecar wrote: audit records, sidecar logs and runtime lines
kubectl --context minikube -n attache-demo logs -l attache.io/service=inventory -c attached --tail=-1

# Only audit records
kubectl --context minikube -n attache-demo logs -l attache.io/service=inventory -c attached --tail=-1 \
  | grep '^{"ts"'

# Only runtime lines (deploy/<name> picks one pod)
kubectl --context minikube -n attache-demo logs deploy/inventory -c attached | grep '^\[runtime\]'

# The previous run of the container, after a crash
kubectl --context minikube -n attache-demo logs deploy/inventory -c attached --previous

With -l, kubectl prints only the last 10 lines of each container unless you pass --tail. --tail=-1 prints everything.

If you have jq, you can filter the JSON:

kubectl --context minikube -n attache-demo logs -l attache.io/service=inventory -c attached --tail=-1 \
  | grep '^{"ts"' | jq -c 'select(.decision == "deny") | {ts, service, kind, rule, reason}'

With the file sink, read the file inside the container:

kubectl --context minikube -n attache-demo exec deploy/inventory -c attached -- \
  cat /var/log/attache/audit.jsonl | ./target/debug/attache logs --file - --denied

With the attache CLI

Build the CLI once with cargo build -p attache-cli. The binary is ./target/debug/attache. make logs builds it for you if it is missing.

./target/debug/attache logs orders,inventory -n attache-demo --context minikube

What it does:

  1. It runs kubectl [--context C] logs -n NS -l <selector> -c attached --tail N --prefix=false, with -f when you follow. The selector is attache.io/service=<svc> for one service, or attache.io/service in (<svc>,<svc>) for several.
  2. It keeps only lines that parse as audit records. Sidecar logs and [runtime] lines are skipped.
  3. It applies your filters.
  4. It sorts by ts, unless you follow.
  5. It prints one line per record.

Your pods need the label attache.io/service: <svc>, and the sidecar container must be named attached. Otherwise attache logs finds nothing.

Flag Default What it does
<service>[,<service>...] none Services to read, by the label attache.io/service. Required unless you use --file.
-n, --namespace attache-demo The namespace.
--context kubectl’s current context The kubeconfig context.
--agent A none Only records whose agent or caller_agent is exactly A.
--trace T none Only records whose trace_id starts with T.
--request R none Only records whose request_id starts with R.
--kind K none Only records of one kind, for example service_call.
--denied off Only records with decision: deny.
-f, --follow off Follow new records (kubectl logs -f). Records print as they arrive, not sorted.
--raw off Print the JSON lines instead of the table.
--file F none Read records from file F, or from stdin with -, instead of kubectl. The service argument is ignored.
--tail N 1000 How many lines kubectl fetches per container.

Exit status: attache logs exits 1 if kubectl fails. It prints kubectl logs failed (...); no audit records were read. An empty table always means “no matching records”. It never hides a kubectl failure.

The table columns are: time (UTC, HH:MM:SS.mmm), service, agent, kind, direction, caller, → callee, decision, latency. Then, when present: tokens=<in>/<out> $<cost>, error=<text>, and always trace=<first 8 characters>. A SPIFFE caller is shortened to svc/<name>. The calling agent is shown in parentheses, except on an agent_invocation record whose caller is local. A denial shows DENY <rule>. A pii_redact record shows allow guardrail.pii_redact.

This is real output for the denied call of the local demo:

cat .local/logs/orders-audit.jsonl .local/logs/inventory-audit.jsonl \
  | ./target/debug/attache logs --file - --trace e3af3148

13:05:24.709 orders     ops-assistant  model_call       outbound local                        → mock                               allow                                         0ms tokens=101/9 $0.0000 trace=e3af3148
13:05:24.710 inventory  ops-assistant  service_call     inbound  svc/orders (ops-assistant)   → inventory POST /admin/reset        DENY authz.services.deny-admin                0ms trace=e3af3148
13:05:24.711 orders     ops-assistant  tool_call        internal local (ops-assistant)        → inventory_admin_reset              DENY authz.services.deny-admin                0ms trace=e3af3148
13:05:24.711 orders     ops-assistant  service_call     outbound local (ops-assistant)        → inventory POST /admin/reset        DENY authz.services.deny-admin                0ms trace=e3af3148
13:05:24.711 orders     ops-assistant  model_call       outbound local                        → mock                               allow                                         0ms tokens=175/38 $0.0000 trace=e3af3148
13:05:24.711 orders     ops-assistant  agent_invocation inbound  local                        → ops-assistant                      allow                                         3ms tokens=276/47 $0.0000 trace=e3af3148

The invocation itself is allow. The denied tool call did not fail it. The model got the denial as a tool error and answered.

More examples:

# One call across both pods
./target/debug/attache logs orders,inventory --context minikube --trace e3af3148

# One top-level invocation
./target/debug/attache logs orders,inventory --context minikube --request b7e5832a

# Every denial in the last 1000 lines of each pod
./target/debug/attache logs orders,inventory --context minikube --denied

# Follow one agent live
./target/debug/attache logs inventory --context minikube --agent stock-analyst -f

# Only model calls, as raw JSON
./target/debug/attache logs orders --context minikube --kind model_call --raw

Make shortcuts

Target Runs
make logs attache logs orders,inventory -n $(NS) --context $(KUBE_CONTEXT). Add TRACE=<prefix> for --trace and AGENT=<name> for --agent.
make logs-denied The same with --denied.

NS defaults to attache-demo. KUBE_CONTEXT defaults to the value of PROFILE, which is minikube. Both targets read only the two demo services.

make logs TRACE=e3af3148
make logs AGENT=stock-analyst
make logs-denied

Troubleshooting

Start with the response. Note its status, rule, reason and request_id. Then read the audit records of that request in every pod it touched:

./target/debug/attache logs orders,inventory --context minikube --request <request_id>

HTTP errors from an invocation

These come from POST /v1/agents/{name}/invoke.

Symptom Likely cause What to check
400, rule request.invalid The body is not JSON, or it has no input string. Send {"input": "...", "context": {...}}. No audit record is written.
403, rule authz.callers.default The charter has no authz.callers rule for this identity. For calls from the app, add a rule with identity: local. For a peer, add its SPIFFE id.
403, rule authz.callers.<service> A rule for the identity exists, but it does not list this agent, or its caller_agents does not include the calling agent. A misspelled agent name also gives this error when the rule lists agents by name instead of "*", because authz.callers is checked before the agent lookup. agents and caller_agents of that rule. The agent name.
403, rule guardrail.max_input_bytes input plus context is larger than guardrails.max_input_bytes (default 16384). The size of input and context.
403, rule guardrail.input_deny.<name> The named regex matched the input or the context. The regex. The guardrail record shows the input (redacted).
403, rule guardrail.output_deny.<name> The named regex matched the agent’s final output. The regex. The guardrail record shows the output (redacted).
403, rule model.refusal The model refused the request. The last model_call record.
404, rule agent.unknown The agent name is not in the loaded charter, and the caller’s authz.callers rule allows "*". GET /v1/agents on the sidecar (see below). Did you restart the pod after the charter change?
429, rule budget.max_calls_per_minute Too many invocations of this agent in 60 seconds on this replica. Budgets.
429, rule budget.max_cost_usd_per_day The agent’s model cost today (UTC) on this replica reached the limit. cost_usd in the agent’s model_call records.
429, rule budget.max_depth Agent calls are nested more than 4 levels deep. Orchestrators that call each other in a loop.
502, rule runtime.unavailable The sidecar cannot reach the runtime. /readyz shows runtime: false. Look for [runtime] lines and agent runtime exited warnings.
502, rule model.error The model API call failed: wrong key, unknown model id, or no network egress. error in the model_call record.
502, rule budget.max_steps The agent did not finish within max_steps model calls. Raise max_steps, or make the task or tools simpler.
502, rule budget.max_tokens_per_call The model hit max_tokens in the middle of a tool call, or with no text. Raise max_tokens_per_call (at most 128000).

The gRPC app API on 127.0.0.1:50001 maps the same cases to status codes. A denial is PERMISSION_DENIED, or RESOURCE_EXHAUSTED for budget.*. An unknown agent is NOT_FOUND. Everything else is UNAVAILABLE. The rule is in the x-attache-rule metadata.

A tool call was denied or failed, but the invocation returned 200

A denied tool call does not fail the invocation. The model gets an error tool result and goes on. Look at steps[] in the response and at the tool_call records. A denial can come from the caller’s sidecar or from the callee’s sidecar. Read both pods’ logs to see which one denied it.

Rule or text Likely cause What to check
guardrail.allowed_tools The model called a tool that the agent may not use, or did not declare. guardrails.allowed_tools and the agent’s tools. The runtime only gets the allowed tools. If the model names another tool, the runtime refuses it itself: you see the step in steps[] (summary not declared), but no tool_call record.
guardrail.tool_spec The call does not match the tool’s declaration: other service, method, agent or URL origin. The tool entry in the charter.
guardrail.allowed_services The target service is not in guardrails.allowed_services for this agent. That list.
guardrail.path The path does not start with /, or it has whitespace, a control character, \, #, %2f, %5c or %00. The tool input in the tool_call record.
authz.services.<name> with DENY A deny pattern of that rule matched. Which pod wrote the first deny record.
authz.default.deny No allow pattern matched, and authz.default is deny. On the caller, rules for the target service that apply to identity: local and this agent. On the callee, rules for itself that apply to the caller’s SPIFFE id and agent.
tool failed: mesh.unavailable: ... The peer sidecar could not be reached. Mesh and TLS.
tool failed: app.unavailable: ... The target app did not answer on 127.0.0.1:<app_port>. service.app_port in the target’s charter, and the app container.
tool failed: http.error: ... The http tool request failed. Network egress from the pod.

The tool failed: <rule>: ... text is the tool result that the model gets. You find it in the input of the next model_call record. For these cases, the tool_call record has decision: allow and error set. Its error text does not always name the rule. For a service tool it is the gRPC status text, for example The service is currently unavailable: app unreachable at http://127.0.0.1:8000/....

Pod and startup problems

Symptom Likely cause What to check
The pod stays in ContainerCreating. The Secret attache-cert-<svc> or the ConfigMap <svc>-charter does not exist. kubectl describe pod. Run the embassy Job, or make charters.
attached is in CrashLoopBackOff. The log starts with Error: load charter. The charter is invalid. The sidecar exits at startup. kubectl logs --previous -c attached. Run attache charter validate on the file.
The log says tls.crt identity ... does not match charter identity .... service.name or service.namespace in the charter differ from the certificate. The mounted Secret. The Job’s --namespace and --services.
The log says open audit file. audit.sink: file, but the path is not writable. The volume at the file’s directory, and readOnlyRootFilesystem.
The pod is Running, but attached is not ready (for example 1/2). /readyz returns 503. Read the /readyz body (below).
attached restarts every few minutes. The liveness probe runs the same attache-ready check. In the demo Deployments it starts after 20 seconds, runs every 20 seconds, and restarts the container after 6 failures. Fix the readiness cause.

A charter error at startup looks like this:

Error: load charter /etc/attache/charter/charter.yaml

Caused by:
    charter is invalid:
      - audit.sink: file requires audit.file

What attache-ready checks. attache-ready is a small script in the image. It calls http://127.0.0.1:3500/readyz and exits 0 only on HTTP 200. /readyz returns 200 when both are true:

The charter is always loaded while the sidecar runs, because the sidecar exits when it cannot load it. The script uses port 3500. If you change --http-addr, change the probe too.

Read the /readyz body from inside the pod. The image has python3:

kubectl --context minikube -n attache-demo exec deploy/orders -c attached -- python3 -c '
import urllib.request, urllib.error
try:
    print(urllib.request.urlopen("http://127.0.0.1:3500/readyz").read().decode())
except urllib.error.HTTPError as e:
    print(e.code, e.read().decode())'

A sidecar that has no TLS files and no runtime answers:

503 {"ready":false,"runtime":false,"model_provider":"unknown","mesh":false}
Field Meaning
ready true only when mesh and runtime are both true.
mesh false: no TLS files in --tls-dir. The mesh is off and remote calls fail. The sidecar log says no TLS material in ...: mesh disabled, remote calls will fail.
runtime false: the runtime is not running, or did not answer Health within 3 seconds.
model_provider anthropic or mock, as the runtime reports it. unknown until the runtime has answered once.

The same python3 command works for /v1/agents (the agents and the tools each may use) and /v1/charter (the loaded charter as JSON).

Runtime crashes

The sidecar supervises the runtime:

Log lines to look for:

Line Meaning
starting agent runtime The sidecar starts the runtime process.
agent runtime is ready The runtime answered Health. The line shows the provider (anthropic or mock).
agent runtime exited; restarting in ... The runtime stopped. Look at the [runtime] lines just before it.
cannot spawn runtime; retrying in ... The command could not start.
[runtime] report_event failed: ... A model_call record did not reach the sidecar. It is missing from the audit log, and its cost is not counted in the budget.

Mesh and TLS

Symptom Likely cause What to check
A tool fails with mesh.unavailable, and the callee has no record of the call. The connection or the TLS handshake failed before the callee handled the call. The peer is down, the name <svc>.<ns>.svc.cluster.local does not resolve, the Service has no port mesh (50002), or the two certificates come from different CAs. The peer pod is ready. kubectl get svc <svc> shows port 50002. Both Secrets hold the same ca.crt (command below).
The callee writes kind authz, rule authz.identity.invalid, caller unknown. The client certificate is signed by the trusted CA, but has no spiffe://attache/ URI SAN. How that certificate was made. Leaf certificates issued by embassy carry the SAN.
Calls to a service in another namespace fail. Not supported out of the box. Each namespace gets its own CA, because the Job looks up attache-ca in its own namespace. The sidecar also checks the peer certificate for <svc>.<own namespace>.svc.cluster.local. Keep services that call each other in one namespace.
All mesh calls fail after about one year. Leaf certificates expire after 365 days. The expiry date (command in Rotating certificates).
A tool error says mesh disabled: no TLS material loaded. This sidecar started without TLS files. --tls-dir and the Secret mount.

Check that two services share one CA:

for s in orders inventory; do
  kubectl --context minikube -n attache-demo get secret attache-cert-$s \
    -o jsonpath='{.data.ca\.crt}' | shasum -a 256
done

The two hashes must be equal.

Model and provider

Symptom Likely cause What to check
Answers look like Mock answer for: ... or Based on <tool>: .... No ANTHROPIC_API_KEY, so the runtime uses the mock model. An agent with model: mock always uses the mock. model_provider in /readyz. Create the Secret attache-anthropic and restart the pods. make secret-anthropic does both.
502 model.error The API rejected the call or could not be reached. error in the model_call record. The key. The model id. Egress to the API.

Logs

Symptom Likely cause What to check
attache logs prints nothing. Wrong namespace (default attache-demo) or context. The pods lack the label attache.io/service. The sink is file. The records are older than the last --tail lines. The filter is too narrow (--agent is an exact match). Run the same kubectl logs command by hand.
error: kubectl logs failed (...) kubectl could not read the logs: context, permissions, or no pods match. kubectl’s own message, printed above it.
error: usage: attache logs <service> ... No service and no --file. Add the service name.
Lines with odd characters such as [2m or [0m. ANSI colour codes in sidecar log lines. These are not audit records. attache logs skips them.
A model_call record is missing. The runtime could not report it. [runtime] report_event failed lines.

Changing a charter

The sidecar reads the charter once, at startup. Kubernetes updates the files of a mounted ConfigMap in a running pod after a while, but the sidecar does not read the file again. A change takes effect only when the pod restarts.

  1. Edit the charter and validate it.
  2. Update the ConfigMap.
  3. Restart every pod of the service.
  4. Check the loaded charter.
./target/debug/attache charter validate examples/charters/inventory.charter.yaml

kubectl -n attache-demo create configmap inventory-charter \
  --from-file=charter.yaml=examples/charters/inventory.charter.yaml \
  --dry-run=client -o yaml | kubectl --context minikube apply -f -

kubectl --context minikube -n attache-demo rollout restart deployment/inventory
kubectl --context minikube -n attache-demo rollout status deployment/inventory --timeout=300s

For the two demo services, make charters validates both demo charters and updates both ConfigMaps. It does not restart the pods. make deploy updates the ConfigMaps and restarts both Deployments.

Check what a pod runs now:

kubectl --context minikube -n attache-demo exec deploy/inventory -c attached -- \
  python3 -c 'import urllib.request; print(urllib.request.urlopen("http://127.0.0.1:3500/v1/charter").read().decode())'

Things to know:

See Creating agents for what to put in a charter.

Rotating certificates

Certificate rotation is not automatic. These facts matter:

Check when a certificate expires:

kubectl --context minikube -n attache-demo get secret attache-cert-orders \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate

New leaf certificates, same CA

kubectl --context minikube -n attache-demo delete job embassy-bootstrap --ignore-not-found
kubectl --context minikube apply -k deploy/
kubectl --context minikube -n attache-demo wait --for=condition=complete job/embassy-bootstrap --timeout=180s
kubectl --context minikube -n attache-demo logs job/embassy-bootstrap
kubectl --context minikube -n attache-demo rollout restart deployment/orders deployment/inventory

You delete the Job first because a finished Job does not run again. The Job also deletes itself one hour after it finishes (ttlSecondsAfterFinished: 3600). make deploy runs the same steps, except reading the Job log. It also updates the charter ConfigMaps.

Old and new leaf certificates chain to the same CA. So pods keep talking to each other during the rollout.

A new CA

Do this only when you must, for example when the CA key leaked.

kubectl --context minikube -n attache-demo delete secret attache-ca
# then run the same five commands as above

The Job creates a new CA (its log says created new CA) and new leaves. Restart every Deployment in the namespace. Until every pod has restarted, pods with the old CA and pods with the new CA cannot talk over the mesh.

Without Kubernetes

make dev-certs creates a new CA in .local/ca every time it runs (it replaces the old one), and leaf certificates in .local/tls/<svc>. make local-demo issues new leaves on every run and keeps the CA in .local/ca. For one service:

./target/debug/embassy cert-issue --ca .local/ca --namespace attache-demo --service orders --out .local/tls/orders

Budgets on restart and with replicas

Budgets live in the memory of each attached process. They are counted per agent name.

Budget How it is counted
max_calls_per_minute A sliding 60-second window. It counts invocations of the agent on this replica that passed authz.callers, the agent lookup and the depth check. A call that a guardrail denies later still counts. Calls from a local orchestrator’s agent tool count too.
max_cost_usd_per_day The sum of cost_usd of the agent’s model_call records on this replica, per UTC day. It is checked once, at the start of an invocation. A running invocation can go over the limit. The mock model costs 0, so this budget never triggers with the mock.
max_steps Model calls per invocation. The runtime enforces it. No state is kept.
max_tokens_per_call max_tokens of each model call. The runtime enforces it. No state is kept.
Agent-call depth More than 4 nested agent calls are denied with budget.max_depth. Fixed, not configurable.

On restart, all counters go back to zero. This happens on any container restart, pod restart or rollout, including the restart after a charter change.

With replicas, each replica has its own counters. With N replicas, an agent can get up to N times the charter limit in total. Calls from the app go to the sidecar in the same pod. For mesh calls, each sidecar keeps one connection per peer service and reuses it. The Kubernetes Service balances connections, not single calls. So the calls from one pod usually all go to the same replica of the peer, and the load is often not even. To get a total limit, divide it by the number of replicas, and expect some error.

A call to an agent in another pod counts against that agent’s budget in the replica that ran it, not against the caller.

Production checklist (recommendations)

These are recommendations. The repository does not set them up for you, except where a line says “shipped”.

1. Add a NetworkPolicy. No NetworkPolicy is shipped.

An example for the inventory service:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: inventory-mesh-only
  namespace: attache-demo
spec:
  podSelector:
    matchLabels:
      attache.io/service: inventory
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchExpressions:
              - { key: attache.io/service, operator: Exists }
      ports:
        - { port: 50002, protocol: TCP }

2. Set resource requests and limits. Shipped: the attached container requests 50m CPU and 128Mi memory, with a 512Mi memory limit. The app containers have requests only. The sidecar and the Python runtime share the attached container’s limits. If that container is killed for memory, both stop. Measure under real load, then set requests and limits for every container.

3. Harden the securityContext. Shipped on attached only: allowPrivilegeEscalation: false and runAsNonRoot: true. The app containers have none. Consider this for every container:

securityContext:
  runAsNonRoot: true
  runAsUser: 10001
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities: { drop: [ALL] }
  seccompProfile: { type: RuntimeDefault }

With readOnlyRootFilesystem: true, /run/attache is already an emptyDir in the demo Deployments. Mount another one at /var/log/attache if you use the file sink. Test the runtime with these settings before production. The sidecar does not call the Kubernetes API. If your app does not call it either, set automountServiceAccountToken: false on the pod.

4. Ship the audit log. The sidecar writes records. It does not ship them.

5. Manage ANTHROPIC_API_KEY as a secret. Shipped: the Secret attache-anthropic is referenced with optional: true. If it is missing, agents fall back to the mock model without an error.

6. Pin image tags. Shipped: the images are attache/attached:dev, attache/embassy:dev, attache/orders:dev and attache/inventory:dev. They are built locally into minikube’s Docker, with imagePullPolicy: IfNotPresent. No images are published. For production, build in CI, push to your registry, and use a unique tag or a digest for each build. docker/Dockerfile pins its base images by tag (rust:1.99-bookworm, python:3.12-slim-bookworm, ghcr.io/astral-sh/uv:0.10.12, debian:bookworm-slim). Pin them by digest for repeatable builds.

7. Review charters in CI. There is no CI config in the repo. Validate every charter in every pull request. attache charter validate exits 1 and prints the errors:

# for every charter your repository holds
for f in path/to/charters/*.charter.yaml; do
  attache charter validate "$f" || exit 1
done

Require a review for changes to authz and guardrails, for example with a CODEOWNERS file. Keep one committed copy of each charter and create the ConfigMap from it, as make charters does.

8. Alert on denials. There is no metrics endpoint. Build alerts from the audit records in your log store.

Alert on Why
rule: authz.identity.invalid A client presented a certificate without a valid identity.
A rise in decision: deny with authz.callers.* or authz.services.* An agent or a peer tries things it may not do. This can be a prompt injection or a wrong charter.
guardrail.input_deny.* Secrets or banned content arrive in inputs.
budget.max_cost_usd_per_day An agent reached its spending limit.
model_call records with error set The model API has problems.
The sidecar log line agent runtime exited The runtime crashes.
Container restarts and not-ready pods Use your normal Kubernetes monitoring.

9. Other points.

See Microservices for rollout plans for a whole system.

Known limitations

These are true for the current code. “Not built yet” means there is no code for it. CONTRACT.md section 11 has the short list.

  1. Sidecar injection is manual. There is no mutating webhook. The attached container, the charter mount and the TLS mount are written into each Deployment. The attache.io/inject annotation in the demo Deployments has no effect.
  2. Certificates are per service, not per pod. All replicas share one Secret. A one-shot bootstrap Job issues them. There is no rotation. Per-pod certificates and embassy as a running CA server are not built yet.
  3. One CA per namespace. The bootstrap Job looks up the CA Secret attache-ca in its own namespace (the one in --namespace, where it also writes the certificates). So each namespace gets its own CA. Mesh calls across namespaces are not supported out of the box.
  4. Budgets are in memory per replica. They are not shared between replicas, and they reset on restart.
  5. The runtime calls the model API directly. The sidecar does not proxy model calls. It only writes the model_call records the runtime reports. The sidecar cannot block a single model call. If the runtime fails to report a call, the record and its cost are missing.
  6. The runtime runs in the sidecar’s container. It is a child process of attached. It shares the container’s filesystem, environment (including ANTHROPIC_API_KEY), resource limits and network.
  7. pii_redact has a narrow scope. It applies to the agent’s final output and to audit records. It does not apply to prompts and tool results sent to the model provider. With a real model, data such as the demo’s warehouse email reaches the provider. It finds only emails and phone numbers.
  8. Configuration is read once. The charter, the TLS files and ANTHROPIC_API_KEY are read at startup. A ConfigMap or Secret change needs a pod restart.
  9. Refusal fallbacks are not enabled. A model refusal is returned as model.refusal.
  10. No NetworkPolicy is shipped.
  11. No UI. The charter schema (schemas/charter.schema.json) is the API.
  12. No CI config in the repo.
  13. Audit output is basic. There are two sinks (stdout and file). There is no shipping, no file rotation, and write errors are ignored.
  14. No metrics and no trace export. There is no metrics endpoint. Trace ids are propagated, but spans are not exported to a tracing backend. They exist only in the audit records.
  15. The manifests are a demo. The namespace attache-demo and the services orders,inventory are hard-coded in the Job and the kustomization. The images are :dev tags built inside minikube. No images are published.
  16. The readiness probe uses a fixed address. attache-ready always calls 127.0.0.1:3500.

Fixed limits and timeouts in the code:

Limit Value
Audit payload size (input and output, each) 8 KiB, then truncated
Values remembered from audit.redact paths 15 minutes per request, values of 4 or more characters
Agent-call nesting 4 levels (budget.max_depth)
max_tokens_per_call 1 to 128000
guardrails.max_input_bytes default 16384
Sidecar to runtime request timeout 300 seconds
Mesh connect timeout and request timeout 5 seconds and 120 seconds
Sidecar HTTP client (app and http tools) timeout 30 seconds
Runtime restart wait 1 second, doubling up to 30 seconds
Leaf certificate and CA lifetime 365 days and 10 years