Minh Kubernetes Deployment¶
Overview¶
Minh runs as a single-replica Kubernetes Deployment in the ai-coworkers namespace. The manifests live in kubernetes/minh/ in the minh repository and include:
| Manifest | Purpose |
|---|---|
deployment.yaml |
Deployment, ServiceAccount, Role, and RoleBinding |
service.yaml |
ClusterIP Service exposing webhooks (8000), metrics (9090), and coordination MCP (9091) |
ingress.yaml |
Optional Ingress for external webhook access (requires nginx + cert-manager) |
external-secrets/externalsecret.yaml |
ExternalSecret that syncs minh-secrets from AWS Secrets Manager via ESO |
The deployment uses a Recreate strategy (not rolling) because Minh must run as exactly one replica for personality consistency.
Architecture¶
Pod Structure¶
The pod consists of two init containers and two application containers:
setup-claude-config(init) — copies Claude OAuth credentials from a Kubernetes Secret into a writable emptyDir volume.wait-for-databases(init) — blocks until Neo4j (7687), PostgreSQL (5432), and Redis (6379) are reachable.minh(main) — the Minh agent application.coordination-mcp(sidecar) — a FastMCP server (core.mcp_coordination_server) exposing Minh's coordination surface over SSE on port 9091. Taryn's OrchestratorRunner reaches it cross-pod via thecoord-mcpport on the minh Service.
Health Probes¶
Main container (minh)¶
| Probe | Method | Behaviour |
|---|---|---|
| Startup | Checks for /tmp/healthy file |
Up to ~310 s for initial boot (initialDelaySeconds: 10, failureThreshold: 30, periodSeconds: 10) |
| Liveness | Checks for /tmp/healthy file |
Restarts the container after ~90 s of failure (failureThreshold: 3, periodSeconds: 30) |
| Readiness | Calls slack_sdk auth_test() |
Removes pod from Service endpoints if the Slack connection is lost |
The startup probe prevents the liveness probe from killing the container during slow boots.
Sidecar container (coordination-mcp)¶
| Probe | Method | Behaviour |
|---|---|---|
| Liveness | HTTP GET /healthz on port 9091 |
Restarts the sidecar after ~90 s of failure (initialDelaySeconds: 30, failureThreshold: 3, periodSeconds: 30) |
| Readiness | HTTP GET /healthz on port 9091 |
Removes pod from coord-mcp Service endpoint if the DB connection is lost (initialDelaySeconds: 10, failureThreshold: 3, periodSeconds: 15) |
The /healthz endpoint returns 200 when the process is alive and the PostgreSQL cursor opens cleanly; 503 otherwise (with a "status": "degraded" body).
Pod Identity & Crash Diagnostics¶
Environment variables are injected via the Kubernetes Downward API:
POD_NAME—metadata.namePOD_NAMESPACE—metadata.namespaceNODE_NAME—spec.nodeName
On startup, _log_pod_identity() (in src/mvp_slack_bot.py) logs these values and queries the Kubernetes API for termination details of previous pods with the app=minh label. This provides an audit trail when previous pod logs have been lost.
The /healthz endpoint on the webhook server also returns pod, node, and uptime_seconds fields for operational visibility.
Coordination MCP Sidecar¶
The coordination-mcp sidecar (#468) runs core.mcp_coordination_server as a FastMCP server using SSE transport. It exposes five tools defined by the agents_shared.coordination schema contract (minh#79 / pearl#90):
| Tool | Purpose |
|---|---|
describe_capabilities() |
Returns Minh's domain, primitives, and accepted tiers |
accept_task(...) |
Accepts a task from Taryn; layers through IntentReceiver when intent_id is set |
report_status(task_id) |
Returns current status of a previously-accepted task |
current_capacity() |
Reports in-flight count, queue depth, error rate, and whether Minh is accepting new tasks |
validate_against_constitution(...) |
Pre-action self-check against Minh's M1–M5 mandates |
The sidecar shares envFrom (secrets and config) with the main container. Three additional environment variables configure the transport:
| Variable | Value | Purpose |
|---|---|---|
COORD_MCP_TRANSPORT |
sse |
Selects SSE transport (vs. stdio for local dev) |
COORD_MCP_HOST |
0.0.0.0 |
Bind address |
COORD_MCP_PORT |
9091 |
Listen port |
RBAC¶
A dedicated ServiceAccount (minh) is bound to a Role that grants read-only access (get, list) to pods and services within the ai-coworkers namespace. This is used by the pod identity startup logic to query previous pod termination state.
Configuration¶
Secrets & ConfigMaps¶
The deployment loads environment variables from two sources:
| Kind | Name | Contents |
|---|---|---|
Secret |
minh-secrets |
Slack tokens, API keys, database credentials (managed by ESO) |
ConfigMap |
minh-config |
Non-sensitive configuration values |
Secret |
minh-claude-oauth |
Claude OAuth credentials.json (optional) |
minh-secrets (Secret — managed by External Secrets Operator)¶
The minh-secrets Secret is managed by the External Secrets Operator (ESO). An ExternalSecret resource (kubernetes/minh/external-secrets/externalsecret.yaml) syncs secret values from AWS Secrets Manager into the cluster every 5 minutes (chisel#85 / taryn#109 P3; cut over 2026-07-21).
How it works:
- ESO polls the AWS Secrets Manager secret at path
smithy/minheveryrefreshInterval(5 m). dataFrom.extractfans every top-level JSON key in the AWS secret into the target Kubernetes Secret, so adding or rotating a credential is an AWS-side edit — no manifest change required.- The
creationPolicy: Ownermeans ESO owns the Secret lifecycle. The previously manualminh-secretswas deleted at cutover so ESO could create and own it. - ESO authenticates to AWS via the
eso-readerIAM principal, whose activity is auditable in CloudTrail. - The ExternalSecret references the fleet-wide
ClusterSecretStorenamedaws-secretsmanager(defined inroland/kubernetes/roland/external-secrets/clustersecretstore.yaml). Seeroland/docs/infrastructure/external-secrets-operator.mdfor the full setup.
The following keys are expected in the smithy/minh AWS secret:
| Key | Purpose |
|---|---|
SLACK_BOT_TOKEN |
Slack bot OAuth token (xoxb-…) |
SLACK_APP_TOKEN |
Slack app-level token for Socket Mode (xapp-…) |
ANTHROPIC_API_KEY |
Anthropic API key for LLM calls |
OPENAI_API_KEY |
Optional OpenAI fallback key |
NEO4J_USER |
Neo4j username |
NEO4J_PASSWORD |
Neo4j password |
NEO4J_AUTH |
Neo4j auth string in username/password format |
POSTGRES_USER |
PostgreSQL username |
POSTGRES_PASSWORD |
PostgreSQL password |
POSTGRES_DB |
PostgreSQL database name (e.g. minh_memory) |
GITLAB_API_TOKEN |
GitLab personal access token for API operations |
GITLAB_WEBHOOK_SECRET |
Shared secret for GitLab webhook signature verification |
MINH_WEBHOOK_TOKEN |
Token for authenticating inbound webhook requests |
MINH_INTERNAL_API_TOKEN |
Bearer token for inter-agent API requests (/api/v1/status, /api/v1/docs/stale, /api/v1/features); activates the internal API when set |
A template with placeholder values is maintained in kubernetes/secrets.yaml.example in the minh repository for reference. This template predates the ESO migration — in production, credentials are managed in AWS Secrets Manager, not via secrets.yaml.
Note: The ESO bootstrap Secret
eso-reader-creds(containing the IAM credentials that ESO uses to reach AWS Secrets Manager) is provisioned out-of-band and must never be committed. The.gitignoreexcludeseso-reader-creds.yamlandkubernetes/**/eso-reader-creds.yaml.
minh-config (ConfigMap)¶
The ConfigMap contains non-sensitive configuration grouped by subsystem. Key groups:
| Group | Keys (selection) | Purpose |
|---|---|---|
| Personality | AI_NAME, AI_ROLE, AI_TIMEZONE |
Agent identity |
| LLM | CLAUDE_MODEL, MAX_TOKENS, TEMPERATURE |
Model selection and sampling |
| Slack | DEFAULT_CHANNEL, ANNOUNCE_STARTUP |
Channel routing |
| Git | GIT_USER_NAME, GIT_USER_EMAIL |
Commit identity |
| Safety | AUTONOMOUS_COMMITS, REQUIRE_APPROVAL_FOR_CHANGES |
Guardrails for autonomous actions |
| Databases | NEO4J_URI, POSTGRES_HOST, POSTGRES_PORT, REDIS_HOST, REDIS_PORT, WEAVIATE_URL |
Service-discovery endpoints (non-secret) |
| Webhook / API | WEBHOOK_SERVER_ENABLED, WEBHOOK_PORT |
Webhook server toggle and port |
| Merge Monitor | MINH_MERGE_MONITOR_ENABLED, MINH_MERGE_MONITOR_INTERVAL |
Polling frequency for new merges |
| Self-Maintenance | MINH_SELF_MAINTENANCE_ENABLED, MINH_SELF_MAINTENANCE_INTERVAL, MINH_SELF_MAINTENANCE_MAX_TURNS, MINH_SELF_MAINTENANCE_PROJECTS |
Autonomous issue resolution via Agent SDK |
| Weaviate | MINH_SKIP_STARTUP_REINDEX |
Controls whether Weaviate re-indexes on pod startup |
| Agent SDK | MINH_AGENT_SDK_ENABLED, MINH_AGENT_SDK_AUTH, MINH_AGENT_CONVERSATION_ENABLED, MINH_AGENT_CONVERSATION_MODEL, MINH_AGENT_CONVERSATION_MAX_TURNS, MINH_AGENT_CONVERSATION_TIMEOUT |
Multi-turn agentic conversation settings |
| Logging | LOG_LEVEL, LOG_FORMAT |
Log verbosity and format (json for structured logging) |
The full ConfigMap is defined in kubernetes/configmap.yaml in the minh repository.
minh-claude-oauth (Secret, optional)¶
Contains a single key credentials.json — a Claude Code OAuth token file created by claude auth login. The token expires approximately every 8 hours. If the secret is missing, the pod starts normally but the Agent SDK falls back to single-turn API calls. See Claude OAuth token expired in Troubleshooting for refresh instructions.
Resource Limits¶
Main container (minh)¶
| Resource | Request | Limit |
|---|---|---|
| CPU | 500m | 2000m |
| Memory | 1Gi | 3Gi |
Sidecar container (coordination-mcp)¶
| Resource | Request | Limit |
|---|---|---|
| CPU | 20m | 200m |
| Memory | 128Mi | 512Mi |
Volumes¶
| Volume | Type | Mount Path | Purpose |
|---|---|---|---|
workspace |
PVC (minh-workspace) |
/app/workspace |
Persistent working directory |
logs |
emptyDir (5Gi) | /app/logs |
Application logs |
tmp |
emptyDir (1Gi) | /tmp |
Temp files and health-check marker |
claude-config |
emptyDir (100Mi) | /home/minh/.claude-config |
Writable Claude credentials |
Security Context¶
Both containers run as non-root (uid 1000), drop all Linux capabilities, and disallow privilege escalation. The root filesystem is not read-only due to application requirements.
Usage¶
Deploying¶
kubectl apply -f kubernetes/minh/external-secrets/externalsecret.yaml
kubectl apply -f kubernetes/minh/deployment.yaml
kubectl apply -f kubernetes/minh/service.yaml
# Optional — requires an ingress controller and cert-manager:
kubectl apply -f kubernetes/minh/ingress.yaml
Note: The ExternalSecret must be applied before the Deployment so that ESO can materialize the
minh-secretsSecret. If applied out of order the pod will fail to start with aCreateContainerConfigErrorreferencing the missing Secret.
Verifying Health¶
# Pod status
kubectl -n ai-coworkers get pods -l app=minh
# Main health endpoint (port-forward)
kubectl -n ai-coworkers port-forward svc/minh 8000:8000
curl http://localhost:8000/healthz
# → {"status": "ok", "pod": "minh-abc123", "node": "worker-2", "uptime_seconds": 3600}
# Coordination MCP health endpoint
kubectl -n ai-coworkers port-forward svc/minh 9091:9091
curl http://localhost:9091/healthz
# → {"status": "ok"}
Viewing Startup Diagnostics¶
kubectl -n ai-coworkers logs deploy/minh -c minh | grep "Pod identity"
# → Pod identity: name=minh-abc123 namespace=ai-coworkers node=worker-2
Viewing Coordination Sidecar Logs¶
kubectl -n ai-coworkers logs deploy/minh -c coordination-mcp
# → minh-coordination MCP server starting (sse on 0.0.0.0:9091)
Troubleshooting¶
ExternalSecret not syncing minh-secrets¶
If the minh-secrets Secret is missing or stale, the pod will fail to start or may run with outdated credentials. ESO polls AWS Secrets Manager every 5 minutes.
Diagnose:
# Check ExternalSecret sync status
kubectl -n ai-coworkers get externalsecret minh-secrets
# → STATUS should be "SecretSynced"
# Inspect sync events and error details
kubectl -n ai-coworkers describe externalsecret minh-secrets
# Verify the target Secret was created and has expected keys
kubectl -n ai-coworkers get secret minh-secrets -o jsonpath='{.data}' | jq 'keys'
# Check ClusterSecretStore health
kubectl get clustersecretstore aws-secretsmanager
Common causes:
| Cause | Fix |
|---|---|
| ClusterSecretStore not found | Apply the fleet ClusterSecretStore from roland/kubernetes/roland/external-secrets/clustersecretstore.yaml |
eso-reader-creds bootstrap Secret missing |
Provision the IAM credentials Secret out-of-band (see roland/docs/infrastructure/external-secrets-operator.md) |
AWS secret smithy/minh does not exist |
Create the secret in AWS Secrets Manager with the required keys |
| IAM permissions error | Verify the eso-reader principal has secretsmanager:GetSecretValue on smithy/minh (check CloudTrail for denied requests) |
| ESO controller not running | kubectl get pods -n external-secrets — ensure the operator pod is healthy |
Init container wait-for-databases stuck¶
The wait-for-databases init container loops on nc -z against Neo4j (7687), PostgreSQL (5432), and Redis (6379). If any service is unreachable, the pod stays in Init:1/2.
Diagnose:
# See which database the init container is waiting for
kubectl -n ai-coworkers logs deploy/minh -c wait-for-databases
# Verify the database services exist and have endpoints
kubectl -n ai-coworkers get svc neo4j postgres redis
kubectl -n ai-coworkers get endpoints neo4j postgres redis
Common causes:
| Cause | Fix |
|---|---|
| Database pod not running | kubectl -n ai-coworkers get pods -l app=neo4j (repeat for postgres, redis) and check for CrashLoopBackOff or Pending |
| Service missing | Apply the database service manifests: kubectl apply -f kubernetes/neo4j/service.yaml |
| Wrong namespace | The init container resolves bare hostnames (neo4j, postgres, redis) via cluster DNS — the database Services must be in the same namespace (ai-coworkers) |
Readiness probe failing after Slack token rotation¶
When a Slack bot or app token is rotated, the pod remains Running but may become NotReady if the application's internal Slack connection check fails. The readiness probe removes the pod from Service endpoints, so webhooks and coordination MCP requests stop routing to it.
Diagnose:
# Check readiness state
kubectl -n ai-coworkers describe pod -l app=minh | grep -A5 "Readiness"
# Check application logs for Slack errors
kubectl -n ai-coworkers logs deploy/minh -c minh | grep -i "slack\|token\|auth"
Fix:
-
Update the
SLACK_BOT_TOKENand/orSLACK_APP_TOKENin AWS Secrets Manager under thesmithy/minhsecret. ESO will sync the change to the in-clusterminh-secretsSecret within 5 minutes. -
Restart the pod to pick up the new tokens:
Claude OAuth token expired¶
Minh alerts in the EOD Slack channel when the Claude OAuth token is within 2 hours of expiry. If it has already expired, Agent SDK operations fall back to single-turn API calls.
Fix:
# Re-authenticate locally
claude auth login
# Update the Kubernetes secret
kubectl create secret generic minh-claude-oauth \
--from-file=credentials.json=$HOME/.claude/.credentials.json \
-n ai-coworkers --dry-run=client -o yaml | kubectl apply -f -
# Restart Minh to pick up the new secret
kubectl rollout restart deployment/minh -n ai-coworkers
The claude-config-sync sidecar polls the mounted secret every 30 seconds and propagates rotations into the writable emptyDir without requiring a pod restart, but the kubelet refresh can take 60–90 seconds before the new secret file appears.
Pod in CrashLoopBackOff¶
# View recent logs (including previous crash)
kubectl -n ai-coworkers logs deploy/minh -c minh --previous
# Check pod events
kubectl -n ai-coworkers describe pod -l app=minh | tail -20
# Check resource pressure
kubectl -n ai-coworkers top pod -l app=minh
Common causes include out-of-memory kills (check OOMKilled in pod status), missing environment variables (secret or configmap not applied), and Python import errors from image builds missing dependencies.
ImagePullBackOff¶
The container image is pulled from registry.gitlab.com/the-smithy1/agents/minh. A gitlab-registry imagePullSecret must exist in the ai-coworkers namespace:
kubectl create secret docker-registry gitlab-registry \
--docker-server=registry.gitlab.com \
--docker-username=<username> \
--docker-password=<token> \
-n ai-coworkers
Ingress¶
The optional ingress.yaml exposes the webhook endpoint externally. Before applying:
- Replace
minh.yourdomain.comwith the actual domain. - Ensure an nginx Ingress Controller is installed.
- Ensure cert-manager is configured with a
letsencrypt-prodClusterIssuer.
The ingress applies rate limiting (10 req/s) and a 10 MB body-size limit for webhook payloads.
Note: The ingress manifest currently references
namespace: minh. If the cluster enforces namespace-scoped Ingress objects, update it toai-coworkersto match the Deployment and Service.
Webhook Method Restriction¶
Webhook paths (/webhook*) only accept POST requests. All other HTTP methods (GET, HEAD, OPTIONS, etc.) are denied at the ingress level via an nginx server-snippet. This prevents unwanted probes or scanners from hitting the webhook endpoints. The restriction is configured using nginx.ingress.kubernetes.io/server-snippet:
nginx.ingress.kubernetes.io/server-snippet: |
if ($request_uri ~* ^/webhook) {
limit_except POST {
deny all;
}
}
Note: This annotation requires the nginx ingress controller to have
allow-snippet-annotationsenabled. If snippets are disabled, the ingress will be rejected. Check withkubectl get configmap -n ingress-nginx ingress-nginx-controller -o yaml | grep allow-snippet.
TLS Certificate Management¶
TLS is handled by cert-manager with a letsencrypt-prod ClusterIssuer. The ingress manifest references a TLS secret named minh-tls-cert:
Automatic Rotation¶
cert-manager automatically renews Let's Encrypt certificates when they are within 30 days of their 90-day expiry. No manual intervention is required as long as:
- The cert-manager deployment is healthy:
kubectl get pods -n cert-manager - The
letsencrypt-prodClusterIssuer exists:kubectl get clusterissuer letsencrypt-prod - DNS for the ingress host resolves to the cluster's ingress controller
Monitoring Certificate Status¶
# Check certificate status and expiry
kubectl -n ai-coworkers get certificate minh-tls-cert
kubectl -n ai-coworkers describe certificate minh-tls-cert
# Check the underlying CertificateRequest and Order
kubectl -n ai-coworkers get certificaterequest
kubectl -n ai-coworkers get order
# View the TLS secret (expiry is in the certificate itself)
kubectl -n ai-coworkers get secret minh-tls-cert -o jsonpath='{.data.tls\.crt}' \
| base64 -d | openssl x509 -noout -dates
Renewal Failures¶
If cert-manager fails to renew, check the Challenge resources for ACME errors:
Common causes include DNS misconfiguration (the domain does not resolve to the ingress controller), rate limiting by Let's Encrypt (too many requests in a short window), and the ingress controller not routing the /.well-known/acme-challenge/ path required for HTTP-01 validation.
Prerequisites¶
- Kubernetes cluster with available PVCs
- Neo4j, PostgreSQL, and Redis services resolvable within the cluster
- External Secrets Operator (ESO) installed in the cluster, with the
aws-secretsmanagerClusterSecretStore andeso-reader-credsbootstrap Secret provisioned (seeroland/docs/infrastructure/external-secrets-operator.md) minh-configConfigMap created in theai-coworkersnamespace- (Optional) nginx Ingress Controller and cert-manager for external webhook access
Upgrade & Rollback¶
CI/CD Pipeline¶
Minh uses local-ci.sh (in the agents/tooling repository) instead of GitLab CI for the build/deploy flow. The script is shared across all Glass Umbrella agents and is invoked from within an agent directory:
cd minh
# Full pipeline: lint → test → security → build → push → deploy
../tools/local-ci.sh
# Test only (lint + pytest + bandit) — writes a gate stamp
../tools/local-ci.sh test
# Build + push + deploy (requires a passing gate stamp)
../tools/local-ci.sh build
# Restart pod only (image already pushed)
../tools/local-ci.sh deploy
The pipeline stages are:
| Stage | Tools | Gate |
|---|---|---|
| Lint | black --check, flake8 |
Fails on reformatted files |
| Test | pytest (unit tests, excludes integration markers) |
Fails on any test failure |
| Security | bandit -r src/ |
Advisory (does not block) |
| Build | docker build (BuildKit, linux/amd64) |
Requires gate stamp from test |
| Push | docker push to registry.gitlab.com/the-smithy1/agents/minh |
Tags both :<short-sha> and :latest |
| Deploy | SSH → kubectl apply manifests → kubectl rollout restart |
Auto-rollback on failure |
Upgrade Procedure¶
-
Run tests and build from the agent directory:
../tools/local-ci.sh test # lint + test + security ../tools/local-ci.sh build # build + push + deployThe
buildcommand will refuse to run without a passing gate stamp for the current commit. -
Deploy applies all Kubernetes manifests (
kubernetes/*.yamlandkubernetes/minh/*.yaml) viakubectl applybefore restarting the deployment, so changes to the ConfigMap, Service, or Deployment spec take effect in the same cycle. -
Smoke test runs automatically after the rollout completes. It verifies the pod is Running with zero restarts and checks the
/healthzendpoint on port 8000 if available.
Rollback¶
Automatic rollback¶
local-ci.sh records the previous deployment revision before each deploy. If the rollout times out (300 s) or the smoke test fails, it automatically runs:
Manual rollback¶
# View rollout history
kubectl -n ai-coworkers rollout history deployment/minh
# Undo to the previous revision
kubectl -n ai-coworkers rollout undo deployment/minh
# Undo to a specific revision
kubectl -n ai-coworkers rollout undo deployment/minh --to-revision=<N>
# Watch the rollback progress
kubectl -n ai-coworkers rollout status deployment/minh