Skip to content

Minh Kubernetes Deployment

Overview

Minh runs as a single-replica Kubernetes Deployment in the ai-coworkers namespace. The manifests live in kubernetes/minh/ in the minh repository and include:

Manifest Purpose
deployment.yaml Deployment, ServiceAccount, Role, and RoleBinding
service.yaml ClusterIP Service exposing webhooks (8000), metrics (9090), and coordination MCP (9091)
ingress.yaml Optional Ingress for external webhook access (requires nginx + cert-manager)
external-secrets/externalsecret.yaml ExternalSecret that syncs minh-secrets from AWS Secrets Manager via ESO

The deployment uses a Recreate strategy (not rolling) because Minh must run as exactly one replica for personality consistency.

Architecture

Pod Structure

The pod consists of two init containers and two application containers:

  1. setup-claude-config (init) — copies Claude OAuth credentials from a Kubernetes Secret into a writable emptyDir volume.
  2. wait-for-databases (init) — blocks until Neo4j (7687), PostgreSQL (5432), and Redis (6379) are reachable.
  3. minh (main) — the Minh agent application.
  4. coordination-mcp (sidecar) — a FastMCP server (core.mcp_coordination_server) exposing Minh's coordination surface over SSE on port 9091. Taryn's OrchestratorRunner reaches it cross-pod via the coord-mcp port on the minh Service.

Health Probes

Main container (minh)

Probe Method Behaviour
Startup Checks for /tmp/healthy file Up to ~310 s for initial boot (initialDelaySeconds: 10, failureThreshold: 30, periodSeconds: 10)
Liveness Checks for /tmp/healthy file Restarts the container after ~90 s of failure (failureThreshold: 3, periodSeconds: 30)
Readiness Calls slack_sdk auth_test() Removes pod from Service endpoints if the Slack connection is lost

The startup probe prevents the liveness probe from killing the container during slow boots.

Sidecar container (coordination-mcp)

Probe Method Behaviour
Liveness HTTP GET /healthz on port 9091 Restarts the sidecar after ~90 s of failure (initialDelaySeconds: 30, failureThreshold: 3, periodSeconds: 30)
Readiness HTTP GET /healthz on port 9091 Removes pod from coord-mcp Service endpoint if the DB connection is lost (initialDelaySeconds: 10, failureThreshold: 3, periodSeconds: 15)

The /healthz endpoint returns 200 when the process is alive and the PostgreSQL cursor opens cleanly; 503 otherwise (with a "status": "degraded" body).

Pod Identity & Crash Diagnostics

Environment variables are injected via the Kubernetes Downward API:

  • POD_NAMEmetadata.name
  • POD_NAMESPACEmetadata.namespace
  • NODE_NAMEspec.nodeName

On startup, _log_pod_identity() (in src/mvp_slack_bot.py) logs these values and queries the Kubernetes API for termination details of previous pods with the app=minh label. This provides an audit trail when previous pod logs have been lost.

The /healthz endpoint on the webhook server also returns pod, node, and uptime_seconds fields for operational visibility.

Coordination MCP Sidecar

The coordination-mcp sidecar (#468) runs core.mcp_coordination_server as a FastMCP server using SSE transport. It exposes five tools defined by the agents_shared.coordination schema contract (minh#79 / pearl#90):

Tool Purpose
describe_capabilities() Returns Minh's domain, primitives, and accepted tiers
accept_task(...) Accepts a task from Taryn; layers through IntentReceiver when intent_id is set
report_status(task_id) Returns current status of a previously-accepted task
current_capacity() Reports in-flight count, queue depth, error rate, and whether Minh is accepting new tasks
validate_against_constitution(...) Pre-action self-check against Minh's M1–M5 mandates

The sidecar shares envFrom (secrets and config) with the main container. Three additional environment variables configure the transport:

Variable Value Purpose
COORD_MCP_TRANSPORT sse Selects SSE transport (vs. stdio for local dev)
COORD_MCP_HOST 0.0.0.0 Bind address
COORD_MCP_PORT 9091 Listen port

RBAC

A dedicated ServiceAccount (minh) is bound to a Role that grants read-only access (get, list) to pods and services within the ai-coworkers namespace. This is used by the pod identity startup logic to query previous pod termination state.

Configuration

Secrets & ConfigMaps

The deployment loads environment variables from two sources:

Kind Name Contents
Secret minh-secrets Slack tokens, API keys, database credentials (managed by ESO)
ConfigMap minh-config Non-sensitive configuration values
Secret minh-claude-oauth Claude OAuth credentials.json (optional)

minh-secrets (Secret — managed by External Secrets Operator)

The minh-secrets Secret is managed by the External Secrets Operator (ESO). An ExternalSecret resource (kubernetes/minh/external-secrets/externalsecret.yaml) syncs secret values from AWS Secrets Manager into the cluster every 5 minutes (chisel#85 / taryn#109 P3; cut over 2026-07-21).

How it works:

  • ESO polls the AWS Secrets Manager secret at path smithy/minh every refreshInterval (5 m).
  • dataFrom.extract fans every top-level JSON key in the AWS secret into the target Kubernetes Secret, so adding or rotating a credential is an AWS-side edit — no manifest change required.
  • The creationPolicy: Owner means ESO owns the Secret lifecycle. The previously manual minh-secrets was deleted at cutover so ESO could create and own it.
  • ESO authenticates to AWS via the eso-reader IAM principal, whose activity is auditable in CloudTrail.
  • The ExternalSecret references the fleet-wide ClusterSecretStore named aws-secretsmanager (defined in roland/kubernetes/roland/external-secrets/clustersecretstore.yaml). See roland/docs/infrastructure/external-secrets-operator.md for the full setup.

The following keys are expected in the smithy/minh AWS secret:

Key Purpose
SLACK_BOT_TOKEN Slack bot OAuth token (xoxb-…)
SLACK_APP_TOKEN Slack app-level token for Socket Mode (xapp-…)
ANTHROPIC_API_KEY Anthropic API key for LLM calls
OPENAI_API_KEY Optional OpenAI fallback key
NEO4J_USER Neo4j username
NEO4J_PASSWORD Neo4j password
NEO4J_AUTH Neo4j auth string in username/password format
POSTGRES_USER PostgreSQL username
POSTGRES_PASSWORD PostgreSQL password
POSTGRES_DB PostgreSQL database name (e.g. minh_memory)
GITLAB_API_TOKEN GitLab personal access token for API operations
GITLAB_WEBHOOK_SECRET Shared secret for GitLab webhook signature verification
MINH_WEBHOOK_TOKEN Token for authenticating inbound webhook requests
MINH_INTERNAL_API_TOKEN Bearer token for inter-agent API requests (/api/v1/status, /api/v1/docs/stale, /api/v1/features); activates the internal API when set

A template with placeholder values is maintained in kubernetes/secrets.yaml.example in the minh repository for reference. This template predates the ESO migration — in production, credentials are managed in AWS Secrets Manager, not via secrets.yaml.

Note: The ESO bootstrap Secret eso-reader-creds (containing the IAM credentials that ESO uses to reach AWS Secrets Manager) is provisioned out-of-band and must never be committed. The .gitignore excludes eso-reader-creds.yaml and kubernetes/**/eso-reader-creds.yaml.

minh-config (ConfigMap)

The ConfigMap contains non-sensitive configuration grouped by subsystem. Key groups:

Group Keys (selection) Purpose
Personality AI_NAME, AI_ROLE, AI_TIMEZONE Agent identity
LLM CLAUDE_MODEL, MAX_TOKENS, TEMPERATURE Model selection and sampling
Slack DEFAULT_CHANNEL, ANNOUNCE_STARTUP Channel routing
Git GIT_USER_NAME, GIT_USER_EMAIL Commit identity
Safety AUTONOMOUS_COMMITS, REQUIRE_APPROVAL_FOR_CHANGES Guardrails for autonomous actions
Databases NEO4J_URI, POSTGRES_HOST, POSTGRES_PORT, REDIS_HOST, REDIS_PORT, WEAVIATE_URL Service-discovery endpoints (non-secret)
Webhook / API WEBHOOK_SERVER_ENABLED, WEBHOOK_PORT Webhook server toggle and port
Merge Monitor MINH_MERGE_MONITOR_ENABLED, MINH_MERGE_MONITOR_INTERVAL Polling frequency for new merges
Self-Maintenance MINH_SELF_MAINTENANCE_ENABLED, MINH_SELF_MAINTENANCE_INTERVAL, MINH_SELF_MAINTENANCE_MAX_TURNS, MINH_SELF_MAINTENANCE_PROJECTS Autonomous issue resolution via Agent SDK
Weaviate MINH_SKIP_STARTUP_REINDEX Controls whether Weaviate re-indexes on pod startup
Agent SDK MINH_AGENT_SDK_ENABLED, MINH_AGENT_SDK_AUTH, MINH_AGENT_CONVERSATION_ENABLED, MINH_AGENT_CONVERSATION_MODEL, MINH_AGENT_CONVERSATION_MAX_TURNS, MINH_AGENT_CONVERSATION_TIMEOUT Multi-turn agentic conversation settings
Logging LOG_LEVEL, LOG_FORMAT Log verbosity and format (json for structured logging)

The full ConfigMap is defined in kubernetes/configmap.yaml in the minh repository.

minh-claude-oauth (Secret, optional)

Contains a single key credentials.json — a Claude Code OAuth token file created by claude auth login. The token expires approximately every 8 hours. If the secret is missing, the pod starts normally but the Agent SDK falls back to single-turn API calls. See Claude OAuth token expired in Troubleshooting for refresh instructions.

Resource Limits

Main container (minh)

Resource Request Limit
CPU 500m 2000m
Memory 1Gi 3Gi

Sidecar container (coordination-mcp)

Resource Request Limit
CPU 20m 200m
Memory 128Mi 512Mi

Volumes

Volume Type Mount Path Purpose
workspace PVC (minh-workspace) /app/workspace Persistent working directory
logs emptyDir (5Gi) /app/logs Application logs
tmp emptyDir (1Gi) /tmp Temp files and health-check marker
claude-config emptyDir (100Mi) /home/minh/.claude-config Writable Claude credentials

Security Context

Both containers run as non-root (uid 1000), drop all Linux capabilities, and disallow privilege escalation. The root filesystem is not read-only due to application requirements.

Usage

Deploying

kubectl apply -f kubernetes/minh/external-secrets/externalsecret.yaml
kubectl apply -f kubernetes/minh/deployment.yaml
kubectl apply -f kubernetes/minh/service.yaml
# Optional — requires an ingress controller and cert-manager:
kubectl apply -f kubernetes/minh/ingress.yaml

Note: The ExternalSecret must be applied before the Deployment so that ESO can materialize the minh-secrets Secret. If applied out of order the pod will fail to start with a CreateContainerConfigError referencing the missing Secret.

Verifying Health

# Pod status
kubectl -n ai-coworkers get pods -l app=minh

# Main health endpoint (port-forward)
kubectl -n ai-coworkers port-forward svc/minh 8000:8000
curl http://localhost:8000/healthz
# → {"status": "ok", "pod": "minh-abc123", "node": "worker-2", "uptime_seconds": 3600}

# Coordination MCP health endpoint
kubectl -n ai-coworkers port-forward svc/minh 9091:9091
curl http://localhost:9091/healthz
# → {"status": "ok"}

Viewing Startup Diagnostics

kubectl -n ai-coworkers logs deploy/minh -c minh | grep "Pod identity"
# → Pod identity: name=minh-abc123 namespace=ai-coworkers node=worker-2

Viewing Coordination Sidecar Logs

kubectl -n ai-coworkers logs deploy/minh -c coordination-mcp
# → minh-coordination MCP server starting (sse on 0.0.0.0:9091)

Troubleshooting

ExternalSecret not syncing minh-secrets

If the minh-secrets Secret is missing or stale, the pod will fail to start or may run with outdated credentials. ESO polls AWS Secrets Manager every 5 minutes.

Diagnose:

# Check ExternalSecret sync status
kubectl -n ai-coworkers get externalsecret minh-secrets
# → STATUS should be "SecretSynced"

# Inspect sync events and error details
kubectl -n ai-coworkers describe externalsecret minh-secrets

# Verify the target Secret was created and has expected keys
kubectl -n ai-coworkers get secret minh-secrets -o jsonpath='{.data}' | jq 'keys'

# Check ClusterSecretStore health
kubectl get clustersecretstore aws-secretsmanager

Common causes:

Cause Fix
ClusterSecretStore not found Apply the fleet ClusterSecretStore from roland/kubernetes/roland/external-secrets/clustersecretstore.yaml
eso-reader-creds bootstrap Secret missing Provision the IAM credentials Secret out-of-band (see roland/docs/infrastructure/external-secrets-operator.md)
AWS secret smithy/minh does not exist Create the secret in AWS Secrets Manager with the required keys
IAM permissions error Verify the eso-reader principal has secretsmanager:GetSecretValue on smithy/minh (check CloudTrail for denied requests)
ESO controller not running kubectl get pods -n external-secrets — ensure the operator pod is healthy

Init container wait-for-databases stuck

The wait-for-databases init container loops on nc -z against Neo4j (7687), PostgreSQL (5432), and Redis (6379). If any service is unreachable, the pod stays in Init:1/2.

Diagnose:

# See which database the init container is waiting for
kubectl -n ai-coworkers logs deploy/minh -c wait-for-databases

# Verify the database services exist and have endpoints
kubectl -n ai-coworkers get svc neo4j postgres redis
kubectl -n ai-coworkers get endpoints neo4j postgres redis

Common causes:

Cause Fix
Database pod not running kubectl -n ai-coworkers get pods -l app=neo4j (repeat for postgres, redis) and check for CrashLoopBackOff or Pending
Service missing Apply the database service manifests: kubectl apply -f kubernetes/neo4j/service.yaml
Wrong namespace The init container resolves bare hostnames (neo4j, postgres, redis) via cluster DNS — the database Services must be in the same namespace (ai-coworkers)

Readiness probe failing after Slack token rotation

When a Slack bot or app token is rotated, the pod remains Running but may become NotReady if the application's internal Slack connection check fails. The readiness probe removes the pod from Service endpoints, so webhooks and coordination MCP requests stop routing to it.

Diagnose:

# Check readiness state
kubectl -n ai-coworkers describe pod -l app=minh | grep -A5 "Readiness"

# Check application logs for Slack errors
kubectl -n ai-coworkers logs deploy/minh -c minh | grep -i "slack\|token\|auth"

Fix:

  1. Update the SLACK_BOT_TOKEN and/or SLACK_APP_TOKEN in AWS Secrets Manager under the smithy/minh secret. ESO will sync the change to the in-cluster minh-secrets Secret within 5 minutes.

  2. Restart the pod to pick up the new tokens:

    kubectl -n ai-coworkers rollout restart deployment/minh
    

Claude OAuth token expired

Minh alerts in the EOD Slack channel when the Claude OAuth token is within 2 hours of expiry. If it has already expired, Agent SDK operations fall back to single-turn API calls.

Fix:

# Re-authenticate locally
claude auth login

# Update the Kubernetes secret
kubectl create secret generic minh-claude-oauth \
  --from-file=credentials.json=$HOME/.claude/.credentials.json \
  -n ai-coworkers --dry-run=client -o yaml | kubectl apply -f -

# Restart Minh to pick up the new secret
kubectl rollout restart deployment/minh -n ai-coworkers

The claude-config-sync sidecar polls the mounted secret every 30 seconds and propagates rotations into the writable emptyDir without requiring a pod restart, but the kubelet refresh can take 60–90 seconds before the new secret file appears.

Pod in CrashLoopBackOff

# View recent logs (including previous crash)
kubectl -n ai-coworkers logs deploy/minh -c minh --previous

# Check pod events
kubectl -n ai-coworkers describe pod -l app=minh | tail -20

# Check resource pressure
kubectl -n ai-coworkers top pod -l app=minh

Common causes include out-of-memory kills (check OOMKilled in pod status), missing environment variables (secret or configmap not applied), and Python import errors from image builds missing dependencies.

ImagePullBackOff

The container image is pulled from registry.gitlab.com/the-smithy1/agents/minh. A gitlab-registry imagePullSecret must exist in the ai-coworkers namespace:

kubectl create secret docker-registry gitlab-registry \
  --docker-server=registry.gitlab.com \
  --docker-username=<username> \
  --docker-password=<token> \
  -n ai-coworkers

Ingress

The optional ingress.yaml exposes the webhook endpoint externally. Before applying:

  1. Replace minh.yourdomain.com with the actual domain.
  2. Ensure an nginx Ingress Controller is installed.
  3. Ensure cert-manager is configured with a letsencrypt-prod ClusterIssuer.

The ingress applies rate limiting (10 req/s) and a 10 MB body-size limit for webhook payloads.

Note: The ingress manifest currently references namespace: minh. If the cluster enforces namespace-scoped Ingress objects, update it to ai-coworkers to match the Deployment and Service.

Webhook Method Restriction

Webhook paths (/webhook*) only accept POST requests. All other HTTP methods (GET, HEAD, OPTIONS, etc.) are denied at the ingress level via an nginx server-snippet. This prevents unwanted probes or scanners from hitting the webhook endpoints. The restriction is configured using nginx.ingress.kubernetes.io/server-snippet:

nginx.ingress.kubernetes.io/server-snippet: |
  if ($request_uri ~* ^/webhook) {
    limit_except POST {
      deny all;
    }
  }

Note: This annotation requires the nginx ingress controller to have allow-snippet-annotations enabled. If snippets are disabled, the ingress will be rejected. Check with kubectl get configmap -n ingress-nginx ingress-nginx-controller -o yaml | grep allow-snippet.

TLS Certificate Management

TLS is handled by cert-manager with a letsencrypt-prod ClusterIssuer. The ingress manifest references a TLS secret named minh-tls-cert:

tls:
  - hosts:
      - minh.yourdomain.com
    secretName: minh-tls-cert

Automatic Rotation

cert-manager automatically renews Let's Encrypt certificates when they are within 30 days of their 90-day expiry. No manual intervention is required as long as:

  1. The cert-manager deployment is healthy: kubectl get pods -n cert-manager
  2. The letsencrypt-prod ClusterIssuer exists: kubectl get clusterissuer letsencrypt-prod
  3. DNS for the ingress host resolves to the cluster's ingress controller

Monitoring Certificate Status

# Check certificate status and expiry
kubectl -n ai-coworkers get certificate minh-tls-cert
kubectl -n ai-coworkers describe certificate minh-tls-cert

# Check the underlying CertificateRequest and Order
kubectl -n ai-coworkers get certificaterequest
kubectl -n ai-coworkers get order

# View the TLS secret (expiry is in the certificate itself)
kubectl -n ai-coworkers get secret minh-tls-cert -o jsonpath='{.data.tls\.crt}' \
  | base64 -d | openssl x509 -noout -dates

Renewal Failures

If cert-manager fails to renew, check the Challenge resources for ACME errors:

kubectl -n ai-coworkers get challenge
kubectl -n ai-coworkers describe challenge <name>

Common causes include DNS misconfiguration (the domain does not resolve to the ingress controller), rate limiting by Let's Encrypt (too many requests in a short window), and the ingress controller not routing the /.well-known/acme-challenge/ path required for HTTP-01 validation.

Prerequisites

  • Kubernetes cluster with available PVCs
  • Neo4j, PostgreSQL, and Redis services resolvable within the cluster
  • External Secrets Operator (ESO) installed in the cluster, with the aws-secretsmanager ClusterSecretStore and eso-reader-creds bootstrap Secret provisioned (see roland/docs/infrastructure/external-secrets-operator.md)
  • minh-config ConfigMap created in the ai-coworkers namespace
  • (Optional) nginx Ingress Controller and cert-manager for external webhook access

Upgrade & Rollback

CI/CD Pipeline

Minh uses local-ci.sh (in the agents/tooling repository) instead of GitLab CI for the build/deploy flow. The script is shared across all Glass Umbrella agents and is invoked from within an agent directory:

cd minh

# Full pipeline: lint → test → security → build → push → deploy
../tools/local-ci.sh

# Test only (lint + pytest + bandit) — writes a gate stamp
../tools/local-ci.sh test

# Build + push + deploy (requires a passing gate stamp)
../tools/local-ci.sh build

# Restart pod only (image already pushed)
../tools/local-ci.sh deploy

The pipeline stages are:

Stage Tools Gate
Lint black --check, flake8 Fails on reformatted files
Test pytest (unit tests, excludes integration markers) Fails on any test failure
Security bandit -r src/ Advisory (does not block)
Build docker build (BuildKit, linux/amd64) Requires gate stamp from test
Push docker push to registry.gitlab.com/the-smithy1/agents/minh Tags both :<short-sha> and :latest
Deploy SSH → kubectl apply manifests → kubectl rollout restart Auto-rollback on failure

Upgrade Procedure

  1. Run tests and build from the agent directory:

    ../tools/local-ci.sh test    # lint + test + security
    ../tools/local-ci.sh build   # build + push + deploy
    

    The build command will refuse to run without a passing gate stamp for the current commit.

  2. Deploy applies all Kubernetes manifests (kubernetes/*.yaml and kubernetes/minh/*.yaml) via kubectl apply before restarting the deployment, so changes to the ConfigMap, Service, or Deployment spec take effect in the same cycle.

  3. Smoke test runs automatically after the rollout completes. It verifies the pod is Running with zero restarts and checks the /healthz endpoint on port 8000 if available.

Rollback

Automatic rollback

local-ci.sh records the previous deployment revision before each deploy. If the rollout times out (300 s) or the smoke test fails, it automatically runs:

kubectl rollout undo deployment/minh -n ai-coworkers

Manual rollback

# View rollout history
kubectl -n ai-coworkers rollout history deployment/minh

# Undo to the previous revision
kubectl -n ai-coworkers rollout undo deployment/minh

# Undo to a specific revision
kubectl -n ai-coworkers rollout undo deployment/minh --to-revision=<N>

# Watch the rollback progress
kubectl -n ai-coworkers rollout status deployment/minh

Emergency stop

# Scale to zero (stops the pod without deleting the Deployment)
kubectl -n ai-coworkers scale deployment/minh --replicas=0

# Resume
kubectl -n ai-coworkers scale deployment/minh --replicas=1