Minh Self-Maintenance¶

Overview¶

Minh Self-Maintenance is an autonomous issue resolution loop that enables Minh to fix issues in its own codebase. The service fetches qualifying GitLab issues (labelled bug, enhancement, mcp-extension, stale-docs, or broken-links), spawns a Claude Agent SDK subprocess for each, and reports results back to Slack. It can be triggered on-demand via Slack conversation or programmatically after events such as Weaviate re-indexing on pod startup.

Optionally, a second reviewer agent can evaluate the resulting merge request and auto-merge it if the change passes review criteria and CI.

The feature is disabled by default and gated behind multiple safety layers to prevent runaway execution.

Architecture¶

The self-maintenance system is composed of five main components, with the service layer decomposed into three focused sub-components:

`SelfMaintenanceService` (`src/services/self_maintenance_service.py`)¶

The core orchestrator. Responsible for:

Gate checks — verifying the feature is enabled, the daily run limit has not been reached, and a Redis distributed lock can be acquired.
Issue fetching — querying GitLab for open issues with qualifying labels, deduplicating, filtering out excluded labels and overly complex issues (description > 3,000 chars), and sorting by IID ascending (oldest first). Delegated to IssueEvaluator.
Issue processing — spawning one Claude Agent SDK session per issue with a structured system prompt, budget, and turn cap. The agent reads the issue, plans a fix, implements it, runs tests, and opens a merge request. Workdir preparation and agent option building are delegated to AgentExecutor.
Result parsing — extracting structured output from the agent's ResultMessage.
Review & auto-merge (optional) — if enabled, a test coverage gate, CI check, and reviewer agent evaluate the MR diff; the MR is merged automatically when approved or closed when rejected. Leaf review operations are delegated to ReviewGate, which wraps shared helpers in utils/mr_review_utils.py.
Slack reporting — posting per-issue results (including review verdicts when auto-merge is active) and a run summary to the configured Slack channel, optionally threaded.
Multi-project support — the service iterates over one or more ProjectConfig entries (parsed from MINH_SELF_MAINTENANCE_PROJECTS or falling back to a single project from MINH_GITLAB_PROJECT), applying a shared issue cap and budget across all projects.

Shared utilities (build_agent_env, build_stderr_callback, resolve_cli_path) are imported from core.agent_sdk_config and used by both the fix agent and the reviewer agent to configure authentication, stderr capture, and CLI resolution.

Sub-components¶

Component	Module	Responsibility
`IssueEvaluator`	`src/services/issue_evaluator.py`	Fetches qualifying issues from GitLab, deduplicates, filters excluded labels and overly complex descriptions, and excludes issues that already have an open self-maintenance MR
`AgentExecutor`	`src/services/agent_executor.py`	Prepares isolated working directories (shallow clones), builds `ClaudeAgentOptions` with the MCP server config and stderr callback, and cleans up workdirs after use
`ReviewGate`	`src/services/review_gate.py`	Leaf operations for MR review: diff retrieval, CI polling, merge/close, review comment posting, and a static test-coverage check

SelfMaintenanceService delegates to these sub-components via thin wrapper methods, allowing tests to mock individual steps on the service instance while keeping the orchestration logic (_process_issue, _review_and_merge_mr) on the service itself.

`ActionHandler` integration (`src/slack/action_handler.py`)¶

The action_self_maintain intent is routed through the standard action handler. The flow is:

User asks Minh to run self-maintenance (e.g., "work on your backlog").
IntentDetector classifies the message as action_self_maintain.
ActionHandler._handle_self_maintain() previews qualifying issues and stores a pending action.
User confirms with "yes".
ActionHandler._execute_self_maintain() launches the run as a background asyncio task, streaming results into the Slack thread.

MCP Maintenance Tools Server (`src/core/mcp_maintenance_tools_server.py`)¶

A standalone FastMCP stdio server launched as a subprocess alongside the Claude Agent SDK session. It exposes tools the agent subprocess cannot reach on its own:

Tool	Description
`search_docs`	Semantic search against Minh's Weaviate documentation index. Accepts optional `category`, `product`, and `content_type` filters and `limit` (default 5).
`check_pod_health`	Hit the local health endpoint to verify pod status
`get_issue_detail`	Fetch full GitLab issue detail including comments
`close_issue`	Close a GitLab issue with an optional comment

MCP stdio safety — Because the MCP protocol communicates over stdin/stdout using JSON-RPC, any library writing to stdout (e.g., structlog's default ConsoleRenderer) would corrupt the protocol and crash the Claude CLI with exit-code-1. The server redirects all Python-level logging and structlog output to stderr before importing anything that configures loggers. This pattern is also applied to the conversation tools server (mcp_conversation_tools_server.py) and doc tools server (mcp_doc_tools_server.py).

Shared MR Review Utilities (`src/utils/mr_review_utils.py`)¶

A shared utility module providing the review and auto-merge infrastructure used by both SelfMaintenanceService (code fixes on the minh repo) and DocUpdateGenerator (documentation updates on knowledge-sanctuary). It contains:

Export	Description
`ReviewVerdict`	Dataclass with `approved` (bool), `reason` (str), and `merged` (bool) fields
`parse_review_verdict(text)`	Extracts a `ReviewVerdict` from reviewer agent output — checks for fenced `json` blocks first, then falls back to bare JSON containing an `"approved"` key
`get_mr_diff(mr_url, *, repo, workdir)`	Retrieves an MR diff via `glab mr diff`. Accepts either a `repo` project path (for `-R` flag) or a `workdir` (to infer project from git remote)
`wait_for_ci(mr_url, *, repo, workdir, poll_interval, poll_timeout)`	Polls CI pipeline status via `glab mr view --output json` until success, failure, or timeout
`merge_mr(mr_url, *, repo, workdir)`	Squash-merges an MR with source branch removal via `glab mr merge`
`close_mr(mr_url, *, repo, workdir)`	Closes a rejected MR via `glab mr close` so the issue becomes eligible for retry on the next maintenance cycle
`post_review_comment(mr_url, reason, *, repo, workdir)`	Posts a review comment on a rejected MR via `glab mr note`

Default constants exported by the module:

Constant	Value	Description
`DEFAULT_MAX_DIFF_LINES`	`500`	Maximum diff lines eligible for auto-merge
`DEFAULT_CI_POLL_INTERVAL`	`30`	Seconds between CI status polls
`DEFAULT_CI_POLL_TIMEOUT`	`600`	Maximum seconds to wait for CI

SelfMaintenanceService delegates to ReviewGate, which wraps these helpers in instance methods (get_mr_diff, wait_for_ci, merge_mr, close_mr, post_review_comment) that pass the isolated working directory.

Reviewer Agent (auto-merge)¶

When auto-merge is enabled (MINH_SELF_MAINTENANCE_AUTO_MERGE=true), a second Claude agent evaluates each successful MR before it can be merged. This agent:

Runs with read-only tools (Read, Glob, Grep) — it cannot edit files or create commits.
Uses claude-sonnet-4-20250514 (lighter model than the fix agent's claude-opus-4-6).
Is capped at 10 agentic turns.
Captures Claude CLI stderr via build_stderr_callback("reviewer") for diagnostics.
Outputs a JSON verdict with approved (boolean) and reason fields.

The review flow is:

Diff size check — if the MR diff exceeds MINH_SELF_MAINTENANCE_MAX_DIFF_LINES (default 500), the MR is left for human review.
CI check — polls the MR's pipeline status until it passes, fails, or times out.
Test coverage gate — for non-documentation MRs, checks whether the diff touches src/ files and, if so, requires corresponding changes in tests/. MRs that change source code without test coverage are rejected, commented, and closed.
Reviewer agent — evaluates the diff against the issue description using six review criteria: issue alignment, test coverage, minimal scope, no unrelated changes, no security concerns, and code quality. For stale-docs issues, the test criterion is replaced with a documentation-accuracy criterion.
Act on verdict — if approved, the MR is merged (squash merge with source branch removal). If rejected, a review comment is posted and the MR is closed so the issue becomes eligible for retry on the next maintenance cycle. If the reviewer output is unparseable, the MR is left open for human review with an explanatory comment.

All review steps use the shared helpers from utils/mr_review_utils.py via ReviewGate.

Agent SDK Resilience¶

Stderr Capture (`build_stderr_callback`)¶

The Claude Agent SDK only captures subprocess stderr when ClaudeAgentOptions.stderr is set. Without it, exit-code-1 crashes produce only a placeholder message ("Check stderr output for details"). The build_stderr_callback(label) helper in core.agent_sdk_config returns a callback that logs each stderr line under a labelled logger:

from core.agent_sdk_config import build_stderr_callback

options = ClaudeAgentOptions(
    ...,
    stderr=build_stderr_callback("maintenance"),
)

Labels currently in use: "maintenance" (fix agent), "reviewer" (reviewer agent), "doc-updater" (doc generation), "conversation" (conversation agent).

Retry Logic (`resilient_query`)¶

The resilient_query wrapper in core.agent_sdk_utils provides two layers of resilience around the Agent SDK's query() function:

Skip — MessageParseError exceptions (e.g., unknown rate_limit_event messages) are logged and skipped so the remaining stream is still consumed.
Retry — ProcessError, CLIConnectionError, and CLI error messages re-raised as plain Exception by the SDK trigger up to max_retries retries (default 1) with exponential backoff (default base 2.0s). This masks intermittent exit-code-1 crashes.

from core.agent_sdk_utils import resilient_query

async for message in resilient_query(prompt=user_prompt, options=options):
    if isinstance(message, ResultMessage):
        result_message = message

Both the fix agent and the reviewer agent use resilient_query for all Agent SDK interactions.

Safety Layers¶

The service enforces multiple layers of protection:

Kill switch — MINH_SELF_MAINTENANCE_ENABLED defaults to false.
Redis distributed lock — TTL of 1 hour prevents concurrent runs across pods.
Daily run counter — Redis-backed counter caps total runs per calendar day (0 = unlimited).
Excluded labels — Issues labelled self-maintenance-generated, wontfix, blocked, or complex are skipped.
Complexity filter — Issues with descriptions longer than 3,000 characters are excluded.
Open MR deduplication — Issues that already have an open minh-fix-{iid}-* branch MR are skipped.
Per-run budget cap — Cumulative cost across all issues is tracked; processing stops when the budget is exhausted.
Per-issue turn cap — Each agent session is limited to a configurable number of agentic turns.
Stderr capture — All agent sessions use build_stderr_callback to log Claude CLI stderr, ensuring crash diagnostics are not lost.
Auto-merge kill switch — MINH_SELF_MAINTENANCE_AUTO_MERGE defaults to false, requiring explicit opt-in.
Diff size gate — MRs with diffs exceeding MINH_SELF_MAINTENANCE_MAX_DIFF_LINES lines are left for human review.
CI gate — Auto-merge requires a passing CI pipeline; failed or timed-out pipelines block the merge.
Test coverage gate — Non-documentation MRs that change src/ without corresponding tests/ changes are rejected and closed.
Independent reviewer — The reviewer agent is a separate session with read-only access and cannot modify code; when in doubt, it rejects.

All safety mechanisms degrade gracefully when Redis is unavailable (locks are skipped, daily limits are not enforced).

Usage¶

Triggering via Slack¶

Users can trigger self-maintenance through natural language in Slack:

@Minh run self-maintenance
@Minh work on your backlog
@Minh check your issues
@Minh fix your bugs

Minh will respond with a preview of qualifying issues, the configured budget, and max turns, then ask for confirmation:

🔍 Self-Maintenance Preview — 2 qualifying issue(s):

  • #42 Fix broken import in vector_memory  (bug)
  • #58 Add search_docs category filter  (mcp-extension)

Budget: $2.00 | Max turns: 45

Reply `yes` to proceed or `no` to cancel.

After confirmation, the run executes in the background and posts per-issue results and a summary to the thread. When auto-merge is enabled, each successful result also includes the review verdict (auto-merged, approved but merge failed, or left for human review).

Agent Workflow¶

Each agent session follows a structured workflow:

Understand — Read the issue, search docs, explore codebase.
Plan — Identify files to change; keep changes minimal.
Implement — Edit files following existing code style.
Test — Run pytest; create tests if none exist. (Skipped for stale-docs issues, which are documentation-only.)
Commit & Push — Branch naming: minh-fix-{issue_iid}-{short_desc}.
Create MR — Opens a merge request (does not merge).
Report — Outputs a JSON result block via structured output.
Review & Merge (auto-merge only) — A separate reviewer agent evaluates the MR diff and either approves (triggering auto-merge), rejects (posting a review comment and closing the MR), or produces unparseable output (MR left open for human review).

Structured output¶

The fix agent returns its results via Claude Agent SDK structured output. The schema requires four fields:

Field	Type	Description
`success`	`bool`	Whether the agent believes it resolved the issue
`mr_url`	`str`	URL of the merge request created (empty on failure)
`error`	`str`	Error description when `success` is `false`
`follow_up_issues`	`list[str]`	Descriptions of related work the agent identified but could not address

The service reads these from ResultMessage.structured_output and populates an IssueResult dataclass that also tracks operational metadata:

Field	Type	Source
`issue_iid`	`int`	From the GitLab issue
`issue_title`	`str`	From the GitLab issue
`project_id`	`str`	The GitLab project path
`turns_used`	`int`	`ResultMessage.num_turns`
`elapsed_seconds`	`float`	Wall-clock time for the agent session
`cost_usd`	`float`	`ResultMessage.total_cost_usd`
`review_verdict`	`ReviewVerdict \\| None`	Populated when auto-merge is enabled

Example structured output from a successful run:

{
  "success": true,
  "mr_url": "https://gitlab.com/the-smithy1/agents/minh/-/merge_requests/147",
  "error": "",
  "follow_up_issues": [
    "The retry backoff constants in resilient_query are hard-coded; consider making them configurable"
  ]
}

Example Slack notification posted per issue:

✅ Self-maintenance fixed issue #42: Fix broken import in vector_memory
MR: https://gitlab.com/the-smithy1/agents/minh/-/merge_requests/147
Turns: 23 | Time: 94s | Cost: $0.4812
🚀 Auto-merged — Reviewer: Change correctly fixes the import path and adds a regression test.

When an issue fails:

{
  "success": false,
  "mr_url": "",
  "error": "pytest exited with code 1 — 2 test failures remain after 3 fix attempts",
  "follow_up_issues": []
}

A RunResult aggregates all IssueResult entries for the cycle and exposes success_count, failure_count, and success_rate (percentage) properties for KPI tracking.

Configuration¶

All configuration is via environment variables:

Variable	Default	Description
`MINH_SELF_MAINTENANCE_ENABLED`	`false`	Kill switch — must be `true` to enable
`MINH_SELF_MAINTENANCE_MAX_TURNS`	`50`	Maximum agentic turns per issue
`MINH_SELF_MAINTENANCE_MAX_ISSUES`	`0`	Maximum issues to process per run (0 = unlimited)
`MINH_SELF_MAINTENANCE_MAX_DAILY_RUNS`	`0`	Maximum runs per calendar day; 0 = unlimited (Redis-backed)
`MINH_SELF_MAINTENANCE_MAX_BUDGET_USD`	`2.0` (API) / `15.0` (subscription)	Cumulative cost cap per run. Default depends on `MINH_AGENT_SDK_AUTH`
`MINH_AGENT_SDK_AUTH`	`api`	Auth mode (`api` or `subscription`); affects default budget
`MINH_GITLAB_PROJECT`	`the-smithy1/agents/minh`	Default GitLab project for issue queries (single-project fallback)
`MINH_SELF_MAINTENANCE_PROJECTS`	(unset)	JSON object mapping project paths to label lists for multi-project mode (see below)
`MINH_SELF_MAINTENANCE_AUTO_MERGE`	`false`	Enable the reviewer agent and auto-merge for successful MRs
`MINH_SELF_MAINTENANCE_MAX_DIFF_LINES`	`500`	Maximum diff lines eligible for auto-merge; larger diffs are left for human review
`MINH_SELF_MAINTENANCE_CI_POLL_INTERVAL`	`30`	Seconds between CI status polls during auto-merge
`MINH_SELF_MAINTENANCE_CI_POLL_TIMEOUT`	`600`	Maximum seconds to wait for CI before skipping auto-merge

Multi-project configuration¶

When MINH_SELF_MAINTENANCE_PROJECTS is set, the service processes issues across multiple GitLab projects. The value is a JSON object mapping project paths to their qualifying label lists:

{
  "the-smithy1/agents/minh": ["bug", "enhancement", "mcp-extension", "stale-docs", "broken-links"],
  "the-smithy1/agents/knowledge-sanctuary": ["bug", "enhancement", "documentation"]
}

If the variable is absent or invalid, the service falls back to the single project from MINH_GITLAB_PROJECT with the default qualifying labels.

Redis keys and lock tuning¶

The service uses two Redis key families for concurrency control and rate limiting:

Key	Format	TTL	Purpose
`self_maintenance:lock`	String (pod hostname)	3 600 s (1 hour)	Distributed lock — prevents concurrent runs across pods
`self_maintenance:daily_runs:YYYY-MM-DD`	Integer counter	86 400 s (24 hours)	Daily run counter — caps total runs per calendar day

Distributed lock¶

The lock is acquired with SET key value NX EX 3600, where the value is the pod's hostname (via socket.gethostname()). Storing the hostname makes it possible to identify which pod holds a stale lock during debugging. The lock is released with DELETE in a finally block after the run completes.

Tuning the lock TTL — The default 1-hour TTL (REDIS_LOCK_TTL = 3600) is set in self_maintenance_service.py as a module-level constant. It is not currently exposed as an environment variable. If runs consistently complete in under 30 minutes, the TTL could safely be lowered to reduce the window during which a crashed pod blocks subsequent runs. Conversely, if runs process many issues and exceed 1 hour, the lock may expire mid-run and allow a second concurrent execution. In that scenario, either increase the TTL in code or reduce the issue cap (MINH_SELF_MAINTENANCE_MAX_ISSUES) to keep runs within the lock window.

Daily run counter¶

When MINH_SELF_MAINTENANCE_MAX_DAILY_RUNS is non-zero, the service checks the counter before starting and increments it (via a Redis pipeline with INCR + EXPIRE) after a successful run. The key includes the date in ISO format (e.g., self_maintenance:daily_runs:2026-07-06) so it naturally partitions by calendar day. The 24-hour TTL ensures stale keys are cleaned up even if the date rolls over.

A value of 0 (the default) disables the daily limit entirely — the counter is neither checked nor incremented.

Graceful degradation¶

All Redis operations — lock acquisition, lock release, daily limit check, and counter increment — are wrapped in try/except. When Redis is unavailable, locks are skipped (the run proceeds), daily limits are not enforced, and counters are not incremented. This ensures self-maintenance remains functional during Redis outages, at the cost of losing concurrency protection.

Error Handling¶

Redis unavailable — All Redis-dependent gates (lock, daily limit) degrade gracefully and allow the run to proceed.
Agent SDK subprocess crash — resilient_query retries transient CLI crashes (ProcessError, CLIConnectionError, exit-code-1 errors) up to max_retries times (default 1) with exponential backoff. If all retries are exhausted, the exception propagates and is recorded in IssueResult.error. Stderr lines from the CLI subprocess are captured via build_stderr_callback for post-mortem diagnostics.
Budget exhaustion — When cumulative cost exceeds the budget, remaining issues are skipped and a Slack notification is posted.
Unparseable agent output — The service reads structured output from the agent's ResultMessage. If the result message is an error or absent, the issue is marked as failed.
Slack notification failures — All Slack posts are wrapped in try/except to prevent notification errors from aborting the run.
Review agent failure — Exceptions during the review phase are captured in ReviewVerdict.reason; the MR is left for human review. The fix result itself is not affected.
Unparseable reviewer output — If the reviewer agent returns output that cannot be parsed into a verdict, the MR is left open for human review with an explanatory comment (rather than being auto-closed).
CI timeout — If the CI pipeline does not reach a terminal state within MINH_SELF_MAINTENANCE_CI_POLL_TIMEOUT seconds, auto-merge is skipped and the MR is left for human review.
Test coverage rejection — Non-documentation MRs that change src/ files without corresponding tests/ changes are rejected with a review comment and closed, so the issue becomes eligible for retry.
Merge failure — If the reviewer approves but the merge command fails (e.g., merge conflict), the verdict records approved=True, merged=False and a warning is logged.
MCP stdout corruption — Prevented by the stdio safety pattern: all MCP tool servers redirect structlog and logging to stderr before any imports, ensuring stdout is reserved for JSON-RPC communication.

Qualifying Labels¶

Issues must have at least one of the default qualifying labels to be picked up:

bug — Code defects, broken imports, incorrect behaviour.
enhancement — Small improvements to existing functionality (new config options, better error messages, minor refactors).
mcp-extension — Additions or changes to the MCP tool servers (new tools, parameter changes).
stale-docs — Documentation that no longer matches the current code. These issues trigger a specialised docs-only workflow: the agent skips testing and focuses on reading the code and updating the corresponding documentation.
broken-links — Documentation pages with broken internal links. Like stale-docs, these trigger a docs-only workflow where the agent locates the correct target or removes dead links.

Issues are excluded if they carry any of:

self-maintenance-generated — Applied automatically to issues created by self-maintenance follow-ups, preventing infinite loops.
wontfix — Explicitly marked as not worth fixing.
blocked — Waiting on an external dependency or decision.
complex — Requires multi-system coordination, architectural changes, or human judgment beyond what the agent can reliably handle.

Labelling an issue for self-maintenance¶

To make an issue eligible for autonomous resolution:

Apply exactly one qualifying label from the list above. The service queries GitLab once per label and deduplicates by IID, so multiple qualifying labels on the same issue will not cause duplicate processing, but a single label keeps the intent clear.
Keep the description under 3,000 characters. Issues with longer descriptions are assumed to be complex and are silently excluded. If the issue needs extensive context, link to external documents or code files rather than inlining everything.
Do not apply any excluded label. An issue with both bug and blocked will be skipped. Remove the blocking label when the issue becomes actionable.
Assign the issue to Minh's GitLab user (if your project filters by assignee). The IssueEvaluator resolves the authenticated GitLab username at query time and only fetches issues assigned to that user.
Check for open MRs. The service scans open merge requests for branches matching the minh-fix-{iid}-* pattern. If an open MR already exists for the issue IID, the issue is skipped to avoid duplicate work. Close or merge the stale MR first if you want the issue to be retried.

Per-project label customisation¶

When MINH_SELF_MAINTENANCE_PROJECTS is configured, each project specifies its own qualifying label list. These override the default QUALIFYING_LABELS for that project. For example, in the current production configmap:

{
  "the-smithy1/agents/minh": ["bug", "enhancement", "mcp-extension", "stale-docs", "broken-links"],
  "the-smithy1/agents/knowledge-sanctuary": ["bug", "enhancement", "documentation"]
}

The knowledge-sanctuary project uses documentation instead of stale-docs because its issues follow a different labelling convention. The excluded labels (self-maintenance-generated, wontfix, blocked, complex) apply globally across all projects and are not configurable per project.

Tips for writing self-maintenance-friendly issues¶

One fix per issue. The agent works best with tightly scoped problems. If an issue requires changes across multiple subsystems, split it.
Include file paths. Mentioning src/services/foo.py in the description helps the agent locate the relevant code faster, reducing turn usage.
Describe expected behaviour. "The /status endpoint should return 200" is more actionable than "status is broken."
Link to related docs. For stale-docs issues, list both the code paths that changed and the doc paths that need updating.