Weaviate Documentation Indexing¶

Overview¶

Weaviate Documentation Indexing is the pipeline that fetches documentation files (Markdown, text, reStructuredText) from GitLab repositories and stores them in Minh's Weaviate vector database for semantic search. This enables Minh to answer questions by searching across all indexed project documentation using natural-language similarity rather than keyword matching.

Each repository is tagged with a product and content_type to support product-aware search and timeline-aware RAG responses. This metadata flows from the repo registry through indexing into Weaviate, and is surfaced in search results and LLM context.

The pipeline supports both full reindexing (clear-and-rebuild) and incremental indexing (diff-based updates since the last indexed commit). Incremental indexing reduces GitLab API calls and Weaviate write load by only processing added, modified, renamed, and deleted files.

The feature comprises three main components:

Repo Registry (src/core/repo_registry.py) — the single source of truth for which repositories are indexable, including product and content-type classification.
Repository Indexing Service (src/services/repository_indexing_service.py) — orchestrates fetching files from GitLab and storing them in Weaviate, supporting both full and incremental indexing modes.
Vector Memory (src/core/vector_memory.py) — manages the Weaviate connection, the Documentation collection where indexed content is stored, and the IndexingState collection that tracks per-repo commit SHAs for incremental indexing.

Architecture¶

Repo Registry¶

RepositoryConfig is a frozen dataclass that defines each indexable repository:

Field	Type	Default	Description
`name`	`str`	—	Short repository identifier (also used as the Weaviate `category` value)
`gitlab_path`	`str`	—	Full GitLab project path (e.g. `the-smithy1/agents/minh`)
`product`	`str`	`"glass-umbrella"`	Product grouping — must be one of `VALID_PRODUCTS`
`content_type`	`str`	`"technical"`	Content classification — must be one of `VALID_CONTENT_TYPES`
`branch`	`str`	`"main"`	Branch to read files from
`file_patterns`	`tuple`	`(".md", ".txt", "*.rst")`	Glob patterns for files to index
`max_files`	`int`	`200`	Maximum number of files indexed per repo

Valid product values (`VALID_PRODUCTS`)¶

grit, pariah, paragon, paramount, glass-umbrella, agents, mastery

Valid content-type values (`VALID_CONTENT_TYPES`)¶

worldbuilding, narrative, game-implementation, technical

All configured repositories are listed in the REPO_REGISTRY tuple. As of the latest changes, this includes repositories such as knowledge-sanctuary, minh, glass-umbrella-knowledge-repo, agents-shared, wisp, spark, pearl, taryn, arturo, roland, chisel, and the Paradigm trilogy repos (paradigm-pariah, paradigm-paragon, paradigm-paramount, shared-universe) and mobile repos (reader, smithy-auth), among others.

Helper functions¶

get_repo_config(name) — return the RepositoryConfig for a given repo name.
get_repo_path(name) — return the gitlab_path for a given repo name.
get_repos_by_product(product) — return all repos belonging to a product (e.g. "grit", "agents").
get_repos_by_content_type(content_type) — return all repos with a given content type (e.g. "worldbuilding").
get_gitlab_base_url(category) — return the GitLab blob URL for a Weaviate document category.

Weaviate Collections¶

Documentation Collection¶

The Documentation collection in Weaviate stores each indexed file with these properties:

title (text) — extracted from the first # heading, or the filename
content (text) — full file content (truncated at 50 000 characters)
file_path (text) — path within the repository
url (text) — GitLab blob URL base for the source repo
category (text) — repository name, used for filtering
tags (text array) — includes the repo name and file extension
product (text) — product grouping from the repo config (e.g. "grit", "agents"); stored only when non-empty
content_type (text) — content classification from the repo config (e.g. "worldbuilding", "technical"); stored only when non-empty
last_updated (date) — timestamp when the document was indexed
access_count (int) — initialised to 0

Vectorisation uses the text2vec-transformers module.

IndexingState Collection¶

The IndexingState collection tracks per-repository commit SHAs to support incremental indexing. It stores metadata only — no vectoriser is configured (uses Vectorizer.none() to save CPU).

Properties:

repo_name (text) — the repository name (matches the category value in Documentation)
last_sha (text) — the HEAD commit SHA at the time of the last successful index
last_indexed_at (date) — RFC 3339 timestamp of when the state was recorded

This collection is created automatically by VectorMemory.ensure_collections() if it does not already exist.

GitLab Client — Incremental Indexing Support¶

The GitLabClient provides two methods used by incremental indexing:

get_head_sha(project_id, ref="main") — returns the latest commit SHA for a branch. Returns None on error.
compare_commits(project_id, from_sha, to_sha) — compares two commits and returns a dict with keys added, modified, and deleted (each a list of file paths). Renamed files are treated as a delete of the old path plus an add of the new path, keeping the index clean. Returns None on error (caller falls back to full reindex).

Indexing Flow¶

Full Reindex¶

Repo Registry ──▶ RepositoryIndexingService.index_repository()
                        │
                        ├── GitLabClient.get_documentation_files()
                        │       (list files matching extensions)
                        │
                        ├── clear_repository()
                        │       (delete existing Weaviate docs for this repo)
                        │
                        ├── for each file:
                        │       GitLabClient.get_file_content()
                        │         ──▶ VectorMemory.store_documentation()
                        │               (includes product & content_type from repo config)
                        │
                        └── VectorMemory.set_indexing_state()
                                (record HEAD SHA for future incremental runs)

Before re-indexing a repository, the service deletes all existing documents with that repository's category to ensure a clean state. After a successful run, the current HEAD SHA is recorded so future runs can index incrementally.

Incremental Reindex¶

Repo Registry ──▶ RepositoryIndexingService.index_repository_incremental()
                        │
                        ├── VectorMemory.get_indexing_state()
                        │       (fetch last-indexed SHA)
                        │
                        ├── GitLabClient.get_head_sha()
                        │       (current HEAD)
                        │
                        ├── GitLabClient.compare_commits(last_sha → head_sha)
                        │       (returns added / modified / deleted file lists)
                        │
                        ├── deleted files ──▶ VectorMemory.delete_documentation_by_paths()
                        │
                        ├── added / modified files ──▶ for each file:
                        │       GitLabClient.get_file_content()
                        │         ──▶ VectorMemory.upsert_documentation()
                        │
                        └── VectorMemory.set_indexing_state()
                                (record new HEAD SHA)

Incremental indexing falls back to full reindex when:

No previous SHA is recorded (first run for this repo).
The diff exceeds MAX_INCREMENTAL_FILES (default: 50) — the overhead of many individual GitLab file fetches and Weaviate upserts dominates above this point.
The GitLab compare API fails (e.g. force-push, rebase, API error).

No clear_repository() call is made during incremental indexing — the index remains searchable throughout.

Vector Memory — Incremental Indexing Methods¶

delete_documentation_by_paths(category, file_paths) — deletes specific documents from a repo category by file path, without clearing the entire repo. Returns the number of objects deleted (0 on error or when not connected).
upsert_documentation(...) — insert-or-replace a single document by (category, file_path). Deletes any existing match first, then inserts the new version. Guarantees no duplicates while keeping the collection searchable throughout.
get_indexing_state(repo_name) — fetch the stored indexing state for a repo. Returns a dict with last_sha and last_indexed_at, or None if no record exists (callers should treat missing state as "do a full reindex").
set_indexing_state(repo_name, sha) — upsert the indexing state for a repo. Deletes any prior record then inserts a fresh one (Weaviate v4 does not provide native upsert for non-deterministic UUIDs). Returns True on success, False on error.

RAG Response Generation¶

When Minh answers a question using indexed documentation, the RAG pipeline now annotates each document heading with its product provenance. For example, a document from the pariah product with content type narrative is rendered as:

### Document Title [Pariah / narrative]

The LLM prompt instructs the model to note distinctions when documents come from different products or timelines (e.g. Pariah vs Paragon), helping produce more accurate, context-aware responses.

Usage¶

Index all repositories¶

from services.repository_indexing_service import RepositoryIndexingService

service = RepositoryIndexingService(
    gitlab_client=gitlab_client,
    vector_memory=vector_memory,
)
results = service.index_all()

for name, result in results.items():
    print(f"{name}: indexed={result.files_indexed}, errors={len(result.errors)}")

Index a single repository by name¶

result = service.index_by_name("knowledge-sanctuary")
if result and result.success:
    print(f"Indexed {result.files_indexed} files")

Index a single repository incrementally¶

from core.repo_registry import get_repo_config

repo_config = get_repo_config("knowledge-sanctuary")
result = service.index_repository_incremental(repo_config)
if result and result.success:
    print(f"Indexed {result.files_indexed} files (incremental)")

Incremental indexing automatically falls back to a full reindex when no prior state exists or the diff is too large (see Indexing Flow — Incremental Reindex for details).

Search indexed documentation¶

from core.vector_memory import VectorMemory

vm = VectorMemory()
results = vm.search_documentation("how does the repo registry work", limit=5)

# Filter by repository
results = vm.search_documentation("deployment steps", category="minh", limit=3)

# Results now include product and content_type fields
for doc in results:
    print(f"{doc['title']} [{doc['product']} / {doc['content_type']}]")

Query repositories by product or content type¶

from core.repo_registry import get_repos_by_product, get_repos_by_content_type

# Get all agent repositories
agent_repos = get_repos_by_product("agents")
for repo in agent_repos:
    print(f"{repo.name}: {repo.gitlab_path}")

# Get all worldbuilding repositories
worldbuilding_repos = get_repos_by_content_type("worldbuilding")

# Get all Paradigm narrative repositories
pariah_repos = get_repos_by_product("pariah")
paramount_repos = get_repos_by_product("paramount")

Inspect and manage indexing state¶

from core.vector_memory import VectorMemory

vm = VectorMemory()

# Check last-indexed SHA for a repo
state = vm.get_indexing_state("knowledge-sanctuary")
if state:
    print(f"Last SHA: {state['last_sha']}, indexed at: {state['last_indexed_at']}")
else:
    print("No indexing state — full reindex required")

# Manually set indexing state (e.g. after a migration)
vm.set_indexing_state("knowledge-sanctuary", "abc123def456")

Delete specific documents by file path¶

# Remove specific files from the index without clearing the entire repo
deleted = vm.delete_documentation_by_paths("minh", ["docs/old-page.md", "docs/renamed.md"])
print(f"Deleted {deleted} documents")

Clear a repository's indexed documents¶

service.clear_repository("knowledge-sanctuary")

Configuration¶

Environment Variables¶

Variable	Default	Description
`WEAVIATE_URL`	—	Full Weaviate URL (e.g. `http://weaviate:8080`). Takes priority over host/port.
`WEAVIATE_HOST`	`localhost`	Weaviate hostname (used if `WEAVIATE_URL` is unset)
`WEAVIATE_PORT`	`8080`	Weaviate port (used if `WEAVIATE_URL` is unset)
`GITLAB_GROUP`	`the-smithy1`	GitLab group prefix for all repository paths

Constants¶

Constant	Value	Location	Description
`MAX_CONTENT_LENGTH`	50 000	`repository_indexing_service.py`	Maximum characters per file before truncation
`MAX_INCREMENTAL_FILES`	50	`repository_indexing_service.py`	Diff size above which incremental indexing falls back to full reindex

Prerequisites¶

A running Weaviate instance with the text2vec-transformers vectoriser module enabled.
A GitLab access token with read access to the repositories listed in the registry.
The weaviate-client Python package (v4 API).

Error Handling¶

If the GitLab client or vector memory is not configured, index_repository returns an IndexingResult with the error recorded in the errors list.
Files that fail to fetch or store are logged individually; the service continues with the remaining files.
Empty files are skipped (counted in files_skipped).
Content exceeding 50 000 characters is truncated with a [Content truncated...] marker.
VectorMemory methods silently log errors and return empty results when Weaviate is unreachable, so callers degrade gracefully.
Every RepositoryConfig must use a product value from VALID_PRODUCTS and a content_type value from VALID_CONTENT_TYPES; tests enforce this constraint across the entire registry.
get_indexing_state() returns None on any error; callers should treat missing state as "do a full reindex".
set_indexing_state() returns False on error, and the consequence depends on where it failed. An early return — not connected, read-only cooldown, or missing arguments — leaves any prior record intact, so the next run still indexes incrementally, just from the older SHA (safe: the comparison still spans every change since). If the delete succeeds and the insert then fails, no record remains and the next run falls back to a full reindex. Either way the index stays correct; a persistently failing state write is worth alerting on because of the repeated cost, not because of drift.