The Compile Pipeline

The compile pipeline transforms raw content into structured wiki articles through four sequential LLM stages. The Go backend orchestrates execution with concurrent workers; the Python AI engine (kb-ai) runs the actual LLM calls.

Pipeline Overview

┌─────────┐    ┌──────────┐    ┌───────┐    ┌───────┐
│ Extract │───▶│ Classify │───▶│ Write │───▶│ Index │
└─────────┘    └──────────┘    └───────┘    └───────┘
   parallel       sequential      sequential    sequential

The Extract stage runs in parallel across a worker pool. Once extraction completes for a batch, the remaining stages run sequentially per user within the Pipeline Worker.

Stage 1: Extract

Goal: Pull structured information out of raw text.

The Extract stage reads the submitted content and produces a structured extraction containing:

• Concepts and entities
• Decisions and rationale
• Action items and ownership
• Key facts and relationships

Extraction Strategies

Strategy	When used	How it works
chunked	Short-to-medium documents	Split into chunks; extract from each independently
summarize	Long documents, transcripts	Summarize chunks with a cheaper model first, then extract from summaries
auto	Default	Automatically picks chunked or summarize based on chunk count

The summarize strategy is significantly cheaper for long content (meeting transcripts, large documents) because it uses a smaller model for the summarization pass before feeding condensed text to the extraction model.

Parallel Execution

The Go Dispatcher assigns Extract tasks to a pool of workers (default: 4). Each worker independently calls the Python AI engine's /extract endpoint. Multiple documents are extracted concurrently, bounded by the semaphore in the Dispatcher.

Stage 2: Classify

Goal: Decide where each extracted item belongs in the wiki.

The Classify stage examines the extracted items and the current wiki structure, then:

1. Maps items to existing wiki articles (for merging)
2. Identifies items that need new articles
3. Groups related items to avoid fragmentation

This stage reads the current master-index.md to understand existing wiki topology before making decisions.

Stage 3: Write

Goal: Produce or update Markdown articles.

Based on classification results, the Write stage either:

• Creates a new Markdown article from the extracted items
• Merges new information into an existing article, deduplicating and restructuring as needed

Output is clean Markdown with headings, lists, and interlinks — designed to be readable by both humans and LLMs.

Stage 4: Index

Goal: Keep navigation indexes current.

After articles are written, the Index stage updates:

• master-index.md — the top-level catalog used by LLM retrieval to navigate the wiki
• Topic indexes — per-topic groupings of related articles
• People stubs — brief reference pages for mentioned people

The master-index is the critical artifact: it serves as the entry point for all LLM-iterative retrieval queries.

Incremental Compilation

KaaS only recompiles new or changed content. When you submit material that overlaps with existing wiki content:

1. Extract runs on the new submission only
2. Classify identifies overlap with existing articles
3. Write merges (not replaces) — preserving existing content while incorporating new information
4. Index refreshes only affected entries

This means submitting 10 documents doesn't re-process the previous 100.

Concurrency & Acceleration

Dispatcher

The Dispatcher is the central coordinator:

• Polls the task queue on a configurable interval (poll_interval_ms)
• Claims tasks with a lease (prevents double-processing)
• Enforces a concurrency cap via Go channel-based semaphore
• Pauses claiming when the circuit breaker is open

Extract Worker Pool

Multiple goroutines process Extract tasks in parallel:

• Default concurrency: 4 workers (extract_workers in config)
• Each worker calls Python's /extract endpoint via HTTP
• Progress is reported via SSE streaming back to the frontend

Pipeline Worker

After extraction, a single Pipeline Worker per user batch runs the remaining stages:

• Classify → Write → Index executed sequentially
• Batches all pending extracted items for one user together
• Calls Python's /pipeline-stream endpoint (SSE progress)

Circuit Breaker

Protects against LLM provider outages:

Parameter	Default	Description
cb_failure_threshold	5	Consecutive failures to trigger open state
cb_cooldown_sec	30	Seconds before half-open probe

State machine:

1. Closed — normal operation; failures increment a counter
2. Open — all calls rejected immediately; Dispatcher stops claiming tasks
3. Half-Open — after cooldown, one probe request is allowed; success closes the breaker, failure re-opens it

Lease & Heartbeat

Each claimed task carries a lease with a timeout (lease_timeout_sec, default 300s). Workers periodically renew the lease via heartbeat. If a worker crashes:

• The lease expires
• The Dispatcher's RecoverExpired sweep re-enqueues the task
• Another worker picks it up

This guarantees at-least-once processing without manual intervention.

Configuration Reference

All compile pipeline settings live in etc/kaas.toml under [worker]:

[worker]
extract_workers = 4          # Extract parallel worker count
pipeline_concurrency = 2     # Pipeline worker concurrency
poll_interval_ms = 1000      # Queue poll interval (ms)
lease_timeout_sec = 300      # Lease timeout before recovery (s)
cb_failure_threshold = 5     # Consecutive failures to open breaker
cb_cooldown_sec = 30         # Cooldown before half-open probe (s)