How the Compile Pipeline Denoises Your Knowledge
Most knowledge management tools chunk documents, vectorize them, and let an LLM answer questions from raw fragments. KaaS takes the opposite approach: it first "compiles" raw content into structured Wiki articles, then lets the LLM retrieve from those articles on demand.
This design requires a reliable compile pipeline. The challenge: when users submit dozens of files at once, a single hanging LLM call or multiple files independently generating the same concept can introduce noise (timeout residues) and duplication (near-identical wiki articles). This post breaks down KaaS's 4-phase compile pipeline, focusing on the two critical points where denoising and deduplication happen.
The Two Problems
KaaS compilation faces two engineering challenges:
1. Noise: A single LLM call can stall the entire pipeline
The Extract phase makes an LLM call for each document. If one call hangs due to network or model issues, it occupies one of 16 concurrent workers indefinitely, eventually stalling the entire batch.
2. Duplication: Parallel classification independently "invents" the same article
The Classify phase processes multiple file groups in parallel. Different groups seeing the same topic will each produce a create_new entry with the same name, resulting in two nearly identical Wiki articles.
Phase 1: Parallel Extract — Two Lines of Defense Against Noise
The first phase uses a ThreadPoolExecutor to concurrently call extract_knowledge_chunked() on all files. Denoising operates at two layers:
Defense 1: Per-call timeout of 180 seconds
Each Extract LLM call is wrapped with a @_with_extract_timeout decorator that sets a 180-second timeout. This ensures any single call occupies at most 180 seconds of worker time, preventing timeout propagation to other workers.
Defense 2: Chunked summary compression
For long documents, the pipeline splits content into ~4000-token chunks, then applies a two-phase summarization:
- Phase A: Parallel Haiku calls generate 200–500 word summaries per chunk
- Phase B: Concatenated summaries go to Sonnet for structured extraction
When chunk count reaches 20 or summaries exceed 60,000 characters, an L2 hierarchical merge kicks in — Haiku batches with fanout=5 compress multiple chunk summaries before Phase B. This guarantees that regardless of original document length, Sonnet's input stays bounded.
Adaptive dispatch rules:
| Successful chunks | Strategy |
|---|---|
| ≤ 3 | K=1, single Sonnet extraction |
| 4–7 | K=2, parallel A/B field groups |
| 8–19 | K=3, parallel A/B/C field groups |
| ≥ 20 or summaries > 60K | L2 Haiku merge → K=3 |
Phase 2a: Classify — Two Gates Against Duplication
The classify_article() function asks Sonnet to decide: should this content create a new article or merge into an existing one? Two dedup gates prevent duplicates:
Gate 1: Title word-overlap within a group
After each classification, dedup_create_new() checks the create_new list against existing articles using title word overlap. Titles are lowercased and split into word sets; overlap ratio = intersection size / smaller set size. At a threshold of 0.7 (70% word overlap), the new article is redirected to merge instead.
Gate 2: Cross-group dedup
After all Classify groups complete in parallel, a Phase 1.5 step aggregates create_new lists from every group and runs dedup_create_new() again — this time treating other groups' planned articles as "existing." Duplicates get moved from create_new to merge_into.
The _cluster_by_topic_overlap() function pre-groups topically similar files before classification to minimize cross-group collisions, but the Phase 1.5 dedup serves as the final safety net.
Phase 2b: Parallel Write
After deduplication, operations are aggregated by target article path. Each target article becomes an independent task submitted to a thread pool (up to 16 workers):
- Existing articles →
merge_into_article()(incremental diff merge) - New articles →
create_new_article()(full generation)
Phase 3: Index
After writing completes, update_markdown_index() scans all Markdown files under wiki/ and generates three plain-text indexes:
- master-index.md — alphabetically sorted full article list
- topic-index.md — topic tags that meet a frequency threshold
- topic-index-longtail.md — low-frequency long-tail tags
These indexes are the entry point for the retrieval phase: the LLM "browses the table of contents first, then reads full articles."
Conclusion
KaaS's choice of "compile first, retrieve later" over "direct RAG" means compilation quality directly determines retrieval quality. Denoising (180s timeout isolation + L2 summary compression) and deduplication (title word-overlap gate + cross-group dedup) are the two pillars keeping this pipeline stable.
The current title-overlap dedup is pure string computation — no vectors or embeddings needed — which is particularly friendly for resource-constrained local deployments. The tradeoff: it's blind to synonyms ("API Gateway" and "网关" are treated as completely different). This is a known limitation and a direction for future exploration.
Interested? Browse the py/src/kb_ai/ directory on GitHub — the code is all open source.