LLMs Are Smart, So Why Don’t They Know How Our Company Works?
LLMs handle public knowledge well but struggle with company-specific questions because relevant evidence is scattered across documents, code, meetings, and chat—and may be outdated or contradictory. The post argues that this is not merely a search problem: organizations need a shared layer for managing trustworthy context. Topic addresses this by extracting source-aware units, linking concepts and relationships, and verifying their freshness, consistency, and evidentiary support. ## Why Search Alone Is Not Enough - Search retrieves relevant text but cannot determine whether it is current, authoritative, or consistent with other sources. - A retry-policy example might contain: - Documentation saying requests fail immediately - A meeting discussion proposing three retries - Code currently implementing two retries - Agents must still determine: - Whether the meeting produced a final decision - Which source is newer - Whether the code reflects an intentional change or an unfinished implementation - Whether different sources describe the same behavior - Topic provides a shared context layer so humans and LLMs use the same sources, relationships, freshness information, and conflict states. ## Six Dimensions of Trust Rather than compressing trust into one score, Topic evaluates six separate dimensions: - **Granularity:** Whether the context is a meaningful, independently manageable unit - **Faithfulness:** Whether the source actually supports the claim - **Staleness:** Whether the evidence remains valid - **Canonicality:** Whether different names refer to the same entity - **Consistency:** Whether sources are compatible - **Coverage:** Whether important evidence or perspectives are missing Different checks use different methods: rules and hashes for deterministic validation, LLMs for semantic interpretation, and humans for ambiguous or high-impact decisions. ## Ingesting Documents, Chat, and Code Topic normalizes information into a common `ContentUnit` containing source type, unit type, original URI, content, hashes, timestamps, and source-specific metadata. It uses different boundaries for each source rather than splitting everything into fixed-size text chunks. ### Structured Document Sections - Markdown documents are divided by heading hierarchy. - Parent headings are preserved to retain context. - Long sections are split only when necessary. - URLs, document paths, and creation or modification times remain attached to the unit. ### Conversation Threads - Entire messenger threads are treated as the semantic unit, not individual messages. - Summaries preserve: - Technical identifiers such as function names and file paths - Questions, alternatives, and final outcomes - Decisions versus unresolved issues - The system avoids inventing consensus and ignores threads containing only casual conversation. ### Code Symbols and Semantic Cards - Parsers extract functions, classes, file paths, line ranges, imports, and other symbols without using an LLM. - Multiple symbols are then grouped into **code semantic cards** describing business behavior. - Cards retain domain terms, code identifiers, source spans, and the relevant commit SHA. - LLM-generated cards are checked against actual files, line ranges, supporting spans, and duplicate-card patterns. - Cards are an intermediate layer for connecting code to business concepts, not a replacement for the code itself. ## Extracting Concepts and Relationships - Topic extracts concept candidates and supporting evidence from each content unit. - It preserves the relationship between every concept and its original evidence. - Similar names are not automatically merged merely because they appear close in meaning. - Concepts can be consolidated into canonical entities only when sufficient evidence exists. ### Human Review for Ambiguous Terminology - Normalization and embeddings can identify obvious duplicates. - Internal abbreviations and aliases may require organizational knowledge. - Topic creates synonym proposals with their supporting context. - Humans approve or reject ambiguous aliases; rejected proposals are remembered to prevent repeated suggestions. ### Typed Document–Code Relationships Topic distinguishes among: - `supported_by`: code behavior supports the document’s claim - `contradicted_by`: code behavior conflicts with the document - `mentions`: both refer to the same area, but support or contradiction is unconfirmed Embedding search first narrows possible matches, after which semantic verification is performed. Low-confidence or failed checks do not create relationships; an absent relationship means “not yet verified,” not necessarily “unrelated.” ## Incremental Verification and Change Detection - Stable identifiers and content hashes allow unchanged units to reuse previous extraction and relationship results. - Deleted sources trigger cleanup of dependent relationships. - Code anchors store the validating commit and span hash. - If an anchor disappears, it is marked orphaned. - If the span remains unchanged, semantic verification can be skipped. - If the span changes, faithfulness must be checked again. - Rule-based checks happen before LLM calls, reducing cost and limiting nondeterministic reasoning to cases that require it. Topic’s practical recommendation is to treat trustworthy internal context as a managed system rather than a search result. Preserve source structure, keep evidence attached to every claim, use automation for deterministic work, and route ambiguous organizational judgments to people.
Read original(opens in new tab)