tossJul 30, 2026

LLMs Are Smart, So Why Don’t They Know How Our Company Works? (opens in new tab)

LLMs handle public knowledge well but struggle with company-specific questions because relevant evidence is scattered across documents, code, meetings, and chat—and may be outdated or contradictory. The post argues that this is not merely a search problem: organizations need a shared layer for managing trustworthy context. Topic addresses this by extracting source-aware units, linking concepts and relationships, and verifying their freshness, consistency, and evidentiary support. ## Why Search Alone Is Not Enough - Search retrieves relevant text but cannot determine whether it is current, authoritative, or consistent with other sources. - A retry-policy example might contain: - Documentation saying requests fail immediately - A meeting discussion proposing three retries - Code currently implementing two retries - Agents must still determine: - Whether the meeting produced a final decision - Which source is newer - Whether the code reflects an intentional change or an unfinished implementation - Whether different sources describe the same behavior - Topic provides a shared context layer so humans and LLMs use the same sources, relationships, freshness information, and conflict states. ## Six Dimensions of Trust Rather than compressing trust into one score, Topic evaluates six separate dimensions: - **Granularity:** Whether the context is a meaningful, independently manageable unit - **Faithfulness:** Whether the source actually supports the claim - **Staleness:** Whether the evidence remains valid - **Canonicality:** Whether different names refer to the same entity - **Consistency:** Whether sources are compatible - **Coverage:** Whether important evidence or perspectives are missing Different checks use different methods: rules and hashes for deterministic validation, LLMs for semantic interpretation, and humans for ambiguous or high-impact decisions. ## Ingesting Documents, Chat, and Code Topic normalizes information into a common `ContentUnit` containing source type, unit type, original URI, content, hashes, timestamps, and source-specific metadata. It uses different boundaries for each source rather than splitting everything into fixed-size text chunks. ### Structured Document Sections - Markdown documents are divided by heading hierarchy. - Parent headings are preserved to retain context. - Long sections are split only when necessary. - URLs, document paths, and creation or modification times remain attached to the unit. ### Conversation Threads - Entire messenger threads are treated as the semantic unit, not individual messages. - Summaries preserve: - Technical identifiers such as function names and file paths - Questions, alternatives, and final outcomes - Decisions versus unresolved issues - The system avoids inventing consensus and ignores threads containing only casual conversation. ### Code Symbols and Semantic Cards - Parsers extract functions, classes, file paths, line ranges, imports, and other symbols without using an LLM. - Multiple symbols are then grouped into **code semantic cards** describing business behavior. - Cards retain domain terms, code identifiers, source spans, and the relevant commit SHA. - LLM-generated cards are checked against actual files, line ranges, supporting spans, and duplicate-card patterns. - Cards are an intermediate layer for connecting code to business concepts, not a replacement for the code itself. ## Extracting Concepts and Relationships - Topic extracts concept candidates and supporting evidence from each content unit. - It preserves the relationship between every concept and its original evidence. - Similar names are not automatically merged merely because they appear close in meaning. - Concepts can be consolidated into canonical entities only when sufficient evidence exists. ### Human Review for Ambiguous Terminology - Normalization and embeddings can identify obvious duplicates. - Internal abbreviations and aliases may require organizational knowledge. - Topic creates synonym proposals with their supporting context. - Humans approve or reject ambiguous aliases; rejected proposals are remembered to prevent repeated suggestions. ### Typed Document–Code Relationships Topic distinguishes among: - `supported_by`: code behavior supports the document’s claim - `contradicted_by`: code behavior conflicts with the document - `mentions`: both refer to the same area, but support or contradiction is unconfirmed Embedding search first narrows possible matches, after which semantic verification is performed. Low-confidence or failed checks do not create relationships; an absent relationship means “not yet verified,” not necessarily “unrelated.” ## Incremental Verification and Change Detection - Stable identifiers and content hashes allow unchanged units to reuse previous extraction and relationship results. - Deleted sources trigger cleanup of dependent relationships. - Code anchors store the validating commit and span hash. - If an anchor disappears, it is marked orphaned. - If the span remains unchanged, semantic verification can be skipped. - If the span changes, faithfulness must be checked again. - Rule-based checks happen before LLM calls, reducing cost and limiting nondeterministic reasoning to cases that require it. Topic’s practical recommendation is to treat trustworthy internal context as a managed system rather than a search result. Preserve source structure, keep evidence attached to every claim, use automation for deterministic work, and route ambiguous organizational judgments to people.

context-managementllmhuman-in-the-loopsemantic-search+4

slackApr 13, 2026

Managing context in long-run agentic applications (opens in new tab)

Long-running multi-agent applications cannot rely on unlimited conversation history: model APIs are stateless, and growing context windows eventually reduce quality or hit hard limits. Slack’s security-investigation system addresses this by giving agents complementary, purpose-specific context rather than exposing every agent to the full investigation history. Its three main channels—the Director’s Journal, Critic’s Review, and Critic’s Timeline—preserve coherence while leaving room for independent reasoning. ## The Challenge of Long-Run Coherence - Agent frameworks usually maintain continuity by resending the complete message history with every inference request. - Long investigations can involve hundreds of requests and megabytes of generated output. - Context windows impose both: - A hard limit on how much history can be supplied. - A quality limit, because performance may degrade before the window is completely full. - Multi-agent systems need carefully scoped views: - Too little shared context makes agents disconnected from the investigation. - Too much shared context can suppress creativity and encourage confirmation bias. ## Three Complementary Context Channels Slack uses separate information sources for different purposes: - **Director’s Journal** - Structured working memory for the orchestrating Director. - Records decisions, observations, findings, questions, actions, and hypotheses. - **Critic’s Review** - An annotated report evaluating Expert findings. - Includes credibility scores to distinguish reliable evidence from weaker claims. - **Critic’s Timeline** - A consolidated chronological view of findings. - Also attaches credibility scores, helping agents understand the sequence and evidential strength of events. Together, these channels provide continuity without forcing every agent to process the entire raw conversation. ## The Director’s Journal The Director coordinates the investigation by choosing questions, assigning specialist Experts, assessing progress, and deciding when to stop. The Journal gives it persistent working memory across phases and rounds. - The Director is encouraged to update the Journal frequently with short notes. - Entries can represent: - **Decisions** about investigative strategy - **Observations** about emerging patterns - **Findings** representing confirmed facts - **Questions** that remain unresolved - **Actions** taken or planned - **Hypotheses** about what may be happening - Entries can also include: - Priority levels - Follow-up actions - References to supporting evidence - Investigation phase, round number, and timestamp - The journaling tool itself simply accumulates entries; the agents’ prompts explain how to interpret them. ## Maintaining Alignment Across Agents - The Journal creates a shared narrative around the Director’s evolving plan. - It helps the Director: - Track progress - Identify dead ends - Revise investigative direction - Preserve decisions between rounds - Guide other agents toward a conclusion - Every agent receives the current Journal chronologically, along with instructions describing: - The Director’s role - Each agent’s relationship to the Director - The Journal’s purpose - How its entries should influence their work - This approach keeps specialists anchored to the overall investigation without requiring them to read every prior interaction. ## Example Investigation Context The sample Journal comes from an investigation into an apparent kernel-module-loading alert that turned out to be a false positive. - The Director recorded that: - The event originated from a package-installation hook rather than a direct `modprobe` command. - The host appeared to be a personal development workstation. - Root access was expected in that environment. - The detection rule matched “kmod” in a script path rather than confirming module loading. - The Director identified relevant Expert domains, including: - Endpoint telemetry - Identity and access - Configuration management - User behavior - The Journal captured both the preliminary conclusion and remaining verification tasks, such as checking the parent process chain. The design therefore preserves the reasoning trail while keeping it structured and compact. A practical design for long-running agentic systems is to replace indiscriminate transcript accumulation with multiple, curated context channels. Persistent journals can maintain leadership and continuity, while independent reviews and timelines provide evidence-focused context without overwhelming agents or biasing their reasoning.

context-managementllmai-agentmulti-agent-systems+2