Your documents changed. Your assistant didn't.
Your company has an AI assistant that answers questions from internal policies, product documentation, or customer support materials. Someone updates an important document, but days later the assistant continues quoting the old information. Team members begin questioning whether they can trust its answers.
Imagine a hypothetical customer support assistant that answers questions about subscription policies. The company changes its cancellation procedure and updates the official policy document. However, the assistant continues recommending the previous procedure. The original file is correct, so why is the AI still using outdated information? The answer may be somewhere between document storage and the information the assistant actually searches.
Updating a file does not necessarily update an AI knowledge base
Many document-based assistants use retrieval-augmented generation (RAG). Rather than relying entirely on what an AI model already knows, the system searches an indexed collection of company information before generating an answer. Documents may pass through an ingestion pipeline that extracts text, splits it into smaller chunks, and creates searchable representations called embeddings. If the ingestion process is not rerun correctly, the assistant may continue searching an older version, even though the source file has changed.
A reliable investigation traces one changed fact from its approved source document through the ingestion pipeline, document version, source timestamp, chunk ID, search index, and retrieval results. Developers should also inspect retrieval filters, cached answers, and whether removed documents have been deleted from all relevant downstream indexes. This is known as deletion propagation.
An assistant should ideally provide citations or source references that allow its answer to be traced. Refreshing the knowledge base must also preserve document permissions so users cannot retrieve information they are not authorized to see.
When should you bring in an engineer?
If the assistant supports customer service, employee decisions, or operational procedures, knowledge freshness is a product reliability issue. Repeatedly telling the AI to "use the latest information" will not repair a broken ingestion or retrieval pipeline.
DitchNow helps founders and small teams build and maintain AI assistants that retrieve information from their own documents and business systems.
Discuss your AI knowledge assistant with DitchNow.
Five prompts to investigate outdated answers
Before you begin: Use these prompts one at a time in the same project conversation. Give the assistant access only to approved project materials, use synthetic test data, and review its findings before approving changes.
1. Map
Act as an engineer specializing in document-based AI assistants. Inspect my project's accessible RAG and knowledge-retrieval architecture. Map the journey from source documents through ingestion, text extraction, chunking, embeddings where used, indexing, retrieval filters, caching, and final answer generation. Identify document version fields, source timestamps, chunk IDs, citations, and permission checks. Reference actual implementation files and clearly mark inaccessible systems. Do not modify data or code.
2. Audit
Perform a read-only knowledge-freshness audit. Using approved sample documents, trace this changed fact: [DESCRIBE CHANGE]. Determine whether the latest source version was detected, processed, indexed, retrieved, and cited. Check incremental indexing, failed ingestion jobs, stale embeddings, duplicate chunks, cache invalidation, retrieval filters, and deletion propagation. Include at least one removed-content scenario. Separate confirmed problems from hypotheses and provide evidence.
3. Plan
Create an implementation plan for keeping the assistant synchronized with approved source material. Consider document version tracking, incremental ingestion, failed-job recovery, removal of superseded chunks, deletion propagation, retrieval cache invalidation, and source citations. Include access-control preservation, auditability, acceptance criteria, and methods for verifying that updated information reaches the correct users.
4. Challenge
Perform a second-pass self-review of your proposed solution and challenge its assumptions. Design isolated tests using synthetic documents with changed, unchanged, removed, and access-restricted content. Trace one fact through each processing stage and verify that older versions are not incorrectly retrieved. Identify possible race conditions, cache inconsistencies, and unresolved permission risks. Do not modify production indexes.
5. Implement
I approve these bounded changes: [APPROVED SCOPE]. Implement them only in an isolated environment with synthetic documents. Run controlled ingestion, update, deletion, retrieval, and permission tests. Record document versions, chunk IDs, source timestamps, citations, and whether answers reflect the approved source. Report results, remaining risks, and proposed monitoring. Do not reindex live customer documents or deploy automatically.