What is a knowledge base?

1. 01Collect: crawl websites, upload files, connect sources.
2. 02Convert everything to one clean format, usually markdown.
3. 03Deduplicate: normalise URLs, hash content, filter near-duplicates.
4. 04Chunk and embed each document.
5. 05Index for keyword and vector search.
6. 06Refresh incrementally, re-processing only what changed.

import os
from sarvcrawl import KBClient

client = KBClient(api_key=os.environ["KB_API_KEY"])
kb = client.create_kb("support-docs")

client.crawl(kb.id, "https://docs.example.com",
             max_depth=3, limit=500, exclude_paths=["/blog/"])
# ...wait for the job, then:
res = client.embed_search_kb(kb.id, q="how do I reset my password", size=5)

The first five steps happen once per document; the last one never stops. A docs site changes weekly, and a knowledge base that rewrites everything nightly is slow and expensive, while one that never refreshes drifts. Hashing each page's extracted text and re-processing only pages whose hash changed turns refresh from a job proportional to the corpus into one proportional to what changed.

• Ingesting everything. Tag archives, changelogs and old doc versions crowd out the pages that matter. Three versions of every page means two wrong answers per question.
• Building once. A knowledge base that is never refreshed answers confidently about last year's product.
• Skipping the negative test. Ask something you know is not in it. If it answers anyway, add a confidence check.
• Ignoring documents. Real corpora are half web pages and half PDFs and Office files.
• Unclear data handling. Know where the content is stored, who can query it and whether a vendor trains on it.
• No provenance. Every chunk should carry its source URL or file, so answers can cite it and a wrong answer can be traced back to the document that caused it.

In SarvCrawl the knowledge base is the core object. You fill it by crawling or scraping sites and uploading PDF, Office, EPUB, image and audio files; every item is converted to markdown, chunked on headings, embedded and indexed. You query it with keyword, semantic or hybrid search (paid plans) over REST, the sarvcrawl Node and Python SDK, the sarv-kb CLI, an n8n node, or 21 MCP tools from Claude Code, Cursor and VS Code.

Content exports as JSONL or CSV, and we do not train models on customer content. Text is metered at roughly 500 tokens per page, and new accounts get 1 billion tokens free with no card. A worked example: build a RAG knowledge base from a documentation site.