Use case · AI platforms

Chat with your website, for every customer

Building a chatbot or agent platform? Let your users add their website or docs with one field: create a knowledge base per customer through the API, crawl their site into it, and search it at answer time.

Start free API reference

1
knowledge base per customer, created by API
2
SDKs: Node and Python
1B
free tokens on sign-up, no card

// For chatbot builders, agent platforms and vertical AI products

ONBOARDING ONE CUSTOMER

onboard.js · your backendNode

import { KBClient } from 'sarvcrawl'
const kb = new KBClient({ apiKey: process.env.KB_API_KEY })
// Customer pastes their site URL during onboarding
export async function onboard(customer, siteUrl) {
  const { id } = await kb.create(`cust-${customer.id}`)
  await kb.crawl(id, siteUrl, { limit: 500, exclude_paths: ['/login'] })
  await kb.monitor(id, 'website', { url: siteUrl, schedule: 'daily',
    notify_url: 'https://api.yourapp.example/hooks/kb-updated' })
  return id  // store against the customer
}
// At answer time, only this customer's pages are searched
export const context = (kbId, question) =>
  kb.embedSearchKb(kbId, question, { size: 6 })

Perfect for

Chatbot builders

Offer “train on your website” without running crawlers, browsers, parsers and a vector store yourself.

Agent platforms

Give each customer’s agents a knowledge base they can search by keyword or by meaning.

Vertical AI products

Ingest each customer’s site and documents, from PDFs to scanned forms, into the same pipeline.

Teams that need data separation

Each customer’s pages live in their own knowledge base, searched only by ID.

How it works

01 / 06

Create a KB per customer

Call the API at sign-up to create a knowledge base and store its ID against the customer.

API reference

02 / 06

Crawl their site

Start a crawl with a page limit and excluded paths; pages are rendered, cleaned, chunked and embedded.

Crawl

03 / 06

Accept their files

Forward uploads — PDF, Office, EPUB, images, audio — to the same knowledge base; scans get OCR.

Parse

04 / 06

Search at answer time

Hybrid semantic search returns ranked chunks with titles and source URLs for your LLM to cite. Search is on paid plans.

Semantic search

05 / 06

Keep each KB fresh

Attach a monitor per customer site, with a webhook to your backend when pages change.

Monitor

06 / 06

See what each customer has

Jobs, crawl trees, pages, chunks and usage are visible per knowledge base in the console.

Try the playground

How SarvCrawl compares

Feature	SarvCrawl	Build it in-house	Scraping API only	Vector DB only
Crawling with JavaScript rendering
PDF, Office and scan parsing
Chunking and embeddings included
Hybrid keyword and semantic search
Scheduled refresh with webhooks
No crawler or browser fleet to run

Ship “chat with your website” this week

Per-customer knowledge bases through one API. 1 billion free tokens to start.

Start free API reference