Glossary · 30 terms
Web data glossary
The vocabulary of collecting web and document content for search and AI: how crawlers find pages, how pages become clean text, and how that text is indexed and retrieved. Each entry gives a short definition, how it works in practice and the mistakes to avoid.
Web crawling
How crawlers discover, fetch and revisit pages, and the rules sites set for them.
Crawl budget
Crawl budget is the number of pages a crawler is able and willing to fetch from a site in a given period. For search engines it is set by how fast the server can respond and how much the engine wants the content; for your own crawls it is the page limit and time you allow.
Read definitionCrawl depth
Crawl depth is the number of link hops between the starting URL and a page. The start page is depth 0, pages it links to are depth 1, and so on; a maximum depth setting stops a crawler from following links beyond that distance.
Read definitionHeadless browser
A headless browser is a real web browser, such as Chromium or Firefox, running without a visible window and controlled by code. It loads pages, runs JavaScript and exposes the rendered page to a script for scraping, testing or screenshots.
Read definitionHTTP 429 and rate limiting
Rate limiting is a server capping how many requests a client may make in a period of time. HTTP 429 Too Many Requests is the status code a server returns when that cap is exceeded, often with a Retry-After header saying when to try again.
Read definitionIncremental crawling
Incremental crawling keeps a previously crawled collection up to date by fetching only new or changed pages instead of recrawling the whole site each time. It relies on change signals such as sitemap dates, HTTP validators and content hashes.
Read definitionJavaScript rendering
JavaScript rendering is running a page’s scripts, usually in a headless browser, so that content built in the browser exists in the HTML a crawler reads. Without it, crawlers see only the initial HTML the server sent.
Read definitionRobots.txt
Robots.txt is a plain text file at the root of a website that tells crawlers which paths they may and may not fetch. It follows the Robots Exclusion Protocol, standardised as RFC 9309, and is a request rather than an access control.
Read definitionURL normalization
URL normalization (or canonicalization) is the process of rewriting URLs into a consistent form so that different spellings of the same address are recognised as one page. Crawlers use it to avoid fetching and storing duplicates.
Read definitionWeb crawler
A web crawler is a program that starts from one or more URLs, downloads each page, extracts the links it finds and follows them to discover more pages. Search engines, archives and AI data pipelines all use crawlers to collect pages at scale.
Read definitionXML sitemap
An XML sitemap, usually served as sitemap.xml, is a file that lists the URLs on a site a crawler should know about, optionally with the date each page last changed. It helps crawlers find pages that internal links alone might miss.
Read definition
Extraction and parsing
Turning fetched HTML and documents into clean text, links and structured data.
HTML to Markdown
HTML to Markdown conversion turns a web page’s markup into Markdown, a lightweight plain-text format that keeps headings, lists, links, tables and code while dropping tags, attributes and styling. It is a common way to prepare web content for LLMs and search.
Read definitionLink extraction
Link extraction is pulling every hyperlink out of a page and resolving each one to an absolute URL. Crawlers use it to discover new pages, and analysts use it to map a site’s structure or audit its outbound links.
Read definitionMain content extraction
Main content extraction, also called boilerplate removal, isolates the primary content of a web page, such as an article or documentation body, and discards repeated page furniture like navigation, headers, footers, sidebars, ads and cookie banners.
Read definitionNear-duplicate detection
Near-duplicate detection finds documents that are almost but not exactly identical, such as a page and its print version or two locale variants that differ only in navigation, so they can be removed or merged before indexing.
Read definitionOCR (optical character recognition)
Optical character recognition (OCR) converts images of text, such as scans, photos and screenshots, into machine-readable characters. It is the only way to get text out of a document that has no text layer.
Read definitionPDF parsing
PDF parsing is extracting text and structure, such as headings, paragraphs, lists and tables, from a PDF file into a usable format like markdown, JSON or plain text.
Read definitionScanned vs text PDFs
A text PDF contains a text layer, the actual characters, which can be extracted exactly. A scanned PDF contains only images of pages, so its text has to be reconstructed with OCR. The two need different parsing and different quality checks.
Read definitionWeb scraping
Web scraping is the automated extraction of content or data from web pages: fetching a page and turning its HTML into something a program can use, such as text, markdown, tables or JSON fields.
Read definitionWeb scraping API
A web scraping API is a hosted service that fetches web pages on your behalf and returns their content in a structured form, such as markdown, HTML or JSON, through an HTTP request. It replaces running your own browsers, queues and parsers.
Read definition
Search and retrieval
Indexing, ranking and finding the right passage once the content is collected.
BM25
BM25 (Okapi Best Matching 25) is the standard keyword relevance formula used by Elasticsearch, OpenSearch and Lucene. It scores a document by how often the query terms appear in it, how rare those terms are across the corpus, and how long the document is.
Read definitionEmbeddings
An embedding is a fixed-length list of numbers that a model produces for a piece of text, placed so that texts with similar meaning end up close together. Embeddings are what make semantic search and most RAG retrieval possible.
Read definitionHybrid search
Hybrid search runs a keyword search (usually BM25) and a vector search over the same corpus and merges the two ranked lists into one, so results match both exact terms and meaning.
Read definitionReciprocal rank fusion
Reciprocal rank fusion (RRF) is a method for merging several ranked result lists into one. Each document scores 1 / (k + rank) from every list it appears in, the scores are summed, and the merged list is sorted by that sum.
Read definitionSemantic search
Semantic search retrieves documents by meaning rather than by matching words: the query and the documents are turned into embeddings, and the results are the documents whose vectors are closest to the query's.
Read definition
AI and LLM data
Preparing web and document content for language models and AI agents.
Document chunking
Document chunking is splitting a document into smaller passages, called chunks, that are embedded and retrieved individually in a search or RAG system. Where the cuts fall decides what retrieval can return.
Read definitionKnowledge base (for AI)
A knowledge base for AI is a searchable collection of documents, split into chunks and indexed by keyword and by embedding, that a chatbot, agent or RAG pipeline queries to answer from your own content.
Read definitionLLM tokens
Tokens are the units a large language model reads and writes: pieces of words, whole words, punctuation or bytes, produced by a tokenizer. Context limits, prices and chunk sizes are all measured in tokens, not characters or words.
Read definitionllms.txt
llms.txt is a proposed markdown file served at the root of a website that lists its most important pages with one-line descriptions, so a language model or coding agent gets a curated map of the site instead of having to work it out from HTML.
Read definitionModel Context Protocol (MCP)
The Model Context Protocol (MCP) is an open protocol, introduced by Anthropic in 2024, that standardises how AI applications connect to external tools and data. A service exposes an MCP server once, and any MCP client, such as Claude Code, Cursor or VS Code, can use it.
Read definitionRetrieval-augmented generation (RAG)
Retrieval-augmented generation (RAG) is a pattern in which a system first retrieves relevant passages from an external corpus and then gives them to a language model as context, so the answer is grounded in your documents rather than only in what the model memorised during training.
Read definition