What Is Link Extraction? How to Extract URLs From a Page

Link extraction in more depth

Links are how a web crawler finds its way around a site, so link extraction runs on every page it fetches. The job sounds trivial, find the href attributes, but most links in real HTML are relative (../guide, ?page=2, //cdn.example.com/x) and only make sense once resolved against the page’s URL.

The output is usually a list of absolute URLs, often tagged as internal (same host) or external, along with the anchor text, which describes the target and is useful for ranking and for understanding a site.

How link extraction works

Parse the HTML (after JavaScript rendering if links are added by scripts).
Collect href from <a> and <area> elements; optionally <link rel="next">, canonical links and iframes.
Determine the base URL: the page URL, unless a <base href> element overrides it.
Resolve each href against the base according to RFC 3986.
Drop non-navigational schemes such as mailto:, tel: and javascript:, and strip fragments.
Normalize and deduplicate.
Classify each link as internal or external and apply scope filters.

Example in Node

const html = await (await fetch('https://example.com/docs/')).text();
const base = 'https://example.com/docs/';
const links = new Set();
for (const [, href] of html.matchAll(/<a[^>]+href="([^"]+)"/g)) {
  if (/^(mailto|tel|javascript):/i.test(href)) continue;
  const url = new URL(href, base);
  url.hash = '';
  links.add(url.href);
}
const internal = [...links].filter(u => new URL(u).host === new URL(base).host);

A regex is fine for a quick script; use an HTML parser in production, because real markup has single-quoted, unquoted and entity-encoded attributes. For a one-off list, the URL extractor does this in the browser.

Common pitfalls

Ignoring `<base href>`, which silently changes how every relative link resolves.
Missing script-built links. Menus rendered by JavaScript do not exist in the raw HTML.
Non-anchor navigation. Buttons with click handlers and onclick redirects do not produce extractable links.
Counting duplicates. The same link in header, footer and body is one link, not three.
`rel="nofollow"`. A hint about endorsement for search engines, not a crawl block; decide deliberately whether to follow.

How SarvCrawl handles it

Ask for the links format on a scrape to get every link on the rendered page alongside its markdown. The map endpoint lists a whole site’s URLs from sitemaps and links without fetching content. After a crawl, the console draws the results as a link tree, each page placed under the first page that linked to it, with counts of redundant and external links. The free website tree tool shows the same kind of structure for any site.

Frequently asked questions

What is the difference between internal and external links?

Internal links point to the same host as the page; external links point elsewhere. Crawlers usually follow internal links and record external ones without following them.

How do I convert relative links to absolute URLs?

Resolve each href against the page URL, or the base href if the page sets one, using a standards-compliant URL library such as new URL(href, base) in JavaScript or urljoin in Python.

Should link extraction include links in navigation and footers?

For discovery, yes, since navigation is how most pages are reached. For content analysis, you may want only links inside the main content.

What is link extraction?