A crawler does not read a website the way a person does. There is a common assumption that when a tool reports on a page, it has looked at that page the way a browser paints it on screen, seeing headings, images and buttons in their final arranged state. What actually happens is closer to a rapid series of transactions: a program opens a connection, asks a server for a specific file, receives a stream of bytes and a set of labels describing that stream, then decides what to do next. Every figure that later appears in an audit is a summary of thousands of these small exchanges.
What Happens When a Crawler Sends Its First Request
The process begins with a single seed URL, usually a homepage. The crawler resolves the domain name to an IP address, opens a TCP connection to the server, and if the address uses HTTPS, negotiates a TLS handshake to encrypt the exchange. Only then does it send an HTTP request, a short block of text naming the method (almost always GET), the path being requested, the host, and a user-agent string identifying the crawler.
That user-agent matters more than people expect. Some servers respond differently depending on who is asking, serving a lighter page to known bots or blocking unfamiliar ones entirely. The request also carries headers announcing which content types and encodings the crawler will accept, which is why a fetcher can receive a gzip-compressed response it must decompress before it can read a word of the HTML.
How the Fetcher Reads Response Headers and Status Codes
The server’s reply arrives in two parts. First comes the status line and headers, then the body. Before touching the body at all, the fetcher inspects the status code. A 200 means proceed. A 301 or 302 means the real content lives elsewhere, so the crawler records the redirect target and, up to a sensible limit, follows the chain. A 404 or 410 marks the URL as gone. A 500 signals the server itself failed, which is treated very differently from a page that simply does not exist.
The headers carry more than the status. A canonical hint, a content-type declaration, cache directives, and an X-Robots-Tag can all appear here, out of sight in the HTTP response rather than in the visible page. A crawler that only looked at page markup would miss a noindex instruction delivered in a header. This is one reason two tools can disagree: one reads the header, another does not.
Turning Raw HTML Into a Parsed Link Graph
Once a 200 response with an HTML body arrives, the crawler parses it. Parsing means walking through the raw markup and building a structured tree of elements. From that tree the fetcher extracts the pieces an audit cares about: the title, meta description, heading tags, canonical link, meta robots directive, and above all the anchor tags.
Each href becomes a candidate for the next round of requests. Relative paths are resolved against the current URL, fragments are stripped, and duplicates collapse. The result is a growing graph, with pages as nodes and links as edges. The crawler adds new URLs to a queue, respects a politeness delay so it does not hammer the server, and repeats the whole request-response cycle until the queue empties or a configured limit is reached.
Why Robots Rules and Rendering Change What Gets Seen
Before requesting anything, a well-behaved crawler fetches robots.txt and checks each candidate URL against its rules. A disallowed path is never requested, so it never generates the response data an audit would otherwise collect. That absence is meaningful, not an error.
Rendering complicates matters further. Some crawlers stop at the raw HTML; others execute JavaScript in a headless browser to see links and content that only appear after scripts run. The two approaches can produce entirely different link graphs for the same site. Because a sitemap represents URLs the site owner explicitly wants discovered, many practitioners cross-check it against the crawl using a service that can validate an XML sitemap and extract its URLs, a tool offered among the free utilities at RapidIndexChecker, and reconcile that list against what the fetcher actually reached.
From Collected Signals to the Numbers in Your Report
By the time the queue empties, the crawler holds a record for every URL it touched: the status code, response time, headers, extracted metadata, and inbound and outbound links. The report you read is an aggregation of that record. A count of broken links is the number of edges pointing at nodes that returned an error. An orphan-page warning is a node the sitemap claims exists but no internal link reaches. A duplicate-title alert is a grouping of nodes sharing the same parsed title text.
None of these numbers are magic. Each traces back to a request that succeeded or failed and a response that was read a particular way. Understanding that chain is worth revisiting whenever a report surprises you. Sites change constantly, so a crawl is only a snapshot; schedule a fresh one on a regular cadence rather than trusting a picture taken months ago.