SEO Crawler: Scope, HTTP Behavior, and Safe Sampling

By William Zhu · Cofounder, InfiniSynapse · Published: 2026-09-11 · Last updated: 2026-09-14 · Last verified: 2026-09-14 · Reviewed by the InfiniSynapse Data Team.

Reviewed by: InfiniSynapse Data Team · method review 2026-09-14. robots.txt Disallow list plus public HTML fetches; no request into /api/ or /auth/.

Trust / COI: About · Corrections. SEO Health is a commercial product from this team. We fetched public HTML only; we did not request /api/ or /auth/.

TL;DR

A crawler that ignores Disallow is not faster. It is a different product than the one robots.txt described. Configure an seo crawler so every request has a written reason to be in the queue.

Direct answer: Treat an seo crawler as a polite frontier with a stop rule. On 2026-09-14 this host’s robots file allowed / and disallowed /auth/, /api/, /v1/, /webhooks/, /dashboard, /analyze, and /Volumes/. The desk fetched public HTML only. 1,568 locs in the main map are a declared list, not a discovered graph.

The sample size is one robots file, three declared maps, and eight public requests. That desk cannot establish the full link graph, and it does not prove a later bot would see the same HTML. An seo crawler memo that treats 1,568 <loc> values as pages already fetched is already inflating coverage.

What you will learn

• Which Disallow prefixes this host actually named
• Why the desk never requested /api/ or /auth/
• How a sitemap seed differs from link discovery
• Which hop fields belong in the ledger
• Where this seo crawler page stops and a desktop-spider comparison owns a different budget

What a polite frontier is

Key definition: An seo crawler is a bounded requester that honors robots, records hops, and stops at a declared ceiling. It is not a license to walk authenticated apps. It is not an official Googlebot replacement.

RFC 9309 Robots Exclusion Protocol is the public contract for matching a user-agent to Allow and Disallow lines. RFC 9112 HTTP/1.1 is the matching contract for status lines and hop semantics. Neither specification recommends this product.

People open an seo crawler page when a seed list is about to become an uncontrolled scrape. They need a frontier, a robots decision, and a stop.

SEO Health remains the evidence layer for one-URL lights on URLs the frontier already allowed. After the fetch ledger exists, a team may stage those rows in an InfiniSynapse Server API long-task workspace; the API does not request /auth/ paths that robots already disallows.

Identify the agent first

This desk used InfiniSynapse-P2-desk/2026-09-14 for HTML and a pinned CLI for one hub check. Googlebot verification guidance is why we do not pretend that string is Googlebot. Cloudflare verified bots reference is the independent reminder that some edges treat named bots differently.

Disallow is a stop, not a suggestion

The robots body allowed / and named the prefixes above. An seo crawler that still queues /api/ is already off the policy we observed.

Observed robots on 2026-09-14

robots.txt answered 200 (938 bytes) with x-robots-tag: noindex, nofollow on the robots response itself. That header describes the robots file, not the HTML URLs listed in the maps. The body allowed GPTBot, ClaudeBot, PerplexityBot, and ChatGPT-User.

Three Sitemap lines were present. All three maps answered 200. The main file held 1,568 <loc> values. Those locs are seeds if you choose them. They are not a graph the desk walked.

We did not request /api/, /auth/, /v1/, /webhooks/, /dashboard, or /analyze. That omission is the policy, not a missing row. An seo crawler that later adds those paths has left the 2026-09-14 robots body.

Figure: Two series from the same robots desk. Allow is not a crawl. Disallow is not a fetched 403.

How we stayed on public HTML

curl -sS -A 'InfiniSynapse-P2-desk/2026-09-14' \
  https://infinisynapse.com/robots.txt

MDN redirections guide is why first status and Location are stored as separate fields. OWASP Web Security Testing Guide is why authenticated and administrative paths stay out of a public seo crawler queue. Neither is a testimonial.

As of 2026-09-14 those Disallow lines matched this desk. A later deploy can change robots. Refresh the policy before a long run.

The invented path was the only follow in this desk. It 301’d into a blog 404 whose title was the homepage title. That hop belongs in the ledger. It is not a reason to start walking /api/. A polite seo crawler records the miss and stays on public HTML.

A four-rule fetch ledger

Establish authority

Confirm host permission, name the user-agent, retrieve robots, and list allowed hostnames. Exclude signed and administrative locations. This first stage is unique to an seo crawler and sets what later evidence can support.

Configure the frontier

Choose sitemap seeds, allowed schemes, query handling, depth, deduplication key, concurrency, delay, maximum bytes, and a URL ceiling. Write every parameter before the first request so the run cannot be tuned after an inconvenient 404.

Persist every hop

Store requested URL, robots decision, enqueue source, each redirect, final URL, status, MIME type, bytes, and timing. Keep parse failures distinct from network failures. A final 200 must not erase a 308 source.

Stop and export the remainder

Stop at the URL ceiling, host budget, byte limit, or time budget. Export unvisited frontier counts. A partial run that hides the remainder is already claiming a graph it did not walk.

How a loc list differs from a graph

1,568 locs are a declared list. Link discovery is a different provenance class. An seo crawler that labels sitemap seeds as “found pages” is mixing those classes.

The complete SEO website audit guide owns the three-map inventory. This seo crawler page owns the robots decision and the hop ledger.

The seo site audit page owns template names before the queue grows. Do not expand into 1,131 blog locs because the map listed them.

The Screaming Frog alternative page owns the budget question for this host: 1,568–1,738 public locs do not require a 500k desktop crawl. We did not run Frog.

A later operator who follows every in-page link without a ceiling will leave the declared list and may still miss orphans. Orphan discovery is a different coverage claim. Say so.

Provenance classes on this host

Sitemap seeds, discovered links, and redirect hops are different provenance classes. The 1,568 locs are seeds if you choose them. The invented path was a hop we followed once. We did not walk in-page links. An seo crawler ledger that stores one “found” flag for all three classes will later be unable to explain the queue.

Write the enqueue source beside each URL. robots.txt named three Sitemap lines. That is how the seeds entered. It is not how the invented miss entered. Keep those stories apart.

Fonts and media prefixes were also Disallowed. This desk did not fetch .woff2 files. An seo crawler that still downloads font binaries is wasting the byte budget on assets the policy already excluded.

The robots response itself carried x-robots-tag: noindex, nofollow. That header is about the robots file. It is not a reason to skip HTML the body allowed. Read the body, then the header, then the queue.

Budgets before launch

Set separate limits for queued URLs, fetched URLs, redirects per request, bytes per response, wall-clock time, and concurrent requests per host. Then define what happens when a limit is reached: reject the new URL, preserve it as unvisited, or pause the host. A single “maximum pages” value cannot explain whether an seo crawler stopped because of duplicates, oversized files, or server pressure.

Canonicalize the queue key conservatively. Lowercasing a path or dropping every query parameter can collapse resources the server treats differently. Store the original discovered address beside the normalized key.

If traps appear—calendar paths, faceted combinations, session identifiers—update an explicit exclusion rule, save the rejected count, and begin a new versioned run. Do not silently change scope midway.

We did not run audit --pages 50. That CLI range is a later sample size, not a graph walk. An seo crawler memo that quotes 50 from help text is still looking at documentation.

The hub CLI on the tool URL exited 0 with one warning. Process 0 is not a reason to enlarge the frontier. Lights come after the URL is allowed and has stopped moving.

Two people who repeat the same eight public requests should see the same first codes unless a deploy landed. That repeatability is the point of a bounded seo crawler desk. Unbounded scrapes are a different job.

Failure modes that fake discovery

Using a browser user-agent without identification. This desk named its UA. Keep yours.

Treating 1,568 locs as already crawled. They are seeds until fetched.

Ignoring robots during a long run. Refresh the policy. Record the decision per URL.

Downloading unlimited bodies. Set a byte ceiling. The homepage HTML was 91,875 bytes. That is not a license for multi-megabyte traps.

Discarding intermediate hops. /en was 308. The invented path was 301 then 404. Keep both.

Calling an unfinished frontier complete. Export the unvisited count. An seo crawler that hides remainder is already a story.

A later seo crawler that starts from in-page links and never reads robots is already off this host’s policy. Read the 938-byte file first. Then decide whether a loc is even eligible.

Concurrency and delay belong in the same note as the UA. This desk was sequential curl. It is not a load test. Do not scale this seo crawler desk into parallel fetches against production without a written host budget.

429 and 5xx are pressure signals. We did not observe them on these eight public requests. Absence in a tiny desk is not a capacity proof. Keep that limit next to the ledger.

Cross-host hops are graph findings even when the last page returns 200. This desk stayed on infinisynapse.com. An seo crawler that follows a Location off-host has left the declared frontier unless the policy said so.

Store MIME type and bytes. The homepage HTML was 91,875 bytes. The hub tool HTML was 95,445 bytes. Those sizes help a later operator spot an empty shell without claiming a full render.

If you need a desktop spider comparison, open the sibling. This page does not replace Frog. It records a polite public desk.

Cluster guides for this pillar

This cluster page returns to the complete SEO website audit guide for loc counts. Template mapping and desktop-spider scope stay on siblings. Keep the seo crawler ledger in its own file so a later Frog comparison cannot overwrite the Disallow list.

Frequently Asked Questions

What boundary should an seo crawler enforce?

Define allowed hosts, paths, schemes, parameters, content types, depth, concurrency, bytes, time, and URL count before starting. On this host the Disallow list already removes /auth/ and /api/. Stop when any hard budget is reached.

Should every redirect be followed?

Follow only within the hop and host policy. This desk followed the invented path once and stopped at a blog 404. A final 200 must not erase a problematic source chain.

How should robots.txt change the queue?

Evaluate the declared user-agent before fetching a queued URL and retain the decision. For long jobs, define whether and when policy is refreshed. This file answered 200 and named three Sitemap lines.

Does a sitemap seed discover orphans?

No. Sitemap seeds examine declared URLs. Orphan discovery requires permitted link traversal or another inventory. After the ledger exists, a credited memo may be opened once in the InfiniSynapse web app.

Did this desk crawl /api/?

No. An seo crawler on this host must honor the observed Disallow list. We fetched public HTML only. Do not invent an authenticated crawl we did not run.

Conclusion

Use an seo crawler when the job is to keep a frontier, a robots decision, and a hop ledger you can explain. The 2026-09-14 desk shows Allow /, a named Disallow list, and 1,568 locs that remain a list until fetched. Close the file when the unvisited remainder and the paths you refused are still visible. A polite ledger is more useful than an unsupported graph claim.