Sitemap URL: Find the File When Crawlers Cannot

By the SEO Health Team · Last updated: 2026-08-16 · We build the SEO Health checker at aimeetup.center. Methods below come from the product’s page audits, sitemap samples, and GSC export math — not from a claimed Google score.

Table of Contents

  1. TL;DR
  2. What a location problem looks like
  3. Common paths that usually work
  4. The robots Sitemap line
  5. When the file cannot be found
  6. Redirects and IRI mistakes
  7. How this differs from generation
  8. Tool landscape
  9. Implementation steps
  10. Failure modes
  11. Frequently Asked Questions
  12. Conclusion

TL;DR

Direct answer: A sitemap url is the absolute address of the XML file or sitemap index crawlers fetch. Declare it in robots.txt, keep it on a 200, and fix redirects before you audit. The 2026 finish is a parse plus a 50–500 page sample, not a scavenger hunt.

What you'll learn

  • A 44-word definition of sitemap url you can quote
  • Common paths: /sitemap.xml and /sitemap_index.xml
  • How the robots Sitemap: line must be written
  • Why “couldn’t fetch” is usually a path, redirect, or HTML error page
  • What to do when Search Console cannot find the file

If the host is live, run a sampled site audit. Paste the domain. Let the checker resolve the sitemap url, parse the map, and draw 50–500 pages. A path that 404s makes every later report a guess from homepage links.

What a location problem looks like

Key Definition: A sitemap url is the absolute address of the XML file or sitemap index crawlers fetch. Publish that address in robots.txt, serve a 200 with XML bytes, and resolve hops before any sample. The file path is not a ranking lever.

Illustrative grouped bar chart: sitemap URL declaration errors by robots versus fetch

Quick answer: A sitemap url is the absolute address of the XML file or sitemap index crawlers fetch. Declare it in robots.txt, keep it on a 200, and fix redirects before you audit. The 2026 finish is a parse plus a 50–500 page sample, not a scavenger hunt. Key terms

TermMeaning
XML sitemapThe crawler-parseable inventory of canonical URLs.
Sitemap indexA list of sitemap files used when one file would exceed the size cap.
lastmodThe date the content last changed in a way a crawler should care about.
Stratified sampleA 50–500 URL draw on this page that respects locale and type buckets. (#576)

People type sitemap url when Search Console says “couldn’t fetch,” a plugin wrote /sitemap_index.xml while robots still points at /sitemap.xml, or a human cannot find the file in the browser. Those are location jobs. How you build the XML is Generate Sitemap. What the file is lives in What Is a Sitemap?. This page stays on finding it.

The Sitemap for SEO hub is the method after the path works. Do not sample a 404.

A location ticket usually has one of four shapes: robots points at a path the CMS does not write, the CMS writes a path robots does not declare, the path 301s to HTML, or the path 200s with an HTML 404 template. Write the sitemap url you expect at the top of the ticket. Then request it. If the bytes are not XML, stop talking about lastmod. We see this pattern in sampled audits: the host is live, the homepage ranks, and the declared file is a theme page. The draw then falls back to nav links and misses help, legal, or a locale.

Keep a short sitemap url runbook next to the generator: declared address, HTTP status, Content-Type, first tag, whether the index lists every child. Four facts. If any fact is wrong, the audit is not ready.

Key definition, in practice

The address must be absolute: scheme, host, path. Sitemap: /sitemap.xml is not a legal line. Sitemap: https://example.com/sitemap.xml is. The response must be 200 with an XML body. A 301 to an HTML page is a hop, not a map. Sitemap url work is finished when a crawler can fetch bytes that parse.

Common paths that usually work

Most CMS tools write /sitemap.xml or an index at /sitemap_index.xml. Some write /sitemap.xml.gz. Any of those is fine if you declare the one you actually serve. Rotating /sitemap-2026-08-16.xml as the public address is how last week’s declared path 404s.

W3C’s Addressing overview is the reminder that a URL is a location, not a nickname. Pick one path and keep it. RFC 3987 is the IRI document: if you use non-ASCII in the path, encode it so a crawler can request it. A pretty Unicode path that fails in a fetch is still a miss.

/sitemap.xml versus the index

If both files exist, declare the index. The index lists children. Declaring a child and hiding the index is how a locale never gets fetched. Sitemap url hygiene is “declare the root of the tree.”

Absolute URLs only

Write https://example.com/sitemap.xml, not example.com/sitemap.xml, not //example.com/sitemap.xml in robots. The line is not a browser bar. Search Console’s sitemap report fetch errors are where a missing scheme shows up as “couldn’t fetch.”

The robots Sitemap line

Google’s robots.txt documentation is the current how-to for the Sitemap: line: put the absolute address in robots.txt at the host root. The Wikipedia Robots Exclusion Standard article is the history of that file. The line can appear more than once if you have more than one map. Each line must still be absolute.

A sitemap url that exists but is never declared will still be guessed at common paths. Do not rely on the guess. Declare it. If you use Search Console, submit that same absolute address there too.

If you inherited a host with three historic files, pick one root and retire the others from robots and Search Console. Leaving two live declarations is how a stale child keeps sending 404s while the new index is fine. Sitemap url cleanup is deleting the old line, not adding a fourth.

Where the line must live

robots.txt lives at https://example.com/robots.txt. A Sitemap: line in a subdirectory robots file is the wrong file. A line inside an HTML comment on the homepage is not a declaration. Sitemap url discovery starts at the host root.

When the file cannot be found

Open the declared address in a browser. You want raw XML, not a theme. If you see a 404 template, the plugin wrote a different path. If you see a login wall, the file is not public. If you see a 200 HTML “page not found,” the server is lying. Fix the route before you resubmit.

HTML error pages dressed as XML

A 200 with an HTML body is the most common silent miss. Search Console may say it fetched. A parser will stop. Sitemap url debugging includes Content-Type and the first bytes. If the first tag is <!doctype html>, you published a page.

Search Console fetch errors

“Couldn’t fetch” is usually DNS, robots blocked, 404, 5xx, or a redirect loop. Read the error. Do not rebuild the XML as a substitute. A new file at the same broken sitemap url is the same miss. Google’s fetch-error article above is the taxonomy; your job is to make one address return XML.

Redirects and IRI mistakes

MDN’s Location header is what a 301 sends. If /sitemap.xml redirects to /sitemap.xml/ and that route is HTML, you lost the file. If HTTP redirects to HTTPS, list HTTPS as the sitemap url and stop listing HTTP. Chains waste the fetch and confuse auditors.

Location headers that leak

A Location that points at a staging host, a CDN path you did not declare, or a locale homepage is a leak. Follow it once. When the hop lands on HTML, repair the route instead of sampling the homepage. Do not add the destination homepage to the map as a consolation prize.

Internationalized addresses

If the host or path uses non-ASCII, serve a requestable encoded form. RFC 3987 is the rule. A sitemap url that only works when pasted from a slide deck will fail in robots.txt. Test with a raw HTTP client, not only a browser that helpfully encodes.

How this differs from generation

Generation writes the file. This page finds it. Ops cadence lives in Sitemap Best Practices. Spec legality lives in XML Sitemap Best Practices. Crawl-budget cleanup lives in sitemap SEO optimization. None of those jobs start until the sitemap url returns XML.

Sampled pages still need an on page seo tool for titles. A found file does not write the H1. If you still need to generate sitemap output, do that, then point robots at the real path.

Tool landscape

Browsers show the raw file. Search Console reports fetch errors. Desktop crawlers walk links. SEO Health resolves the sitemap url from robots or common paths, parses the map, and scores a stratified sample of 50–500 pages. We do not walk a million URLs. If the path 404s, we cannot pretend the homepage nav is the site.

JobTypical toolWhat SEO Health does
Open the raw fileBrowserWe fetch the declared address
Explain fetch errorsSearch ConsoleWe fail a file that will not parse
Full link crawlDesktop crawlerOut of scope; we sample
Sampled site auditSEO HealthResolve path → parse → 50–500 draw

Start the sampled site audit after the address returns XML. Do not paste a domain whose robots line 404s and expect cluster reports.

Implementation steps

  1. Open https://your-host/robots.txt. Read every Sitemap: line.
  2. Request each declared sitemap url. You want 200 and XML.
  3. If nothing is declared, try /sitemap.xml and /sitemap_index.xml.
  4. Follow one redirect. If the destination is not XML, fix the route.
  5. Put the working absolute address in robots.txt. Submit the same address in Search Console if you use it.
  6. Confirm Content-Type is XML, not HTML.
  7. Parse. Stratify. Draw 50–500 pages.
  8. Read tech, on-page, compliance, and cluster reports.

When the path is stable, re-run the sampled site audit so the four reports attach to the real file, not to homepage guesses.

After a CDN or HTTPS cutover, re-request the sitemap url from a machine that is not your laptop cache. A local 200 can hide a regional 301 or a certificate miss. If the cutover changed the host, update robots the same hour. A leftover http:// line is a location bug, not a content bug.

Failure modes

These are location failures, not scoring-theater failures.

  1. Wrong path declared. robots points at /sitemap.xml while the CMS writes /sitemap_index.xml. Search Console cannot fetch. The sitemap url you talk about in Slack is not the one crawlers request.
  2. HTML dressed as 200. The route returns a theme 404 with status 200. Parsers stop. Audits that ignore Content-Type call the file “live.”
  3. Redirect to a page. Location sends the crawler to a homepage or a locale root. The sample then looks like one URL. Fix the route; do not sample the homepage and call it the site.

None of these are “the checker was unreliable.” They are path failures. Fix the address, then the inventory.

Frequently Asked Questions

Where should I put the file?

Bottom line: At a stable path on the same host, usually /sitemap.xml or /sitemap_index.xml. Declare that sitemap url in robots.txt. Do not rotate dated filenames as the public address.

Can I use a sitemap on a different host?

Bottom line: sitemap url: Only if you control both hosts and the file lists URLs you are allowed to declare. Most teams should keep the file on the same host as the pages. Cross-host setups fail in surprising ways; do not start there.

What if both /sitemap.xml and /sitemap_index.xml exist?

Bottom line: Declare the index. Make sure the index lists every child. A leftover /sitemap.xml that is stale will confuse humans; do not declare both unless both are true.

Do I need a million-page crawl to find the file?

Bottom line: No. Request robots.txt and the common paths. That is a one-minute job. After the file fetches, sample 50–500 pages. SEO Health will not pretend to be a desktop crawler.

Why does Search Console say it found a different file than I expected?

Bottom line: A plugin may have submitted an old address, or a previous Sitemap: line is still live. Open robots.txt. Align Search Console with the one sitemap url that returns XML. Then remove the stale submission.

Conclusion

A sitemap url is a location job: one absolute address, a robots line, a 200, and XML bytes. Fix redirects and HTML error pages before you argue about titles. Then run the sampled site audit, parse the map, draw 50–500 pages, and continue on Sitemap for SEO for the method around the file.

About the author — SEO Health Team. Reviewer: William Zhu (GitHub). Published and updated 2026-08-16. Credentials appear only here.

Sitemap URL: Find the File When Crawlers Cannot