4010 articles

Sitemaps

XML sitemaps, robots.txt, crawl budget, and how search engines discover and recrawl URLs.

Articles in this topic

Tool

Sitemap Audit: Find, Parse, Then Sample URLs

A sitemap audit checks that the file resolves, lists only indexable URLs, and a sample returns 200 with a matching canonical. Read the robots.txt line first.

Read article
Tool

Robots Txt Check: See Allowed or Blocked

Run a robots txt check on one URL. See allowed or blocked, the winning Allow or Disallow line, and which bot group decided that path. Then fix the file.

Read article
Tool

Generate Sitemap: Build XML, Then Sample Audit

Learn to generate sitemap XML from live canonical URLs, validate the file, then immediately sample 50–500 pages. Generation is not the finish line in 2026.

Read article
Sitemaps: Guides, Articles & Best Practices