Check sitemap health

A sitemap is a promise to Google about which URLs matter. Broken promises (URLs that redirect or 404, lastmod dates that never change, whole sections missing) make Google trust the file less, and the file is often the only way large or new sections get discovered at all.

Sequel supplies Search Console's view of each sitemap. Reading the XML itself and checking its URLs uses your own browsing.

Steps

1. List what Search Console knows. google_search_console.list_sitemaps. For each: type, whether it is an index, pending state, last submitted, last downloaded, warning and error counts, submitted and indexed totals. If any row is a sitemap index, call again with sitemapIndex set to its path to see the children. Use google_search_console.get_sitemap on any sitemap with errors for the detail.

2. Read the XML. Fetch each sitemap (or the index and its children) with your browsing. Count the URLs, note the lastmod values, and compare the URL count to the submitted total Search Console reports.

3. Check a sample of listed URLs. Thirty to fifty URLs spread across the file. For each: does it return 200, is its canonical itself, is it free of noindex. Any redirect, error or noindex in a sitemap is a finding. Use sequel_workbench for the loop if you can execute code.

4. Check completeness against traffic. google_search_console.query with dimensions: ["page"], 90 days ending 3 days ago, rowLimit: 2000. Pages with clicks that are absent from every sitemap are the pages the sitemap forgot.

5. Check freshness. A last downloaded date weeks old on a sitemap that changes often, a lastmod that is the same on every URL, or a lastmod in the future all tell Google the dates are meaningless. Note which applies.

6. Report. Per sitemap: submitted, indexed, gap, errors, warnings, last downloaded, URL count from the XML. Then the findings: listed URLs that redirect or error (with the final URL), listed URLs carrying noindex, pages with traffic missing from the sitemap, and freshness problems. Fixes are specific: remove or replace listed URLs, add the missing pages, generate real lastmod values. Continue with index-coverage-review if the indexed gap is large.

Gotchas

• Indexed counts in the sitemap report lag and are per sitemap. A URL listed in two sitemaps is counted twice.
• Sitemaps should list only canonical, indexable, 200 URLs. That is the whole rule; every finding above is a violation of it.
• lastmod only helps when it is honest. Google has said it ignores the field on sites that set it to the current time on every URL.
• Limits are 50,000 URLs and 50 MB uncompressed per file. A file near either should be split.
• Search Console lags 2 to 3 days. End the window 3 days back.