Debug why a page is not indexed

"Why isn't this page indexed" has about eight possible answers, and Search Console's URL Inspection API gives most of them directly. The job is to read the inspection result correctly, confirm it against the live page, and stop at one diagnosis.

Steps

1. Confirm the URL belongs to the connected property. The URL must be under the property this connection is pinned to. If the site has both a domain property and a URL-prefix property, or www and non-www, run google_search_console.list_sites and check. An inspection against the wrong property fails or lies.

2. Inspect the URL. google_search_console.inspect_url with the URL (up to ten per call if several pages are affected). Read every column: verdict, coverage state, indexing state, robots.txt state, page fetch state, last crawl time, the Google canonical against the user canonical, sitemaps, referring URLs, and rich result issues.

3. Diagnose from the coverage state.

   • "URL is unknown to Google": Google has never seen it. Not in any sitemap and no referring URLs is the usual reason. Continue with sitemap-health and internal-link-audit.
   • "Discovered - currently not indexed": Google knows the URL and has not bothered to crawl it. A priority problem: too few internal links, or a very large site. Internal links and a fresh sitemap entry are the levers.
   • "Crawled - currently not indexed": Google fetched it and chose not to index it. The hardest case. Usually thin or near-duplicate content, or a page type Google has decided is low value. Check whether similar pages on the site are indexed and how this one differs.
   • Blocked by robots.txt (robots.txt state DISALLOWED or fetch state BLOCKED_ROBOTS_TXT): fetch /robots.txt yourself and find the matching rule.
   • Excluded by noindex (indexing state BLOCKED_BY_META_TAG or BLOCKED_BY_HTTP_HEADER): fetch the page and confirm the tag or header. Ask whether it is intended before calling it a bug.
   • Fetch errors (NOT_FOUND, SERVER_ERROR, SOFT_404, REDIRECT_ERROR, ACCESS_DENIED): fetch the page now and compare with the last crawl time. If it is fine now, the error was transient and a recrawl fixes it; if not, the page is broken.
   • Google chose a different canonical (google canonical differs from user canonical, or the coverage state says so): Google decided another URL is the real one. Inspect that URL too. Either the site's canonical tag is wrong, or the two pages are near-duplicates and Google is right.
   • "Alternate page with proper canonical tag": working as intended. The canonical is indexed instead. Not a problem unless the canonical is wrong.

4. Check the live page with your own browsing. Status code, the canonical tag, meta robots and X-Robots-Tag, and whether the main content is present in the raw HTML. Compare with what Google saw at the last crawl time; a fix deployed after that time has not been seen yet.

5. Check whether it ever earned impressions. google_search_console.query with dimensions: ["page", "date"] over 90 days ending 3 days ago and a filters entry with dimension: "page", operator: "equals" and the URL. A page that had impressions and then stopped was indexed and dropped; a page with none may never have been.

6. Report. Per URL: the coverage state, the diagnosis in one sentence, the evidence columns that support it, the single fix, and how to verify it: re-inspect after the next crawl, which cannot be scheduled and may take days to weeks. Do not promise when.

Gotchas

• Inspection reports the indexed version, not the live one. If the page changed after the last crawl time, the result describes the old page.
• The API is quota-limited per property. A handful of URLs is the right scale; for a whole section of the site continue with index-coverage-review, which samples.
• Not indexed can be correct. Tag pages, filtered listings, thin variants and paginated archives are often better left out. Say when the state looks intended.
• A canonical pointing at another page the site owns is not a bug by itself. It is a decision; the question is whether it is the right one.
• Property mismatches produce confident nonsense. Inspecting https://www.example.com/page against a property for https://example.com/ returns an error or an unrelated result.