To debug a page that is not indexed, run it through Search Console's URL Inspection API and read the coverage state, because that one field names most of the eight possible causes. "URL is unknown to Google" means no sitemap entry and no internal links. "Discovered, currently not indexed" means Google has not bothered to crawl it, which is a priority problem. "Crawled, currently not indexed" means Google fetched it and chose not to index it, usually thin or duplicate content. A robots.txt state of disallowed means a blocking rule. An indexing state of blocked by meta tag or HTTP header means a noindex. A fetch state of not found, server error, soft 404 or redirect error means the page was broken at crawl time. A Google canonical that differs from the declared one means Google picked another URL. Then confirm against the live page with your own browser, check whether the URL ever earned impressions, and stop at one diagnosis and one fix. An AI agent with a Search Console connection does all of this in a single pass.
TL;DR
| Question | Answer |
|---|---|
| First call | google_search_console.inspect_url, up to ten URLs per call |
| The field that matters | Coverage state, then robots state, indexing state, fetch state and canonical |
| Confirm against | The live page: status code, canonical tag, meta robots, X-Robots-Tag, content in raw HTML |
| Then check | Whether the URL ever had impressions in the last 90 days |
| Cannot do | Request indexing through the API. Verify on the next crawl instead |
| Playbook | page-not-indexed-debug |
"Why isn't this page indexed" has about eight possible answers, and the URL Inspection API gives most of them directly. The job is to read the inspection result correctly, confirm it against the live page, and stop at one diagnosis.
This is the procedure Sequel's page-not-indexed-debug playbook hands to an agent. It is written for a handful of URLs. For a whole section of the site, index-coverage-review samples instead.
What the URL Inspection API tells you
The URL Inspection API returns Google's view of one URL: the verdict, the coverage state, the indexing state, the robots.txt state, the page fetch state, the last crawl time, the canonical Google chose against the one the page declares, which sitemaps list it, which pages link to it, and any rich result issues. It is the same data as the URL Inspection tool in the Search Console UI, available programmatically and rate-limited per property.
Read every field. Most wrong diagnoses come from reading the coverage state alone and skipping the canonical.
Step 1: confirm the URL belongs to the property
The URL must sit under the property the connection is pinned to. Sites with both a domain property and a URL-prefix property, or www and non-www, trip this constantly. Run google_search_console.list_sites and check.
Inspecting https://www.example.com/page against a property for https://example.com/ returns an error or an unrelated result. Property mismatches produce confident nonsense.
Step 2: inspect the URL
google_search_console.inspect_url with the URL, or up to ten if several pages are affected. The prompt:
Why isn't https://example.com/guides/new-guide indexed?
The agent returns the inspection columns and moves straight to diagnosis.
Step 3: diagnose from the coverage state
| Inspection result | Cause | Fix |
|---|---|---|
| "URL is unknown to Google" | Never seen. Not in a sitemap, no referring URLs. | Add to the sitemap, link it internally. Continue with sitemap-health and internal-link-audit |
| "Discovered, currently not indexed" | Known but not crawled. Too few internal links or a very large site. | Internal links from pages with traffic, fresh sitemap entry |
| "Crawled, currently not indexed" | Fetched and rejected. Thin or near-duplicate content, or a low-value page type. | Compare with indexed siblings; improve or consolidate |
| Robots state DISALLOWED or fetch state BLOCKED_ROBOTS_TXT | A robots.txt rule matches. | Fetch /robots.txt and find the rule |
| Indexing state BLOCKED_BY_META_TAG or BLOCKED_BY_HTTP_HEADER | A noindex. | Fetch the page and confirm. Ask whether it is intended |
| Fetch state NOT_FOUND, SERVER_ERROR, SOFT_404, REDIRECT_ERROR, ACCESS_DENIED | The page was broken at crawl time. | Fetch it now. If fine, the error was transient and a recrawl fixes it |
| Google canonical differs from user canonical | Google picked another URL as the real one. | Inspect that URL too. Either the canonical tag is wrong or the pages are near-duplicates and Google is right |
| "Alternate page with proper canonical tag" | Working as intended. | Nothing, unless the canonical is wrong |
The third row is the one people argue about most. One r/TechSEO thread frames it as an authority problem: a strong site gets thin pages indexed, a weak one does not. That is true and unhelpful. The actionable question is how this page differs from the indexed pages next to it.
Step 4: check the live page
Inspection reports the indexed version, not the live one. Fetch the page with your own browser and record:
- Status code
- The canonical tag
- Meta robots and the
X-Robots-Tagheader - Whether the main content is present in the raw HTML, or only after JavaScript runs
Compare with the last crawl time. A fix deployed after that time has not been seen yet, and the inspection result will keep describing the old page until the next crawl.
Step 5: check whether it ever earned impressions
google_search_console.query with dimensions: ["page", "date"] over 90 days ending three days ago and a page filter equal to the URL.
A page that had impressions and then stopped was indexed and dropped. A page with none may never have been. These are different problems with different fixes, and the coverage state alone does not distinguish them.
Step 6: report one diagnosis per URL
Per URL: the coverage state, the diagnosis in one sentence, the evidence columns that support it, the single fix, and how to verify it. Verification means re-inspecting after the next crawl, which cannot be scheduled and may take days to weeks. Do not promise when.
What the API cannot do
You cannot request indexing through the API. Google's Indexing API accepts only JobPosting and BroadcastEvent pages, and the Request Indexing button in the Search Console UI has no programmatic equivalent. An agent can diagnose, fix and verify. The crawl itself is Google's schedule.
Gotchas
- Not indexed can be correct. Tag pages, filtered listings, thin variants and paginated archives are often better left out. Say when the state looks intended.
- A canonical pointing at another page the site owns is not a bug by itself. It is a decision. The question is whether it is the right one.
- The API is quota-limited per property. A handful of URLs is the right scale. For a whole section, sample with index-coverage-review.
- JavaScript-rendered content is invisible to a raw fetch. If the HTML looks thin and the page renders client-side, that is a finding in itself.
Running it with an agent
The inspection takes seconds. Reading the result correctly, opening the live page, checking the impression history and reaching one diagnosis is where the time goes, and where a written procedure beats memory.
Sequel exposes URL Inspection, sitemaps and the performance report from Search Console over MCP, read-only, and the playbook so the agent runs the steps in order. Install both:
npm install -g @sequelsh/cli@latest
sequel install claude-codeConnect Search Console once (the Claude Code guide takes five minutes, and the ChatGPT guide covers that route), then ask the question with the URL. The agent reads page-not-indexed-debug through sequel_read_skill, inspects, fetches, checks history, and returns the diagnosis with the evidence. When the pattern turns out to be site-wide, it hands off to sitemap-health or technical-crawl-audit.
Eight causes, one field, one fix. Get started free and stop guessing which one it is.