A central article page sits between a navigation panel and smaller cards with icons and text lines.

Keep three records separate: the pages you intend to have, the pages returned by your search, and the exact pages you inspect through your own account. Those records help you investigate individual pages. Subtracting search results from a sitemap does not produce a count of unindexed pages.

1. Start with the pages you intend to investigate

A public question describes the problem well: a site owner submitted a sitemap with 10,000 links but saw 17 results from a site search. A sitemap is a file that lists page addresses. Those two numbers describe different evidence. The public post records one person’s question from June 4, 2020. It does not verify the site’s index state or show how common the problem is. Public site-owner question — published June 4, 2020, retrieved October 25, 2026; reported counts were not independently verified

The useful next question is: “What evidence do I have for each page I expected Google to index?”

For your own site, follow these steps:

  1. Save the list of page addresses you intend to investigate. A URL is a web address. Use your intended URL list as the starting record.
  2. Save the sitemap separately. Record when you obtained it and what part of the site it covers.
  3. Compare the exact addresses in those two records. Note repeated entries and differences before you count them.
  4. Select exact URLs for owner-access inspection. Start with pages that matter to your decision and pages with unclear evidence.
  5. Add public search observations as another evidence source.

A sitemap entry shows that an address appears in that file. It does not prove that Google has indexed the address. Your intended URL list describes what you want to investigate. It does not establish what Google currently stores.

State your counting unit. “100 sitemap entries” can differ from “100 distinct URL strings” if entries repeat. Neither number is automatically a count of distinct indexed pages.

If you do not own the site or have authorized Search Console access, record that limit. Public search evidence can still show returned addresses. It cannot replace the owner’s private records.

2. Keep inventory, search observations, and owner checks separate

Use three linked tables in your investigation sheet. Link them through exact URL strings while preserving each source value.

2. Keep inventory, search observations, and owner checks separate
RecordWhat it containsWhat it supports
Intended URL and sitemap recordIntended addresses, sitemap entries, source, and record dateWhich addresses you expected or listed
Public search recordExact query, settings, returned addresses, and observation timesWhich results appeared in that collected sample
Owner inspection recordExact inspected URL, reported findings, source, and check timeWhat the owner-access check reported for that URL

Do not replace these tables with one “indexed” column. That would hide the source and strength of each finding.

Also separate these terms:

2. Keep inventory, search observations, and owner checks separate
TermMeaning in this investigation
DiscoveredEvidence says Google knows about an address
CrawledEvidence says Google fetched the page
IndexedEvidence reports that Google included the page in its index
Returned by this queryThe address appeared in the retained search results
Not returned by this queryThe retained results did not contain the address
UnknownThe required evidence is missing or unclear

Knowing about a URL, fetching it, indexing it, and returning it for one query are different states. Record only the state that your evidence supports. If you have no crawl evidence, keep crawl status unknown. A search result does not supply a complete crawl history.

Attach a time to each finding. A search observation and an owner inspection can occur at different times. Keep both times. A later check does not change what an earlier record showed.

3. Save the exact site query and search conditions

Before you search, write down the question and scope. For example, you might investigate whether selected URLs under one documentation path appear in a small public search sample.

Record these items:

  1. The exact query text, including the `site:` operator and path.
  2. The search engine.
  3. The requested country and language.
  4. The result cap, which is the maximum number of results you ask to retain.
  5. The actual observation time, when available.
  6. The collection outcome and any known failure.

Keep the actual returned URLs. A displayed total does not provide an exact list to compare with your intended addresses. Neither a displayed total nor the retained rows establish the complete index count.

Chronometrica’s Google Search Scraper can fit the public-search part of this method. An Actor is a hosted data collection program. The checked public build declares the input fields below and the result fields used in this guide. That is documented fit for collecting search observations. It is not evidence of a successful collection for this guide.

This supplied input requests a small sample:

json
{"country":"US","engines":["google"],"language":"en","maxResultsPerExecutedQuery":3,"queries":["site:developers.google.com/search/docs/"]}
3. Save the exact site query and search conditions

JSON is a text format for named values and lists. Replace the sample values in this order:

  1. Replace the whole `queries` list with one intended site query. The supplied query concerns Google’s documentation path, not your site.
  2. Keep `engines` set to `["google"]` for this method.
  3. Set `country` and `language` to your intended search conditions. Here, `US` requests the United States and `en` requests English.
  4. Keep the first cap small. The supplied value is 3. The checked allowed range for `maxResultsPerExecutedQuery` is 1 to 100.
  5. Save the complete input beside the collection record.

The supplied input passed a static check against the public default-build rules. A static check checks the input format and allowed values without running the Actor. No Actor collection occurred for this guide. The check does not prove result availability, complete coverage, or successful execution. Public default build 1.0.15 — checked October 25, 2026; input and output rules inspected, sample not run

The public Google Search Results Scraper Task is another starting point. A Task is a saved input preset. Its public page and exact Task ID were verified. Its current saved settings were not verified. Inspect and replace those settings before use. Do not assume they match the supplied input. Google Search Results Scraper Task — checked October 25, 2026; public presence verified, saved settings and runtime unverified

4. Preserve the search rows and build the investigation sheet

The following collection steps are proposed reader actions. They are not results from a run performed for this article.

After collection, check whether it finished, failed, or stopped with an unclear outcome. Save the available result rows and collection evidence. If collection fails, record a collection failure. Do not report that Google returned no matching pages.

Retain these documented result fields:

4. Preserve the search rows and build the investigation sheet
FieldWhat to retain
`engine`Search engine attached to the result
`executedQuery`Query attached to the result
`canonicalUrl`Returned result address string
`resultDomain`Returned domain, the site-name part of an address
`rank`Running position among supported result blocks parsed across fetched pages
`title`Search-result title
`snippet`Search-result text, including a supplied null value
`isSponsored`Flag that identifies whether the result is sponsored
`observedAt`Recorded observation time

These fields are declared in the checked build. Their declaration does not prove that a live collection supplied every value. Keep the returned address unchanged. The field name `canonicalUrl` alone does not prove that someone checked the destination page’s preferred address. A snippet is search-result text, not a fresh copy of the destination page. Public default build 1.0.15 — checked October 25, 2026; declared result fields, no live output verification

Build a separate investigation table with one row for each intended URL:

4. Preserve the search rows and build the investigation sheet
Investigation fieldWhat you record
Intended URLExact address you want to investigate
Inventory source and dateWhere the address came from and when you saved it
Sitemap membershipListed, not listed in the checked file, or unknown
Site-query referenceLink or reference to the saved query and its settings
Search stateReturned, not returned in the retained sample, failed, or unresolved
Matching result referencesReferences to actual search rows
Ordinary-query checkExact query, observed result, and time
Owner inspectionExact inspected URL, reported findings, and check time
Address relationship evidenceAny observed preferred-address or redirect finding
Access evidenceAny observed page-access finding and its time
Next actionThe specific check needed to resolve the question

Preserve missing values. A missing rank is unknown, not rank zero. A missing timestamp is unknown, not the time you opened the sheet. A missing snippet is not proof that a page has no text. Zero returned rows is a supported count only when the collection evidence establishes zero within that request’s scope.

If you need help moving retained results into a spreadsheet, use the separate Google search export guide. That guide covers export checks. This investigation uses those rows to assess individual URLs.

5. Check selected exact URLs through the right source

Use an ordinary search to look for selected exact URLs as a second public observation. Save the query and returned address. Compare exact strings before deciding that a result matches your intended page.

If the address differs, keep both values. Do not silently treat a different path, address parameter, or redirect as the same page. A redirect sends a request from one address to another. If you cannot establish the relationship, mark the match unresolved.

An ordinary-query absence has the same basic limit as a site-query absence: it records what was not returned by that query. It does not establish exclusion from the entire index.

For an owned site, use Google Search Console’s URL Inspection tool through your own authorized access. URL Inspection supplies an owner-access check for a specific address. Follow the current official tool guidance rather than relying on a fixed sequence of screen controls.

Use this sequence for each selected URL:

  1. Inspect the exact intended address.
  2. Save the reported findings and the check time.
  3. Keep any different address identified by the check as a separate value.
  4. Record any available discovery, crawl, or index evidence separately.
  5. Compare that evidence with the public search observation.
  6. Select the next check from the reported evidence.

If the evidence identifies a canonical URL, that means a preferred page address. Preserve that address beside the inspected address. Do not rewrite the original record. If the evidence identifies a redirect or access problem, retain the finding and consult current official guidance for that condition.

Do not choose a technical repair from a search absence alone. A missing result does not identify a cause. Likewise, an owner check for one URL does not establish the state of every URL in the sitemap.

Keep owner-only records clearly labeled. The Actor’s public result rows are separate from private Search Console information. This guide does not establish that the Actor can read your Search Console account.

If your question later becomes whether a result position changed, use a separate comparison record. The Bing rank snapshot guide explains why comparable settings and unknown states matter. It concerns Bing rank comparisons and supplies no runtime evidence for this Google investigation.

6. Original fiction: 100 sitemap entries and seven returned URLs

This entire worked example is original fiction. It is not an Actor run, customer result, or observed company finding.

A fictional site owner saves a sitemap containing 100 URL entries. A separate site-query sample contains seven returned URLs. For this example, the owner has checked that all seven match entries in that sitemap.

The correct initial record is:

6. Original fiction: 100 sitemap entries and seven returned URLs
EvidenceFictional valueMeaning
Sitemap inventory100 URL entriesAddresses listed in the saved file
Retained site-query sample7 returned URLsAddresses observed in that search sample
Complete indexed-page countUnknownNeither record establishes this count
Owner inspection resultsNot yet recordedFurther checks are proposed

The arithmetic difference is 100 − 7 = 93. Its units are a difference between sitemap-entry and search-result counts. It is not 93 confirmed unindexed pages. The sitemap and search results are not matched complete sets of indexed and unindexed pages.

Even in this example, where the seven returned URLs match sitemap entries, absence from the search sample cannot establish index exclusion. Do not report seven percent index coverage. The seven observed URLs do not provide a complete indexed-URL numerator.

The owner then selects one intended URL that was not returned. The next action is an owner-access URL Inspection check. No inspection result has occurred in this example, so the inspection state remains “not checked.”

If a later inspection reports that the exact URL is indexed, record that finding for that URL and check time. Keep the earlier search state as “not returned in the retained site-query sample.” Those records can coexist because they answer different questions.

If the later check reports a different condition, preserve its exact meaning and choose the next action from that evidence. Do not extend the result to the other unchecked URLs.

7. Check the sheet before you report a finding

Use these acceptance checks:

  1. Confirm the counting unit for every number: entries, distinct addresses, search rows, or inspected URLs.
  2. Confirm that every search state points to an exact query and saved conditions.
  3. Confirm that returned URLs point to actual retained result rows.
  4. Confirm that “not returned” states refer only to the retained sample.
  5. Confirm that failed collections remain failures rather than zero-result claims.
  6. Confirm that owner findings name the exact inspected URL and check time.
  7. Keep missing, zero, unresolved, and not checked as different values.
  8. Remove any total-index estimate derived from search counts.

A useful report states the inventory size, observed search results, inspected subset, confirmed findings, and unresolved questions. If you inspected only selected URLs, say so. Do not apply the subset’s findings to the whole site without supporting evidence.

When records disagree, retain both with their sources and times. The next decision is which exact URL needs another check, not which number should replace the others.

Sources & further reading

By Chronometrica · Updated

Back to top ↑