
Keep three records separate: the pages you intend to have, the pages returned by your search, and the exact pages you inspect through your own account. Those records help you investigate individual pages. Subtracting search results from a sitemap does not produce a count of unindexed pages.
1. Start with the pages you intend to investigate
A public question describes the problem well: a site owner submitted a sitemap with 10,000 links but saw 17 results from a site search. A sitemap is a file that lists page addresses. Those two numbers describe different evidence. The public post records one person’s question from June 4, 2020. It does not verify the site’s index state or show how common the problem is. Public site-owner question — published June 4, 2020, retrieved October 25, 2026; reported counts were not independently verified
The useful next question is: “What evidence do I have for each page I expected Google to index?”
For your own site, follow these steps:
- Save the list of page addresses you intend to investigate. A URL is a web address. Use your intended URL list as the starting record.
- Save the sitemap separately. Record when you obtained it and what part of the site it covers.
- Compare the exact addresses in those two records. Note repeated entries and differences before you count them.
- Select exact URLs for owner-access inspection. Start with pages that matter to your decision and pages with unclear evidence.
- Add public search observations as another evidence source.
A sitemap entry shows that an address appears in that file. It does not prove that Google has indexed the address. Your intended URL list describes what you want to investigate. It does not establish what Google currently stores.
State your counting unit. “100 sitemap entries” can differ from “100 distinct URL strings” if entries repeat. Neither number is automatically a count of distinct indexed pages.
If you do not own the site or have authorized Search Console access, record that limit. Public search evidence can still show returned addresses. It cannot replace the owner’s private records.
2. Keep inventory, search observations, and owner checks separate
Use three linked tables in your investigation sheet. Link them through exact URL strings while preserving each source value.
| Record | What it contains | What it supports |
|---|---|---|
| Intended URL and sitemap record | Intended addresses, sitemap entries, source, and record date | Which addresses you expected or listed |
| Public search record | Exact query, settings, returned addresses, and observation times | Which results appeared in that collected sample |
| Owner inspection record | Exact inspected URL, reported findings, source, and check time | What the owner-access check reported for that URL |
Do not replace these tables with one “indexed” column. That would hide the source and strength of each finding.
Also separate these terms:
| Term | Meaning in this investigation |
|---|---|
| Discovered | Evidence says Google knows about an address |
| Crawled | Evidence says Google fetched the page |
| Indexed | Evidence reports that Google included the page in its index |
| Returned by this query | The address appeared in the retained search results |
| Not returned by this query | The retained results did not contain the address |
| Unknown | The required evidence is missing or unclear |
Knowing about a URL, fetching it, indexing it, and returning it for one query are different states. Record only the state that your evidence supports. If you have no crawl evidence, keep crawl status unknown. A search result does not supply a complete crawl history.
Attach a time to each finding. A search observation and an owner inspection can occur at different times. Keep both times. A later check does not change what an earlier record showed.
3. Save the exact site query and search conditions
Before you search, write down the question and scope. For example, you might investigate whether selected URLs under one documentation path appear in a small public search sample.
Record these items:
- The exact query text, including the `site:` operator and path.
- The search engine.
- The requested country and language.
- The result cap, which is the maximum number of results you ask to retain.
- The actual observation time, when available.
- The collection outcome and any known failure.
Keep the actual returned URLs. A displayed total does not provide an exact list to compare with your intended addresses. Neither a displayed total nor the retained rows establish the complete index count.
Chronometrica’s Google Search Scraper can fit the public-search part of this method. An Actor is a hosted data collection program. The checked public build declares the input fields below and the result fields used in this guide. That is documented fit for collecting search observations. It is not evidence of a successful collection for this guide.
This supplied input requests a small sample:
{"country":"US","engines":["google"],"language":"en","maxResultsPerExecutedQuery":3,"queries":["site:developers.google.com/search/docs/"]}JSON is a text format for named values and lists. Replace the sample values in this order:
- Replace the whole `queries` list with one intended site query. The supplied query concerns Google’s documentation path, not your site.
- Keep `engines` set to `["google"]` for this method.
- Set `country` and `language` to your intended search conditions. Here, `US` requests the United States and `en` requests English.
- Keep the first cap small. The supplied value is 3. The checked allowed range for `maxResultsPerExecutedQuery` is 1 to 100.
- Save the complete input beside the collection record.
The supplied input passed a static check against the public default-build rules. A static check checks the input format and allowed values without running the Actor. No Actor collection occurred for this guide. The check does not prove result availability, complete coverage, or successful execution. Public default build 1.0.15 — checked October 25, 2026; input and output rules inspected, sample not run
The public Google Search Results Scraper Task is another starting point. A Task is a saved input preset. Its public page and exact Task ID were verified. Its current saved settings were not verified. Inspect and replace those settings before use. Do not assume they match the supplied input. Google Search Results Scraper Task — checked October 25, 2026; public presence verified, saved settings and runtime unverified
4. Preserve the search rows and build the investigation sheet
The following collection steps are proposed reader actions. They are not results from a run performed for this article.
After collection, check whether it finished, failed, or stopped with an unclear outcome. Save the available result rows and collection evidence. If collection fails, record a collection failure. Do not report that Google returned no matching pages.
Retain these documented result fields:
| Field | What to retain |
|---|---|
| `engine` | Search engine attached to the result |
| `executedQuery` | Query attached to the result |
| `canonicalUrl` | Returned result address string |
| `resultDomain` | Returned domain, the site-name part of an address |
| `rank` | Running position among supported result blocks parsed across fetched pages |
| `title` | Search-result title |
| `snippet` | Search-result text, including a supplied null value |
| `isSponsored` | Flag that identifies whether the result is sponsored |
| `observedAt` | Recorded observation time |
These fields are declared in the checked build. Their declaration does not prove that a live collection supplied every value. Keep the returned address unchanged. The field name `canonicalUrl` alone does not prove that someone checked the destination page’s preferred address. A snippet is search-result text, not a fresh copy of the destination page. Public default build 1.0.15 — checked October 25, 2026; declared result fields, no live output verification
Build a separate investigation table with one row for each intended URL:
| Investigation field | What you record |
|---|---|
| Intended URL | Exact address you want to investigate |
| Inventory source and date | Where the address came from and when you saved it |
| Sitemap membership | Listed, not listed in the checked file, or unknown |
| Site-query reference | Link or reference to the saved query and its settings |
| Search state | Returned, not returned in the retained sample, failed, or unresolved |
| Matching result references | References to actual search rows |
| Ordinary-query check | Exact query, observed result, and time |
| Owner inspection | Exact inspected URL, reported findings, and check time |
| Address relationship evidence | Any observed preferred-address or redirect finding |
| Access evidence | Any observed page-access finding and its time |
| Next action | The specific check needed to resolve the question |
Preserve missing values. A missing rank is unknown, not rank zero. A missing timestamp is unknown, not the time you opened the sheet. A missing snippet is not proof that a page has no text. Zero returned rows is a supported count only when the collection evidence establishes zero within that request’s scope.
If you need help moving retained results into a spreadsheet, use the separate Google search export guide. That guide covers export checks. This investigation uses those rows to assess individual URLs.
5. Check selected exact URLs through the right source
Use an ordinary search to look for selected exact URLs as a second public observation. Save the query and returned address. Compare exact strings before deciding that a result matches your intended page.
If the address differs, keep both values. Do not silently treat a different path, address parameter, or redirect as the same page. A redirect sends a request from one address to another. If you cannot establish the relationship, mark the match unresolved.
An ordinary-query absence has the same basic limit as a site-query absence: it records what was not returned by that query. It does not establish exclusion from the entire index.
For an owned site, use Google Search Console’s URL Inspection tool through your own authorized access. URL Inspection supplies an owner-access check for a specific address. Follow the current official tool guidance rather than relying on a fixed sequence of screen controls.
Use this sequence for each selected URL:
- Inspect the exact intended address.
- Save the reported findings and the check time.
- Keep any different address identified by the check as a separate value.
- Record any available discovery, crawl, or index evidence separately.
- Compare that evidence with the public search observation.
- Select the next check from the reported evidence.
If the evidence identifies a canonical URL, that means a preferred page address. Preserve that address beside the inspected address. Do not rewrite the original record. If the evidence identifies a redirect or access problem, retain the finding and consult current official guidance for that condition.
Do not choose a technical repair from a search absence alone. A missing result does not identify a cause. Likewise, an owner check for one URL does not establish the state of every URL in the sitemap.
Keep owner-only records clearly labeled. The Actor’s public result rows are separate from private Search Console information. This guide does not establish that the Actor can read your Search Console account.
If your question later becomes whether a result position changed, use a separate comparison record. The Bing rank snapshot guide explains why comparable settings and unknown states matter. It concerns Bing rank comparisons and supplies no runtime evidence for this Google investigation.
6. Original fiction: 100 sitemap entries and seven returned URLs
This entire worked example is original fiction. It is not an Actor run, customer result, or observed company finding.
A fictional site owner saves a sitemap containing 100 URL entries. A separate site-query sample contains seven returned URLs. For this example, the owner has checked that all seven match entries in that sitemap.
The correct initial record is:
| Evidence | Fictional value | Meaning |
|---|---|---|
| Sitemap inventory | 100 URL entries | Addresses listed in the saved file |
| Retained site-query sample | 7 returned URLs | Addresses observed in that search sample |
| Complete indexed-page count | Unknown | Neither record establishes this count |
| Owner inspection results | Not yet recorded | Further checks are proposed |
The arithmetic difference is 100 − 7 = 93. Its units are a difference between sitemap-entry and search-result counts. It is not 93 confirmed unindexed pages. The sitemap and search results are not matched complete sets of indexed and unindexed pages.
Even in this example, where the seven returned URLs match sitemap entries, absence from the search sample cannot establish index exclusion. Do not report seven percent index coverage. The seven observed URLs do not provide a complete indexed-URL numerator.
The owner then selects one intended URL that was not returned. The next action is an owner-access URL Inspection check. No inspection result has occurred in this example, so the inspection state remains “not checked.”
If a later inspection reports that the exact URL is indexed, record that finding for that URL and check time. Keep the earlier search state as “not returned in the retained site-query sample.” Those records can coexist because they answer different questions.
If the later check reports a different condition, preserve its exact meaning and choose the next action from that evidence. Do not extend the result to the other unchecked URLs.
7. Check the sheet before you report a finding
Use these acceptance checks:
- Confirm the counting unit for every number: entries, distinct addresses, search rows, or inspected URLs.
- Confirm that every search state points to an exact query and saved conditions.
- Confirm that returned URLs point to actual retained result rows.
- Confirm that “not returned” states refer only to the retained sample.
- Confirm that failed collections remain failures rather than zero-result claims.
- Confirm that owner findings name the exact inspected URL and check time.
- Keep missing, zero, unresolved, and not checked as different values.
- Remove any total-index estimate derived from search counts.
A useful report states the inventory size, observed search results, inspected subset, confirmed findings, and unresolved questions. If you inspected only selected URLs, say so. Do not apply the subset’s findings to the whole site without supporting evidence.
When records disagree, retain both with their sources and times. The next decision is which exact URL needs another check, not which number should replace the others.
Sources & further reading
- Google Search Central: site search operator — retrieved October 25, 2026; returned lists are not exhaustive and indexed URLs need not appear
- Chronometrica public default build 1.0.15 — checked October 25, 2026; input and output contract inspected, static validation only, no live collection
- Google Search Results Scraper Task — checked October 25, 2026; exact Task presence verified, saved settings and runtime unverified
- Public site-owner question — published June 4, 2020, retrieved October 25, 2026; reported sitemap and search counts provide question context only
By Chronometrica · Updated
Back to top ↑
