Domain cards connect to query cards in a simple presence grid.
Compare only the fixed panel you checked.

An Actor is a hosted data collection program. A Task is a saved input preset. The Actor collects source rows. You make the comparison in this article.

What you are comparing

A domain is the website host you chose to count. A query panel is a short list of exact phrases. Fixed means you write that list before you see results, then you do not change it. A cap is the most results you ask to keep for one exact query.

Organic means the row is not a paid ad. For an organic row, isSponsored is false. Presence means the domain appeared in the kept sample for that query.

not_seen_within_cap means the query succeeded and the domain was absent inside the cap. source_failed means coverage for that query is unknown. Do not store a failed query as zero. unresolved_identity means you still cannot tie the host to the company. That host is not a known miss.

Rank is the order in that exact collected sample. It is not a rank for every person.

Set the Bing input

The current public Google Search Scraper build is 1.0.15. The build can take Google, Bing, Brave, and DuckDuckGo. These steps accept only Bing. The saved Task is a starting input. It does not do this comparison for you.

The current saved input is below. You will replace the whole query list and the whole engine list.

json
{
  "queries": ["competitor brand", "product category", "best product alternatives"],
  "engines": ["google", "bing", "brave", "duckduckgo"],
  "maxResultsPerExecutedQuery": 20,
  "country": "US",
  "language": "en",
  "safeSearch": true,
  "includeSponsoredResults": false
}
Set the Bing input

For a first check, use one intended phrase and Bing only. Set the cap to 2. The checked acceptance used country US and language en. safeSearch true turns the safe-search filter on. includeSponsoredResults false asks to leave paid ads out. No expansion setting exists. Do not add a page-limit field. The checked two-row pilot read exactly one page.

Replace the words one intended phrase with your real phrase before you run this input.

json
{
  "queries": ["one intended phrase"],
  "engines": ["bing"],
  "maxResultsPerExecutedQuery": 2,
  "country": "US",
  "language": "en",
  "safeSearch": true,
  "includeSponsoredResults": false
}
Set the Bing input

For the later panel, pick one cap for every exact query. Keep the same country, language, safe search, and sponsored setting. The later cap is your choice. It is not a hidden field. Country and language are requested settings. They do not prove where a reader lives. They do not promise map or local-pack results.

Read one result row

One row is one result for one engine, one query, one page, and one position. The Actor can keep many queries in one run. It does not make one row for each website in the world.

Use the row fields as source evidence:

  • runId is the id of that collection run.
  • country is the requested country.
  • language is the requested language.
  • safeSearch records that the safe-search filter was on or off.
  • engine is the search engine for that row.
  • executedQuery is the exact phrase that ran.
  • page is the result page number.
  • rankOnPage is the position on that page.
  • rank is the position in the collected sample.
  • title is the result title.
  • canonicalUrl is a cleaned address from the Actor. Keep that string unchanged. It is not a canonical tag read from page HTML. It is not a full copy of the original link.
  • resultDomain is the host from that address.
  • snippet is the short text on the search result. It is not text fetched from the page. It does not prove a claim.
  • isSponsored is true when the row is a paid ad.
  • advertiserDomain is the advertiser host when the row is an ad.
  • observedAt is the collection time.
  • schemaVersion is the name of the row format.

If a row has isSponsored true, keep it out of the organic rank. The Actor does not open the destination page. It does not strip all extra page text. It does not check citations. It does not score business success. It does not schedule a repeat run. You open the pages that matter to your decision.

Read the run summary

OUTPUT is the run summary. It is not the same thing as one result row. Read OUTPUT before you mark a query as useful.

  • queriesRequested is how many phrases you sent.
  • searchesCompiled is how many searches were prepared.
  • executionsPlanned is how many engine runs were planned.
  • enginesSuccessful is how many engines succeeded.
  • enginesFailed is how many engines failed.
  • pagesAttempted is how many pages were tried.
  • pagesParsed is how many pages were read.
  • resultsSaved is how many result rows were kept.
  • engineStatusCounts groups status by engine.
  • engineResultCounts groups saved rows by engine.
  • queryResultCounts groups saved rows by query.
  • stopReason says why the run stopped.

A current Bing control used the phrase Python asyncio documentation, with US and en. It returned two direct docs.python.org URLs. The engine was bing. The executed query was that same phrase. Both rows were page 1. The ranks were 1 and 2. Each row had observedAt. OUTPUT reported one successful engine, one parsed page, and stopReason result_caps_reached.

That control shows limited Bing support for URL, rank, and collection time. It is not a competitor-panel test. It does not verify Google, Brave, or DuckDuckGo. It does not verify every operator, every country, or a repeated rank test. A failed Google path had redirect links whose final pages were not clear. Do not treat those links as accepted page addresses.

A successful query that hits the cap can support not_seen_within_cap. A failed query cannot. A cap or an empty result does not mean you saw the whole index.

Do the comparison

  1. Choose one target market and one language first. Choose the website decision you need to make. Write the query list. Write the competitor list. Do this before you see who appears. Do not add or drop a query after you see results. Do not add or drop a company after you see results.
  2. Open each company's current public site. Write the exact host rule before the run. Say whether a subdomain counts. A blog, directory, or affiliate with a similar name is not automatically that company. This identity check is yours. The Actor does not verify the company. If the host stays unclear, mark unresolved_identity. Do not treat that cell as a known miss.
  3. Run one phrase first with the Bing input above. Use maxResultsPerExecutedQuery 2. Use your requested country and language. The checked acceptance used US and en only. Set safeSearch to true. Set includeSponsoredResults to false. There is no page-limit field and no expansion field. When that check works, run the full fixed list. Use one cap on every exact query. Keep the same market settings.
  4. Keep every useful row. Save the address, engine, rank, and observedAt. Read OUTPUT for each planned query. If the query fails, coverage is unknown. If the query succeeds and a known domain is missing inside the cap, mark not_seen_within_cap. Do not fill a failed query with zero.
  5. Make a competitor-by-query table. Use only present, not_seen_within_cap, source_failed, or unresolved_identity. Several accepted pages from one domain in one query still count as one presence. Keep the page rows. Those rows show which address and rank support the cell. Do not merge rows by title. Do not drop a distinct query string.
  6. Divide present queries by useful queries for each known domain. A useful query succeeded under the same cap and settings. Also report how many queries you planned. This fraction is not a share of all searches. It is not search volume. It is not ad demand. It is not customer demand. If many planned queries fail, do not give the panel comparison, or label it partial. Do not hide the failed queries.
  7. Keep the best organic rank for each domain on each query where it is present. A conditional mean is the average of those best ranks only. Leave out queries where the domain was not seen. Do not enter a missing rank as zero. Do not enter it as cap plus one. Do not average a missing rank as if you saw it. Domains with different counts of present queries can look stronger or weaker for that reason alone. This article compares presence across queries. It does not track one rank across days.
  8. Review the gaps. Then choose the pages you will read. A domain absent from this sample can appear later. It can appear for another query. It can appear for another person. The table does not show why a page appeared. It does not predict traffic or sales. This Bing method does not claim a Google result or a result from another country.

Fictional worked example

All values in this section are fiction. They are not a product result. The panel is fixed Bing, country US, language en, organic results only, and a top-five cap. The company domains are A, B, and C. The host rule counts the exact company host. A subdomain counts only on that same company host. A same-name blog does not count. No host in this fiction is unresolved_identity.

The planned queries are Q1, Q2, Q3, and Q4. Q1, Q2, and Q3 succeeded. Q4 failed at the source. These are the page observations.

Fictional worked example
QueryDomainPageBest organic rankOther organic ranksState
Q1A124present
Q1B11nonepresent
Q1C1nonenonenot_seen_within_cap
Q2A1nonenonenot_seen_within_cap
Q2B12nonepresent
Q2C13nonepresent
Q3A13nonepresent
Q3B11nonepresent
Q3C1nonenonenot_seen_within_cap
Q4Anonenonenonesource_failed
Q4Bnonenonenonesource_failed
Q4Cnonenonenonesource_failed

A has two Q1 pages, at ranks 2 and 4. Those two pages are still one Q1 presence. They do not change A from 2/3 to 3/3.

Three queries were useful. Four queries were planned. The presence table is below.

Fictional worked example
DomainQ1Q2Q3Q4Present on useful queries
Apresentnot_seen_within_cappresentsource_failed2/3
Bpresentpresentpresentsource_failed3/3
Cnot_seen_within_cappresentnot_seen_within_capsource_failed1/3

Best organic ranks use only the present queries.

Fictional worked example
DomainBest ranksConditional meanQuery count in the average
A2 and 32.52
B1, 2, and 14/3, about 1.333
C331

The conditional mean for A is (2 + 3) / 2 = 2.5. The conditional mean for B is (1 + 2 + 1) / 3 = 4/3, about 1.33. The conditional mean for C is 3 / 1 = 3. These averages do not use the same count of queries. They do not replace the presence table. They do not show that one domain wins in every search.

Each not_seen_within_cap cell stops at the top five organic rows. It is not absence from the whole index.

Q4 is source_failed for A, B, and C. That cell is unknown. It is not a known miss. Do not say B is present on 3 of 4 queries. Do not call 3/3 the coverage of all four planned queries. Keep both counts: four planned and three useful. A panel claim here covers only the three successful queries. If too many planned queries fail, do not give the comparison, or label it partial.

Sources & further reading

By Chronometrica · Updated

Back to top ↑