How to Build a Crawl Queue From Bing Search Results

First-check setup and source contract

The public Google Search Scraper build 1.0.15 supports Google, Bing, Brave, and DuckDuckGo, but only Bing is positively accepted for this method here. The current public task input uses `queries ["AI agents", "small business automation"]`, `engines ["bing"]`, `maxResultsPerExecutedQuery 20`, `country "US"`, `language "en"`, `safeSearch true`, and `includeSponsoredResults false`. For this pilot, replace the whole queries and engines lists with one intended query and `engines ["bing"]`, set `maxResultsPerExecutedQuery 2`, intended country, language, and same booleans, noting that no expansion setting or public page-limit input exists.

Native result fields and output summary

Each dataset result row contains these exact fields: `runId`, `country`, `language`, `safeSearch`, `engine`, `executedQuery`, `page`, `rankOnPage`, `rank`, `title`, `canonicalUrl`, `resultDomain`, `snippet`, `isSponsored`, `advertiserDomain`, `observedAt`, and `schemaVersion`. A row represents one observation at an engine, query, page, and position. The `canonicalUrl` field provides a normalized candidate address, not a checked HTML canonical tag or original link. Snippets are search text, not fetched page contents.

The output run summary reports: `queriesRequested`, `searchesCompiled`, `executionsPlanned`, `enginesSuccessful`, `enginesFailed`, `pagesAttempted`, `pagesParsed`, `resultsSaved`, `engineStatusCounts`, `engineResultCounts`, `queryResultCounts`, and `stopReason`.

Bing control run reference

An exact control run for `Python asyncio documentation` returned:

• `queriesRequested`: 1
• `searchesCompiled`: 1
• `executionsPlanned`: 1
• `enginesSuccessful`: 1
• `enginesFailed`: 0
• `pagesAttempted`: 1
• `pagesParsed`: 1
• `resultsSaved`: 2
• `engineStatusCounts`: `{"bing": {"ok": 1}}`
• `engineResultCounts`: `{"bing": 2}`
• `queryResultCounts`: `{"Python asyncio documentation": 2}`
• `stopReason`: `result_caps_reached`
• Returned results: Two direct `docs.python.org` URLs at ranks 1 and 2.

This proves a bounded Bing metadata sample only. It does not test all markets, search operators, repeated ranks, or destination fetching. Primary sources include the Google Search Scraper Actor and the Bing Search Results Task.

1. Define the pages you need and your inclusion rule before you search. Bing is a supported discovery source here. Use a neutral topic and a finite row cap rather than promise hundreds of valid sites. Search discovery supplies candidates, not a promise that each page can be collected.
2. Use queries with one intended phrase, engines set to bing, and a small row count limit. Inspect the executed query, engine, timestamps, and output. Query wording changes your source sample. Use plain words unless you test the search engine meaning yourself.
3. Preserve returned result rows before you select targets. Retain the title, candidate address, source host, rank, query, time, and source stop. Reject a broken or non-web address under your rule. Keep an opaque redirect or unclear destination unresolved. A normalizer field name like canonical address does not prove the original page's real identity. Keep the returned string unchanged as source evidence.
4. Check each candidate host and path against your collection purpose. Search titles and snippets can be old or wrong. Inspect the public destination through your reader collection process when needed. Record the final URL, content type, and page evidence. Do not guess product content from a title alone. Login pages and private pages remain unavailable.
5. Assign each target as accepted, rejected, or needs review with a clear reason. Accepted means it meets your check at that time. It does not guarantee crawl success, complete rights, or accurate product data. Keep failed destination checks and source gaps out of your accepted queue.
6. Merge only reviewed duplicate target addresses under a safe rule while keeping all result references. Keep observation and target records separate. Do not erase distinct paths or query parameters that identify different pages. The same host does not mean the same page. The same title does not mean the same target.
7. Build your small queue with the target address, source references, acceptance evidence, desired extraction, and state. Pass accepted targets to a separate bounded crawler made for those pages. The search actor does not run that crawl or supply the main text.
8. Reconcile your collected observations, unique reviewed candidates, accepted, rejected, and unresolved targets, and cap state. Zero accepted targets can be a normal gap, not a list to fill with guessed pages. End by checking one target before you expand your discovery budget.

Original fiction worked example

All domains, words, and counts in this section are original fiction. Seven returned observations point to six distinct candidate addresses under an exact target rule. Two observations share the address `https://garden.example/products`. The other five observations point to a product page, a blog guide, an opaque redirect, a login page, and an invalid address.

1. Seven result observations

2. Six unique target candidates and states

3. Two accepted queue targets

Reconciliation shows six candidates resulting in two accepted targets, two rejected targets, and two unresolved targets. The two observations for the product index remain attached to one queue target with both query, rank, and time references. Do not call seven observations seven crawl sites or throw away the duplicate reference. If the product index fails when the separate crawler later runs, record that failure; an earlier acceptance does not guarantee extraction.