# How to Build a Crawl Queue From Bing Search Results

Collect a small Bing search result sample, check each address and page relevance, then mark it accepted, rejected, or unresolved before adding it to a crawl queue. An Actor is a hosted data collection program. A Task is a saved input preset. The Actor collects source rows, while the reader does the analysis.

## First-check setup and source contract

The public Google Search Scraper build 1.0.15 supports Google, Bing, Brave, and DuckDuckGo, but only Bing is positively accepted for this method here. The current public task input uses `queries ["AI agents", "small business automation"]`, `engines ["bing"]`, `maxResultsPerExecutedQuery 20`, `country "US"`, `language "en"`, `safeSearch true`, and `includeSponsoredResults false`. For this pilot, replace the whole queries and engines lists with one intended query and `engines ["bing"]`, set `maxResultsPerExecutedQuery 2`, intended country, language, and same booleans, noting that no expansion setting or public page-limit input exists.

### Native result fields and output summary

Each dataset result row contains these exact fields: `runId`, `country`, `language`, `safeSearch`, `engine`, `executedQuery`, `page`, `rankOnPage`, `rank`, `title`, `canonicalUrl`, `resultDomain`, `snippet`, `isSponsored`, `advertiserDomain`, `observedAt`, and `schemaVersion`. A row represents one observation at an engine, query, page, and position. The `canonicalUrl` field provides a normalized candidate address, not a checked HTML canonical tag or original link. Snippets are search text, not fetched page contents.

The output run summary reports: `queriesRequested`, `searchesCompiled`, `executionsPlanned`, `enginesSuccessful`, `enginesFailed`, `pagesAttempted`, `pagesParsed`, `resultsSaved`, `engineStatusCounts`, `engineResultCounts`, `queryResultCounts`, and `stopReason`.

### Bing control run reference

An exact control run for `Python asyncio documentation` returned:

- `queriesRequested`: 1
- `searchesCompiled`: 1
- `executionsPlanned`: 1
- `enginesSuccessful`: 1
- `enginesFailed`: 0
- `pagesAttempted`: 1
- `pagesParsed`: 1
- `resultsSaved`: 2
- `engineStatusCounts`: `{"bing": {"ok": 1}}`
- `engineResultCounts`: `{"bing": 2}`
- `queryResultCounts`: `{"Python asyncio documentation": 2}`
- `stopReason`: `result_caps_reached`
- Returned results: Two direct `docs.python.org` URLs at ranks 1 and 2.

This proves a bounded Bing metadata sample only. It does not test all markets, search operators, repeated ranks, or destination fetching. Primary sources include the [Google Search Scraper Actor](https://apify.com/chronometrica/google-search-scraper) and the [Bing Search Results Task](https://apify.com/chronometrica/google-search-scraper/examples/bing-search-results-scraper).

1. Define the pages you need and your inclusion rule before you search. Bing is a supported discovery source here. Use a neutral topic and a finite row cap rather than promise hundreds of valid sites. Search discovery supplies candidates, not a promise that each page can be collected.
2. Use queries with one intended phrase, engines set to bing, and a small row count limit. Inspect the executed query, engine, timestamps, and output. Query wording changes your source sample. Use plain words unless you test the search engine meaning yourself.
3. Preserve returned result rows before you select targets. Retain the title, candidate address, source host, rank, query, time, and source stop. Reject a broken or non-web address under your rule. Keep an opaque redirect or unclear destination unresolved. A normalizer field name like canonical address does not prove the original page's real identity. Keep the returned string unchanged as source evidence.
4. Check each candidate host and path against your collection purpose. Search titles and snippets can be old or wrong. Inspect the public destination through your reader collection process when needed. Record the final URL, content type, and page evidence. Do not guess product content from a title alone. Login pages and private pages remain unavailable.
5. Assign each target as accepted, rejected, or needs review with a clear reason. Accepted means it meets your check at that time. It does not guarantee crawl success, complete rights, or accurate product data. Keep failed destination checks and source gaps out of your accepted queue.
6. Merge only [reviewed duplicate target addresses](/en/articles/web-result-dedup/) under a safe rule while keeping all result references. Keep observation and target records separate. Do not erase distinct paths or query parameters that identify different pages. The same host does not mean the same page. The same title does not mean the same target.
7. Build your small queue with the target address, source references, acceptance evidence, desired extraction, and state. Pass accepted targets to a separate bounded crawler made for those pages. The search actor does not run that crawl or supply the [main text](/en/articles/web-main-text-extraction-check/).
8. Reconcile your collected observations, unique reviewed candidates, accepted, rejected, and unresolved targets, and cap state. Zero accepted targets can be a normal gap, not a list to fill with guessed pages. End by checking one target before you expand your discovery budget.

## Original fiction worked example

All domains, words, and counts in this section are original fiction. Seven returned observations point to six distinct candidate addresses under an exact target rule. Two observations share the address `https://garden.example/products`. The other five observations point to a product page, a blog guide, an opaque redirect, a login page, and an invalid address.

### 1. Seven result observations

| Observation ID | Query | Rank | Observed At | Returned URL | Source Reference |
| --- | --- | --- | --- | --- | --- |
| obs-1 | garden tools | 1 | 2026-10-13T06:20:14Z | https://garden.example/products | Bing engine, page 1 |
| obs-2 | indoor plants | 2 | 2026-10-13T06:20:15Z | https://garden.example/products | Bing engine, page 1 |
| obs-3 | garden tools | 3 | 2026-10-13T06:20:16Z | https://garden.example/items/shears | Bing engine, page 1 |
| obs-4 | indoor plants | 4 | 2026-10-13T06:20:17Z | https://blog.example/guide | Bing engine, page 1 |
| obs-5 | garden tools | 5 | 2026-10-13T06:20:18Z | https://redirect.example/out | Bing engine, page 1 |
| obs-6 | indoor plants | 6 | 2026-10-13T06:20:19Z | https://account.example/login | Bing engine, page 1 |
| obs-7 | garden tools | 7 | 2026-10-13T06:20:20Z | ftp://invalid.example/file | Bing engine, page 1 |

### 2. Six unique target candidates and states

| Candidate Target URL | Destination Check Result | Relevance State | Reason / Evidence |
| --- | --- | --- | --- |
| `https://garden.example/products` | Valid HTTP 200 | accepted_target | Shared product index matches collection scope. |
| `https://garden.example/items/shears` | Valid HTTP 200 | accepted_target | Specific product page matches collection scope. |
| `https://blog.example/guide` | Valid HTTP 200 | rejected_target | Blog guide is irrelevant to product collection. |
| `https://redirect.example/out` | Unavailable / Opaque | needs_review | Opaque redirect cannot be verified directly. |
| `https://account.example/login` | Login required | needs_review | Destination requires authentication; unavailable. |
| `ftp://invalid.example/file` | Non-HTTP scheme | rejected_target | Malformed or non-http address violates collection rule. |

### 3. Two accepted queue targets

| Target Address | Source References | Check Evidence | Desired Next Extraction | Target State |
| --- | --- | --- | --- | --- |
| `https://garden.example/products` | obs-1, obs-2 | Verified HTTP 200 status, matches product index criteria | Extract category item list | accepted |
| `https://garden.example/items/shears` | obs-3 | Verified HTTP 200 status, matches single product criteria | Extract product details and price | accepted |

Reconciliation shows six candidates resulting in two accepted targets, two rejected targets, and two unresolved targets. The two observations for the product index remain attached to one queue target with both query, rank, and time references. Do not call seven observations seven crawl sites or throw away the duplicate reference. If the product index fails when the separate crawler later runs, record that failure; an earlier acceptance does not guarantee extraction.

## Next action

Record one candidate destination and relevance check before adding that accepted target to the separate crawl queue, or retain needs_review if unavailable, using the [Google Search Scraper Actor](https://apify.com/chronometrica/google-search-scraper) and the [Bing Search Results Task preset](https://apify.com/chronometrica/google-search-scraper/examples/bing-search-results-scraper).

## Sources

- [Automate the process of finding URLs for scraping Reddit discussion](https://www.reddit.com/r/learnpython/comments/f96o07/automate_the_process_of_finding_urls_for_scraping/)