Use Search & URL Markdown Extractor for LLMs with Python

Turn a search query or a list of URLs into clean Markdown, plain text, or HTML pages, with filters for domain, keyword, and minimum content length. No login or cookies needed.

1. Choose the data you need

Review the published README for supported collection modes, examples, and limits. The recipe below uses the current published input definition.

2. Prepare a source-based input

This JSON contains published prefills and required-field defaults. Review every value for your task before running it; it is not a promise of successful collection.

{
  "chunkMarkdown": false,
  "chunkMaxChars": 2000,
  "chunkOverlapChars": 200,
  "domainAllowlist": [],
  "domainBlocklist": [],
  "emitFailedResults": false,
  "excludeContentSelectors": [],
  "htmlTransformer": "readable-text",
  "includeMetadata": true,
  "keepElementsCssSelector": [],
  "keywordsAnyOf": [],
  "maxContentLength": 0,
  "maxCrawlDepth": 0,
  "maxCrawlPages": 25,
  "maxItems": 0,
  "maxResults": 3,
  "outputFormats": [
    "markdown"
  ],
  "queries": [],
  "query": "what is retrieval augmented generation",
  "removeElementsCssSelector": "nav, footer, aside, script, style, .ad, .advertisement, [aria-hidden='true']",
  "searchCountry": "auto",
  "searchEngine": "auto",
  "searchLanguage": "auto",
  "startUrls": []
}

Download input.example.json and save your reviewed copy as input.json.

Field	Type	Required	Description
`query`	string	optional	What to look up, or a single web page URL to read. Examples: `vector database comparison`, `https://en.wikipedia.org/wiki/Retrieval-augmented_generation`. When `startUrls` or `queries` are also set, everything is processed in one run.
`queries`	array	optional	Additional queries for batch research runs. Each query is searched and its top results are fetched. A query that is itself a URL is fetched directly.
`startUrls`	array	optional	Pages to read directly. With `maxCrawlDepth` above 0, the actor walks links found on these pages within the same website instead.
`searchEngine`	string	optional	Which search engine to use. `Automatic` tries Google first and silently falls back to Bing, then DuckDuckGo when an engine is blocked — recommended.
`searchLanguage`	string	optional	Language for search results. `Automatic` lets each engine choose based on the query.
`searchCountry`	string	optional	Country bias for search results. `Automatic` means no country bias.
`maxResults`	integer	optional	How many top organic results to fetch and emit for each search query. Values above 10 need multiple search-result pages (slower). With `Emit results for pages that could not be fetched` on, each of the requested results yields one record — a page record when the fetch succeeds, or a metadata record with `fetchError` when it does not.
`maxItems`	integer	optional	Overall cap on how many pages the run may emit across all queries, URLs, and crawled pages. 0 means no global cap.
`outputFormats`	array	optional	Which representations to include per page, in any combination. `Links` adds every link found on the page as structured `{url, text}` pairs.
`removeElementsCssSelector`	string	optional	Comma-separated CSS selectors removed before extraction (navigation, footers, ads, cookie bars). Set a non-matching selector such as `dummy_keep_everything` to keep everything.
`keepElementsCssSelector`	array	optional	When set, only content inside these selectors is kept — everything else is dropped before extraction.
`htmlTransformer`	string	optional	`Readable content` finds the article body and drops navigation and boilerplate. `None` keeps the full cleaned page.
`maxContentLength`	integer	optional	Truncate the emitted text and Markdown of each page to this many characters. 0 returns the full content. Useful for controlling embedding cost.
`includeMetadata`	boolean	optional	Add canonical URL, author, published date, keywords, social image, and site name (each only when the page provides it).
`chunkMarkdown`	boolean	optional	Emit a `chunks` array per page — heading-aware Markdown chunks ready for embedding, never split inside code blocks or tables.
`chunkMaxChars`	integer	optional	Target size of each chunk when `chunkMarkdown` is on. Code blocks and tables are kept whole even when larger.
`chunkOverlapChars`	integer	optional	Approximate overlap between consecutive chunks when `chunkMarkdown` is on, so context is not lost at chunk boundaries.
`maxCrawlDepth`	integer	optional	0 reads only the start URLs. 1+ follows links found on those pages, staying on the same website, up to this depth.
`maxCrawlPages`	integer	optional	Safety cap on pages visited in crawl mode.
`domainAllowlist`	array	optional	Keep only pages whose host contains one of these substrings (case-insensitive).
`domainBlocklist`	array	optional	Drop pages whose host contains one of these substrings.
`minTextLength`	integer	optional	Drop pages with fewer than this many characters of extracted text. Useful for skipping thin or placeholder pages.
`excludeContentSelectors`	array	optional	Additional CSS selectors removed before extraction, on top of `Remove elements`.
`keywordsAnyOf`	array	optional	Keep only pages whose title or content contains at least one of these keywords (case-insensitive).
`emitFailedResults`	boolean	optional	When a search result page cannot be fetched after the full retry ladder, still emit one record for it with `url`, `title`, `description`, `searchResult`, `searchRank` and a `fetchError` note (no fabricated content). Turn off to skip failed pages entirely.

3. Start an Actor run

Set APIFY_TOKEN in your environment. Python 3.10 or newer is sufficient; this recipe uses no third-party packages.

Execution may incur charges. Review current pricing on Apify. The request asks for a 120-second timeout and a charge limit; these settings are subject to Apify’s pricing and do not guarantee an all-inclusive spending cap.

import json
import os
import urllib.request

# Save the schema-derived example as input.json; review and edit it first.
with open("input.json", encoding="utf-8") as stream:
    actor_input = json.load(stream)

token = os.environ["APIFY_TOKEN"]
request = urllib.request.Request(
    "https://api.apify.com/v2/acts/svDKrzKCcdfn7GHRJ/runs?timeout=120&maxTotalChargeUsd=0.50",
    data=json.dumps(actor_input).encode("utf-8"),
    headers={"Authorization": "Bearer " + token, "Content-Type": "application/json"},
    method="POST",
)
# This starts a potentially billable run. Do not automatically retry this POST.
with urllib.request.urlopen(request, timeout=40) as response:
    run = json.load(response)["data"]
print("Run ID:", run["id"])
print("Status:", run["status"])
# Inspect and download results in Apify Console once the run succeeds.

4. Inspect the actual results

Open the run in Apify Console and wait for a successful completion. Use the published output schema to locate results and the dataset schema to understand declared views. Missing or optional record fields must not be treated as guaranteed values.

One record per search result or fetched page with clean Markdown/text/HTML, links, page metadata for citations, optional heading-aware chunks, and the search-engine provenance of each result. Pages that could not be fetched still produce a metadata-only record with a `fetchError` note when `emitFailedResults` is on.

Questions before you run

Which inputs are required?

No top-level fields are marked required in the published schema. Conditional requirements can still apply; check the README.

What does this guide verify?

The fields, prefills, and schema links are rendered from the published build. This is a usage recipe, not a paid live extraction test. Source availability and collection results can vary.

What if the run request times out?

Check Apify Console before retrying. The first request may already have created a run. Do not automatically repeat a run-creation POST.

Open on Apify Full Actor reference

Generated from source build gPqkQf6syKae3DRXC and its README and schemas. Source changes regenerate this guide; no third-party editorial claims are inferred.