Vexaloom Labs
Actors/Search & URL Markdown Extractor for LLMs

Search & URL Markdown Extractor for LLMs

Turn a search query or a list of URLs into clean Markdown, plain text, or HTML pages, with filters for domain, keyword, and minimum content length. No login or cookies needed.

Open on ApifyGitHub source

Input schema

Input fields, types, required flags, descriptions, defaults, and prefills come from the published definition. Prefill examples are distinct from defaults.

FieldTypeRequiredDescription
querystringoptionalWhat to look up, or a single web page URL to read. Examples: `vector database comparison`, `https://en.wikipedia.org/wiki/Retrieval-augmented_generation`. When `startUrls` or `queries` are also set, everything is processed in one run.
queriesarrayoptionalAdditional queries for batch research runs. Each query is searched and its top results are fetched. A query that is itself a URL is fetched directly.
startUrlsarrayoptionalPages to read directly. With `maxCrawlDepth` above 0, the actor walks links found on these pages within the same website instead.
Show all input fields
FieldTypeRequiredDescription
querystringoptionalWhat to look up, or a single web page URL to read. Examples: `vector database comparison`, `https://en.wikipedia.org/wiki/Retrieval-augmented_generation`. When `startUrls` or `queries` are also set, everything is processed in one run.
queriesarrayoptionalAdditional queries for batch research runs. Each query is searched and its top results are fetched. A query that is itself a URL is fetched directly.
startUrlsarrayoptionalPages to read directly. With `maxCrawlDepth` above 0, the actor walks links found on these pages within the same website instead.
searchEnginestringoptionalWhich search engine to use. `Automatic` tries Google first and silently falls back to Bing, then DuckDuckGo when an engine is blocked — recommended.
searchLanguagestringoptionalLanguage for search results. `Automatic` lets each engine choose based on the query.
searchCountrystringoptionalCountry bias for search results. `Automatic` means no country bias.
maxResultsintegeroptionalHow many top organic results to fetch and emit for each search query. Values above 10 need multiple search-result pages (slower). With `Emit results for pages that could not be fetched` on, each of the requested results yields one record — a page record when the fetch succeeds, or a metadata record with `fetchError` when it does not.
maxItemsintegeroptionalOverall cap on how many pages the run may emit across all queries, URLs, and crawled pages. 0 means no global cap.
outputFormatsarrayoptionalWhich representations to include per page, in any combination. `Links` adds every link found on the page as structured `{url, text}` pairs.
removeElementsCssSelectorstringoptionalComma-separated CSS selectors removed before extraction (navigation, footers, ads, cookie bars). Set a non-matching selector such as `dummy_keep_everything` to keep everything.
keepElementsCssSelectorarrayoptionalWhen set, only content inside these selectors is kept — everything else is dropped before extraction.
htmlTransformerstringoptional`Readable content` finds the article body and drops navigation and boilerplate. `None` keeps the full cleaned page.
maxContentLengthintegeroptionalTruncate the emitted text and Markdown of each page to this many characters. 0 returns the full content. Useful for controlling embedding cost.
includeMetadatabooleanoptionalAdd canonical URL, author, published date, keywords, social image, and site name (each only when the page provides it).
chunkMarkdownbooleanoptionalEmit a `chunks` array per page — heading-aware Markdown chunks ready for embedding, never split inside code blocks or tables.
chunkMaxCharsintegeroptionalTarget size of each chunk when `chunkMarkdown` is on. Code blocks and tables are kept whole even when larger.
chunkOverlapCharsintegeroptionalApproximate overlap between consecutive chunks when `chunkMarkdown` is on, so context is not lost at chunk boundaries.
maxCrawlDepthintegeroptional0 reads only the start URLs. 1+ follows links found on those pages, staying on the same website, up to this depth.
maxCrawlPagesintegeroptionalSafety cap on pages visited in crawl mode.
domainAllowlistarrayoptionalKeep only pages whose host contains one of these substrings (case-insensitive).
domainBlocklistarrayoptionalDrop pages whose host contains one of these substrings.
minTextLengthintegeroptionalDrop pages with fewer than this many characters of extracted text. Useful for skipping thin or placeholder pages.
excludeContentSelectorsarrayoptionalAdditional CSS selectors removed before extraction, on top of `Remove elements`.
keywordsAnyOfarrayoptionalKeep only pages whose title or content contains at least one of these keywords (case-insensitive).
emitFailedResultsbooleanoptionalWhen a search result page cannot be fetched after the full retry ladder, still emit one record for it with `url`, `title`, `description`, `searchResult`, `searchRank` and a `fetchError` note (no fabricated content). Turn off to skip failed pages entirely.

Download the published input schema

View complete JSON
{
  "description": "Search the web with a query, fetch specific URLs, or crawl a site — and get clean, RAG-ready content (Markdown, text, links, and optional chunks) with citation metadata.",
  "properties": {
    "chunkMarkdown": {
      "default": false,
      "description": "Emit a `chunks` array per page — heading-aware Markdown chunks ready for embedding, never split inside code blocks or tables.",
      "prefill": false,
      "title": "Split content into chunks",
      "type": "boolean"
    },
    "chunkMaxChars": {
      "default": 2000,
      "description": "Target size of each chunk when `chunkMarkdown` is on. Code blocks and tables are kept whole even when larger.",
      "maximum": 10000,
      "minimum": 200,
      "prefill": 2000,
      "title": "Chunk size (characters)",
      "type": "integer"
    },
    "chunkOverlapChars": {
      "default": 200,
      "description": "Approximate overlap between consecutive chunks when `chunkMarkdown` is on, so context is not lost at chunk boundaries.",
      "maximum": 1000,
      "minimum": 0,
      "prefill": 200,
      "title": "Chunk overlap (characters)",
      "type": "integer"
    },
    "domainAllowlist": {
      "default": [],
      "description": "Keep only pages whose host contains one of these substrings (case-insensitive).",
      "editor": "stringList",
      "prefill": [],
      "title": "Allowed domains",
      "type": "array"
    },
    "domainBlocklist": {
      "default": [],
      "description": "Drop pages whose host contains one of these substrings.",
      "editor": "stringList",
      "prefill": [],
      "title": "Blocked domains",
      "type": "array"
    },
    "emitFailedResults": {
      "default": true,
      "description": "When a search result page cannot be fetched after the full retry ladder, still emit one record for it with `url`, `title`, `description`, `searchResult`, `searchRank` and a `fetchError` note (no fabricated content). Turn off to skip failed pages entirely.",
      "prefill": false,
      "title": "Emit results for pages that could not be fetched",
      "type": "boolean"
    },
    "excludeContentSelectors": {
      "default": [],
      "description": "Additional CSS selectors removed before extraction, on top of `Remove elements`.",
      "editor": "stringList",
      "prefill": [],
      "title": "Extra selectors to remove",
      "type": "array"
    },
    "htmlTransformer": {
      "default": "readable-text",
      "description": "`Readable content` finds the article body and drops navigation and boilerplate. `None` keeps the full cleaned page.",
      "editor": "select",
      "enum": [
        "none",
        "readable-text"
      ],
      "enumTitles": [
        "None (full page)",
        "Readable content (main article)"
      ],
      "prefill": "readable-text",
      "title": "Main content extraction",
      "type": "string"
    },
    "includeMetadata": {
      "default": true,
      "description": "Add canonical URL, author, published date, keywords, social image, and site name (each only when the page provides it).",
      "prefill": true,
      "title": "Include page metadata",
      "type": "boolean"
    },
    "keepElementsCssSelector": {
      "default": [],
      "description": "When set, only content inside these selectors is kept — everything else is dropped before extraction.",
      "editor": "stringList",
      "prefill": [],
      "title": "Keep only elements (CSS selector)",
      "type": "array"
    },
    "keywordsAnyOf": {
      "default": [],
      "description": "Keep only pages whose title or content contains at least one of these keywords (case-insensitive).",
      "editor": "stringList",
      "prefill": [],
      "title": "Required keywords (any of)",
      "type": "array"
    },
    "maxContentLength": {
      "default": 0,
      "description": "Truncate the emitted text and Markdown of each page to this many characters. 0 returns the full content. Useful for controlling embedding cost.",
      "maximum": 1000000,
      "minimum": 0,
      "prefill": 0,
      "title": "Max content length (characters)",
      "type": "integer"
    },
    "maxCrawlDepth": {
      "default": 0,
      "description": "0 reads only the start URLs. 1+ follows links found on those pages, staying on the same website, up to this depth.",
      "maximum": 5,
      "minimum": 0,
      "prefill": 0,
      "title": "Crawl depth",
      "type": "integer"
    },
    "maxCrawlPages": {
      "default": 25,
      "description": "Safety cap on pages visited in crawl mode.",
      "maximum": 500,
      "minimum": 1,
      "prefill": 25,
      "title": "Max pages to crawl",
      "type": "integer"
    },
    "maxItems": {
      "default": 0,
      "description": "Overall cap on how many pages the run may emit across all queries, URLs, and crawled pages. 0 means no global cap.",
      "maximum": 10000,
      "minimum": 0,
      "prefill": 0,
      "title": "Max pages per run",
      "type": "integer"
    },
    "maxResults": {
      "default": 3,
      "description": "How many top organic results to fetch and emit for each search query. Values above 10 need multiple search-result pages (slower). With `Emit results for pages that could not be fetched` on, each of the requested results yields one record — a page record when the fetch succeeds, or a metadata record with `fetchError` when it does not.",
      "maximum": 250,
      "minimum": 1,
      "prefill": 3,
      "title": "Max search results per query",
      "type": "integer"
    },
    "minTextLength": {
      "description": "Drop pages with fewer than this many characters of extracted text. Useful for skipping thin or placeholder pages.",
      "maximum": 1000000,
      "minimum": 0,
      "title": "Minimum text length",
      "type": "integer"
    },
    "outputFormats": {
      "default": [
        "markdown"
      ],
      "description": "Which representations to include per page, in any combination. `Links` adds every link found on the page as structured `{url, text}` pairs.",
      "editor": "select",
      "items": {
        "enum": [
          "markdown",
          "text",
          "html",
          "links"
        ],
        "enumTitles": [
          "Markdown",
          "Plain text",
          "HTML",
          "Links"
        ],
        "type": "string"
      },
      "prefill": [
        "markdown"
      ],
      "title": "Output formats",
      "type": "array"
    },
    "queries": {
      "default": [],
      "description": "Additional queries for batch research runs. Each query is searched and its top results are fetched. A query that is itself a URL is fetched directly.",
      "editor": "stringList",
      "prefill": [],
      "title": "More search queries",
      "type": "array"
    },
    "query": {
      "default": "what is retrieval augmented generation",
      "description": "What to look up, or a single web page URL to read. Examples: `vector database comparison`, `https://en.wikipedia.org/wiki/Retrieval-augmented_generation`. When `startUrls` or `queries` are also set, everything is processed in one run.",
      "editor": "textfield",
      "prefill": "what is retrieval augmented generation",
      "title": "Search query (or one URL)",
      "type": "string"
    },
    "removeElementsCssSelector": {
      "default": "nav, footer, aside, script, style, .ad, .advertisement, [aria-hidden='true']",
      "description": "Comma-separated CSS selectors removed before extraction (navigation, footers, ads, cookie bars). Set a non-matching selector such as `dummy_keep_everything` to keep everything.",
      "editor": "textfield",
      "prefill": "nav, footer, aside, script, style, .ad, .advertisement, [aria-hidden='true']",
      "title": "Remove elements (CSS selector)",
      "type": "string"
    },
    "searchCountry": {
      "default": "auto",
      "description": "Country bias for search results. `Automatic` means no country bias.",
      "editor": "select",
      "enum": [
        "auto",
        "ad",
        "ae",
        "af",
        "ag",
        "ai",
        "al",
        "am",
        "ao",
        "aq",
        "ar",
        "as",
        "at",
        "au",
        "aw",
        "ax",
        "az",
        "ba",
        "bb",
        "bd",
        "be",
        "bf",
        "bg",
        "bh",
        "bi",
        "bj",
        "bl",
        "bm",
        "bn",
        "bo",
        "bq",
        "br",
        "bs",
        "bt",
        "bv",
        "bw",
        "by",
        "bz",
        "ca",
        "cc",
        "cd",
        "cf",
        "cg",
        "ch",
        "ci",
        "ck",
        "cl",
        "cm",
        "cn",
        "co",
        "cr",
        "cu",
        "cv",
        "cw",
        "cx",
        "cy",
        "cz",
        "de",
        "dj",
        "dk",
        "dm",
        "do",
        "dz",
        "ec",
        "ee",
        "eg",
        "eh",
        "er",
        "es",
        "et",
        "fi",
        "fj",
        "fk",
        "fm",
        "fo",
        "fr",
        "ga",
        "gb",
        "gd",
        "ge",
        "gf",
        "gg",
        "gh",
        "gi",
        "gl",
        "gm",
        "gn",
        "gp",
        "gq",
        "gr",
        "gs",
        "gt",
        "gu",
        "gw",
        "gy",
        "hk",
        "hm",
        "hn",
        "hr",
        "ht",
        "hu",
        "id",
        "ie",
        "il",
        "im",
        "in",
        "io",
        "iq",
        "ir",
        "is",
        "it",
        "je",
        "jm",
        "jo",
        "jp",
        "ke",
        "kg",
        "kh",
        "ki",
        "km",
        "kn",
        "kp",
        "kr",
        "kw",
        "ky",
        "kz",
        "la",
        "lb",
        "lc",
        "li",
        "lk",
        "lr",
        "ls",
        "lt",
        "lu",
        "lv",
        "ly",
        "ma",
        "mc",
        "md",
        "me",
        "mf",
        "mg",
        "mh",
        "mk",
        "ml",
        "mm",
        "mn",
        "mo",
        "mp",
        "mq",
        "mr",
        "ms",
        "mt",
        "mu",
        "mv",
        "mw",
        "mx",
        "my",
        "mz",
        "na",
        "nc",
        "ne",
        "nf",
        "ng",
        "ni",
        "nl",
        "no",
        "np",
        "nr",
        "nu",
        "nz",
        "om",
        "pa",
        "pe",
        "pf",
        "pg",
        "ph",
        "pk",
        "pl",
        "pm",
        "pn",
        "pr",
        "ps",
        "pt",
        "pw",
        "py",
        "qa",
        "re",
        "ro",
        "rs",
        "ru",
        "rw",
        "sa",
        "sb",
        "sc",
        "sd",
        "se",
        "sg",
        "sh",
        "si",
        "sj",
        "sk",
        "sl",
        "sm",
        "sn",
        "so",
        "sr",
        "ss",
        "st",
        "sv",
        "sx",
        "sy",
        "sz",
        "tc",
        "td",
        "tf",
        "tg",
        "th",
        "tj",
        "tk",
        "tl",
        "tm",
        "tn",
        "to",
        "tr",
        "tt",
        "tv",
        "tw",
        "tz",
        "ua",
        "ug",
        "um",
        "us",
        "uy",
        "uz",
        "va",
        "vc",
        "ve",
        "vg",
        "vi",
        "vn",
        "vu",
        "wf",
        "ws",
        "ye",
        "yt",
        "za",
        "zm",
        "zw"
      ],
      "enumTitles": [
        "Automatic (no country bias)",
        "Andorra",
        "United Arab Emirates",
        "Afghanistan",
        "Antigua and Barbuda",
        "Anguilla",
        "Albania",
        "Armenia",
        "Angola",
        "Antarctica",
        "Argentina",
        "American Samoa",
        "Austria",
        "Australia",
        "Aruba",
        "Åland Islands",
        "Azerbaijan",
        "Bosnia and Herzegovina",
        "Barbados",
        "Bangladesh",
        "Belgium",
        "Burkina Faso",
        "Bulgaria",
        "Bahrain",
        "Burundi",
        "Benin",
        "Saint Barthélemy",
        "Bermuda",
        "Brunei Darussalam",
        "Bolivia, Plurinational State of",
        "Bonaire, Sint Eustatius and Saba",
        "Brazil",
        "Bahamas",
        "Bhutan",
        "Bouvet Island",
        "Botswana",
        "Belarus",
        "Belize",
        "Canada",
        "Cocos (Keeling) Islands",
        "Congo, The Democratic Republic of the",
        "Central African Republic",
        "Congo",
        "Switzerland",
        "Côte d'Ivoire",
        "Cook Islands",
        "Chile",
        "Cameroon",
        "China",
        "Colombia",
        "Costa Rica",
        "Cuba",
        "Cabo Verde",
        "Curaçao",
        "Christmas Island",
        "Cyprus",
        "Czechia",
        "Germany",
        "Djibouti",
        "Denmark",
        "Dominica",
        "Dominican Republic",
        "Algeria",
        "Ecuador",
        "Estonia",
        "Egypt",
        "Western Sahara",
        "Eritrea",
        "Spain",
        "Ethiopia",
        "Finland",
        "Fiji",
        "Falkland Islands (Malvinas)",
        "Micronesia, Federated States of",
        "Faroe Islands",
        "France",
        "Gabon",
        "United Kingdom",
        "Grenada",
        "Georgia",
        "French Guiana",
        "Guernsey",
        "Ghana",
        "Gibraltar",
        "Greenland",
        "Gambia",
        "Guinea",
        "Guadeloupe",
        "Equatorial Guinea",
        "Greece",
        "South Georgia and the South Sandwich Islands",
        "Guatemala",
        "Guam",
        "Guinea-Bissau",
        "Guyana",
        "Hong Kong",
        "Heard Island and McDonald Islands",
        "Honduras",
        "Croatia",
        "Haiti",
        "Hungary",
        "Indonesia",
        "Ireland",
        "Israel",
        "Isle of Man",
        "India",
        "British Indian Ocean Territory",
        "Iraq",
        "Iran, Islamic Republic of",
        "Iceland",
        "Italy",
        "Jersey",
        "Jamaica",
        "Jordan",
        "Japan",
        "Kenya",
        "Kyrgyzstan",
        "Cambodia",
        "Kiribati",
        "Comoros",
        "Saint Kitts and Nevis",
        "Korea, Democratic People's Republic of",
        "Korea, Republic of",
        "Kuwait",
        "Cayman Islands",
        "Kazakhstan",
        "Lao People's Democratic Republic",
        "Lebanon",
        "Saint Lucia",
        "Liechtenstein",
        "Sri Lanka",
        "Liberia",
        "Lesotho",
        "Lithuania",
        "Luxembourg",
        "Latvia",
        "Libya",
        "Morocco",
        "Monaco",
        "Moldova, Republic of",
        "Montenegro",
        "Saint Martin (French part)",
        "Madagascar",
        "Marshall Islands",
        "North Macedonia",
        "Mali",
        "Myanmar",
        "Mongolia",
        "Macao",
        "Northern Mariana Islands",
        "Martinique",
        "Mauritania",
        "Montserrat",
        "Malta",
        "Mauritius",
        "Maldives",
        "Malawi",
        "Mexico",
        "Malaysia",
        "Mozambique",
        "Namibia",
        "New Caledonia",
        "Niger",
        "Norfolk Island",
        "Nigeria",
        "Nicaragua",
        "Netherlands",
        "Norway",
        "Nepal",
        "Nauru",
        "Niue",
        "New Zealand",
        "Oman",
        "Panama",
        "Peru",
        "French Polynesia",
        "Papua New Guinea",
        "Philippines",
        "Pakistan",
        "Poland",
        "Saint Pierre and Miquelon",
        "Pitcairn",
        "Puerto Rico",
        "Palestine, State of",
        "Portugal",
        "Palau",
        "Paraguay",
        "Qatar",
        "Réunion",
        "Romania",
        "Serbia",
        "Russian Federation",
        "Rwanda",
        "Saudi Arabia",
        "Solomon Islands",
        "Seychelles",
        "Sudan",
        "Sweden",
        "Singapore",
        "Saint Helena, Ascension and Tristan da Cunha",
        "Slovenia",
        "Svalbard and Jan Mayen",
        "Slovakia",
        "Sierra Leone",
        "San Marino",
        "Senegal",
        "Somalia",
        "Suriname",
        "South Sudan",
        "Sao Tome and Principe",
        "El Salvador",
        "Sint Maarten (Dutch part)",
        "Syrian Arab Republic",
        "Eswatini",
        "Turks and Caicos Islands",
        "Chad",
        "French Southern Territories",
        "Togo",
        "Thailand",
        "Tajikistan",
        "Tokelau",
        "Timor-Leste",
        "Turkmenistan",
        "Tunisia",
        "Tonga",
        "Türkiye",
        "Trinidad and Tobago",
        "Tuvalu",
        "Taiwan, Province of China",
        "Tanzania, United Republic of",
        "Ukraine",
        "Uganda",
        "United States Minor Outlying Islands",
        "United States",
        "Uruguay",
        "Uzbekistan",
        "Holy See (Vatican City State)",
        "Saint Vincent and the Grenadines",
        "Venezuela, Bolivarian Republic of",
        "Virgin Islands, British",
        "Virgin Islands, U.S.",
        "Viet Nam",
        "Vanuatu",
        "Wallis and Futuna",
        "Samoa",
        "Yemen",
        "Mayotte",
        "South Africa",
        "Zambia",
        "Zimbabwe"
      ],
      "prefill": "auto",
      "title": "Search country",
      "type": "string"
    },
    "searchEngine": {
      "default": "auto",
      "description": "Which search engine to use. `Automatic` tries Google first and silently falls back to Bing, then DuckDuckGo when an engine is blocked — recommended.",
      "editor": "select",
      "enum": [
        "auto",
        "google",
        "bing",
        "duckduckgo"
      ],
      "enumTitles": [
        "Automatic (recommended)",
        "Google",
        "Bing",
        "DuckDuckGo"
      ],
      "prefill": "auto",
      "title": "Search engine",
      "type": "string"
    },
    "searchLanguage": {
      "default": "auto",
      "description": "Language for search results. `Automatic` lets each engine choose based on the query.",
      "editor": "select",
      "enum": [
        "auto",
        "en",
        "de",
        "fr",
        "es",
        "it",
        "pt",
        "nl",
        "sv",
        "da",
        "fi",
        "no",
        "pl",
        "cs",
        "sk",
        "hu",
        "ro",
        "bg",
        "el",
        "tr",
        "ru",
        "uk",
        "ar",
        "he",
        "fa",
        "hi",
        "id",
        "ms",
        "th",
        "vi",
        "ja",
        "ko",
        "zh"
      ],
      "enumTitles": [
        "Automatic (match the query language)",
        "English",
        "German",
        "French",
        "Spanish",
        "Italian",
        "Portuguese",
        "Dutch",
        "Swedish",
        "Danish",
        "Finnish",
        "Norwegian",
        "Polish",
        "Czech",
        "Slovak",
        "Hungarian",
        "Romanian",
        "Bulgarian",
        "Greek",
        "Turkish",
        "Russian",
        "Ukrainian",
        "Arabic",
        "Hebrew",
        "Persian",
        "Hindi",
        "Indonesian",
        "Malay",
        "Thai",
        "Vietnamese",
        "Japanese",
        "Korean",
        "Chinese"
      ],
      "prefill": "auto",
      "title": "Search language",
      "type": "string"
    },
    "startUrls": {
      "default": [],
      "description": "Pages to read directly. With `maxCrawlDepth` above 0, the actor walks links found on these pages within the same website instead.",
      "editor": "stringList",
      "prefill": [],
      "title": "Start URLs",
      "type": "array"
    }
  },
  "required": [],
  "schemaVersion": 1,
  "title": "RAG Web Browser Input",
  "type": "object"
}

Output schema

Published result links and storage locations. This schema is not an exhaustive contract for every field in a dataset record.

FieldTypeRequiredDescription
pagesstringoptionalDataset of fetched pages with content and metadata, ready for LLM and RAG pipelines.

Download the published output schema

View complete JSON
{
  "actorOutputSchemaVersion": 1,
  "description": "One record per search result or fetched page with clean Markdown/text/HTML, links, page metadata for citations, optional heading-aware chunks, and the search-engine provenance of each result. Pages that could not be fetched still produce a metadata-only record with a `fetchError` note when `emitFailedResults` is on.",
  "properties": {
    "pages": {
      "description": "Dataset of fetched pages with content and metadata, ready for LLM and RAG pipelines.",
      "template": "{{links.apiDefaultDatasetUrl}}/items",
      "title": "RAG pages",
      "type": "string"
    }
  },
  "title": "RAG Web Browser Output"
}

Dataset schema

Published dataset views and any declared field definitions. A view can select only a subset of the returned record fields.

Download the published dataset schema

View complete JSON
{
  "actorSpecification": 1,
  "views": {
    "overview": {
      "display": {
        "component": "table",
        "properties": {}
      },
      "title": "RAG Pages",
      "transformation": {
        "fields": [
          "url",
          "title",
          "description",
          "wordCount",
          "httpStatusCode",
          "loadedTime",
          "searchRank",
          "canonicalUrl",
          "author",
          "publishedDate",
          "contentLength",
          "engine",
          "crawlDepth"
        ]
      }
    }
  }
}

Key-value schema

Published key-value store definitions, when provided by this build.

This definition is not present in the default published build. See the published README for available documentation.

Published README

Read the full published documentation

Web Search & RAG Markdown Extractor for LLMs

Give your LLM or RAG pipeline fresh web content in one run: search the web, read specific pages, or crawl a small site — and receive clean Markdown, plain text, HTML, structured links, and citation metadata. The actor starts its search on Google and automatically falls back to Bing or DuckDuckGo when an engine is unavailable, so a query keeps returning results even when one search provider is having a bad day.

What this actor does

  • Three intents in one actor: search by keyword (query / queries), read exact pages (startUrls), or crawl a site by following links within the same domain (maxCrawlDepth)
  • Resilient search: Google first, with automatic Bing and DuckDuckGo fallbacks; every result keeps its title, snippet, displayed address, position, and the engine that found it
  • RAG-ready output: clean Markdown, plain text, or HTML, plus every link on the page as structured {url, text} pairs and optional heading-aware chunks that never cut code blocks or tables in half
  • Citation metadata: canonical URL, author, published date, keywords, social image, and site name — each included when the page actually provides it
  • Token control: maxContentLength caps the returned content per page; includeMetadata and chunkMarkdown are opt-in extras
  • Filters: domain allowlist/blocklist, minimum text length, required keywords, and custom CSS selectors to strip or keep
  • No login, cookies, or API keys required. Fields a page doesn't have are omitted from its record rather than returned empty

Output per page

  • url — the page's address; loadedUrl — where the fetch ended after redirects; redirectedUrl — the final address when it differs from url
  • domain — host name of the page
  • title, description, languageCode
  • markdown, text, html — included only for the formats you select
  • links — every http(s) link found on the page as {url, text}, when links is selected
  • chunks — heading-aware content pieces with index, text, headingPath, charCount, when chunkMarkdown is on
  • wordCount, readingTimeMinutes (at 220 words per minute), contentLength (characters of the emitted content), contentHash (SHA-256 for change detection)
  • canonicalUrl, author, publishedDate (ISO 8601), keywords, ogImage, siteName — page metadata, when present
  • searchRank — 1-based position of the page in the search results (search mode only)
  • searchResult — {url, title, snippet, displayedUrl, position, engine} for pages that came from a search query
  • query — the search query that produced the page; engine — which engine found it
  • crawlDepth — how many link hops from a start URL (crawl mode only)
  • fetchError — why a page could not be fetched, on metadata-only records (see below)
  • httpStatusCode — the status the page answered with; pages that answer with an error status but still return content are kept like any other page
  • loadedTime — seconds spent fetching the page
  • recordType: "page", scrapedAt

Input

Field Type Default Description
query string what is retrieval augmented generation Search terms, or a single page URL to read directly
queries array [] More queries for batch runs; a URL here is fetched directly
startUrls array [] Pages to read; with maxCrawlDepth > 0 these become crawl seeds
searchEngine string auto auto / google / bing / duckduckgo
searchLanguage string auto Language bias for results
searchCountry string auto Country bias for results
maxResults integer 3 Search results fetched per query (1–250; > 10 needs multiple result pages)
maxItems integer 0 Global page cap for the whole run (0 = no cap)
outputFormats array ["markdown"] Any of markdown, text, html, links
removeElementsCssSelector string nav, footer, aside, script, style, .ad, .advertisement, [aria-hidden='true'] Elements removed before extraction
keepElementsCssSelector array [] Keep only content inside these selectors
htmlTransformer string readable-text readable-text (main article) or none (full page)
maxContentLength integer 0 Truncate content per page (0 = full content)
includeMetadata boolean true Include author, dates, canonical URL, and friends
chunkMarkdown boolean false Emit heading-aware chunks
chunkMaxChars integer 2000 Target chunk size (200–10000)
chunkOverlapChars integer 200 Overlap between chunks (0–1000)
maxCrawlDepth integer 0 Follow same-domain links up to this depth (0 = off)
maxCrawlPages integer 25 Page cap in crawl mode
domainAllowlist array [] Only keep pages whose host contains one of these
domainBlocklist array [] Drop pages whose host contains one of these
minTextLength integer – Drop pages with less extracted text than this
excludeContentSelectors array [] Extra CSS selectors to strip
keywordsAnyOf array [] Keep only pages containing at least one keyword
emitFailedResults boolean true Emit a metadata-only record with fetchError for pages that could not be fetched

Older tasks may still carry tuning fields such as scrapingTool, requestTimeoutSecs, desiredConcurrency, maxRequestRetries, dynamicContentWaitSecs, removeCookieWarnings, useApifyProxy, or debugMode. They are accepted and ignored: request timeouts, retries, rendering, proxy rotation, and diagnostics are decided automatically by the actor.

Example: Google a question and keep only substantial pages

{
  "query": "vector database comparison 2026",
  "maxResults": 5,
  "outputFormats": ["markdown", "links"],
  "minTextLength": 1000,
  "domainBlocklist": ["pinterest.com"]
}

Example: batch queries with citation metadata

{
  "queries": ["retrieval augmented generation evaluation", "chunking strategies for RAG"],
  "maxResults": 3,
  "includeMetadata": true,
  "chunkMarkdown": true,
  "chunkMaxChars": 1500
}

Example: read exact pages

{
  "query": "",
  "startUrls": [
    "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "https://docs.langchain.com/oss/python/langchain/retrieval"
  ],
  "outputFormats": ["markdown", "text"],
  "htmlTransformer": "readable-text"
}

Example: crawl a documentation section

{
  "query": "",
  "startUrls": ["https://docs.crawl4ai.com/core/"],
  "maxCrawlDepth": 2,
  "maxCrawlPages": 20,
  "outputFormats": ["markdown", "links"]
}

Use cases

  • RAG ingestion — pull the current top results for a topic and hand the Markdown (or ready-made chunks) straight to your embedding step
  • Agent context gathering — give an AI agent a fresh, citation-bearing view of the web without building a search stack
  • Citation enrichment — resolve each reference URL from a document into clean text plus author and publication date
  • Documentation snapshots — crawl a docs section and keep contentHash values to detect what actually changed between runs
  • Competitive monitoring — schedule queries with a domainAllowlist to watch specific companies or publications
  • Due-diligence research — batch a list of questions, filter to long-form sources with minTextLength, and export one tidy dataset

FAQ

Do I need to log in or provide cookies?

No. Every search and page request is anonymous: no account, no cookies, no API keys.

How does the actor choose a search engine?

searchEngine: auto (the default) tries Google first. If Google cannot be reached or returns no readable results, the actor automatically tries Bing and then DuckDuckGo. You can also pin one engine explicitly — useful when you want strictly one provider for reproducibility.

Do I need to set up a proxy?

No. The actor starts with direct requests and only switches connections when a page actually blocks it. Everything is handled automatically; there is nothing to configure. Blocked pages get a per-request escalation (datacenter rotation, then a rendering relay) before the actor gives up on them, and pages that still fail are reported through fetchError.

Why do some pages have fewer fields?

Metadata like author or publishedDate is emitted only when the page itself declares it. If a page has no publication date, the field is simply absent from that record — never null, never an empty string. The same applies to chunks (only with chunkMarkdown) and links (only when selected).

Can the actor read JavaScript-heavy pages?

Yes, automatically. The actor first requests every page with plain HTTP; when a page answers with a JavaScript challenge, a 202-style JS stub, or an empty app shell, it renders that page in a real browser (dismissing common consent banners and expanding collapsed sections) before extraction. No setting to change — the escalation is decided per page.

How do I control the size of the output?

Set maxContentLength to cap characters per page, or enable chunkMarkdown to get evenly sized pieces. Both help keep embedding and token costs predictable.

What happens when a page can't be fetched?

maxResults counts search results, so every requested result yields one record. When a site refuses the fetch (for example a paywall or a bot challenge that survives the actor's retry ladder), the record still carries url, title, description, searchResult and searchRank, plus a fetchError note explaining what happened — with no fabricated page content. Set emitFailedResults to false if you prefer the dataset to contain successfully fetched pages only.

What happens if a search returns nothing?

The run finishes with a status message explaining what happened and still succeeds — no placeholder rows. If every provider fails for a query, the log lists the reason.

Are file downloads and AI summaries supported?

No — this actor returns text and metadata for pages. It skips media and document files (video, audio, images, PDFs) because they carry no page text, and it does not call any paid AI service.

Is this affiliated with Google, Bing, or DuckDuckGo?

No. This is an independent, third-party actor. It reads public search result pages and public web pages; it is not operated by or affiliated with any search provider.

Python quickstart

Use a schema-derived input and the Python standard library to start a run. An Apify account and token are required for execution.

Open the step-by-step Python guide →

Source build gPqkQf6syKae3DRXC. Documentation is regenerated from the default published build; private drafts are not included.