Use Domain-Scoped Site Crawler for Markdown Datasets with Python

Crawl a website from one or more start URLs, staying on the same domain, and get back clean Markdown, text, or HTML per page with title, description, and crawl depth. Built for feeding LLM, RAG, or knowledge-base pipelines.

1. Choose the data you need

Review the published README for supported collection modes, examples, and limits. The recipe below uses the current published input definition.

2. Prepare a source-based input

This JSON contains published prefills and required-field defaults. Review every value for your task before running it; it is not a promise of successful collection.

{
  "chunkMarkdown": false,
  "chunkMaxChars": 2000,
  "chunkOverlapChars": 200,
  "clickElementsCssSelector": "",
  "crawlOnlyPagesChangedSince": "",
  "customHttpHeaders": [],
  "excludeUrlGlobs": [],
  "excludeUrlRegexps": [],
  "htmlTransformer": "auto",
  "ignoreCanonicalUrl": false,
  "includeImages": false,
  "includeUrlGlobs": [],
  "includeUrlRegexps": [],
  "keepElementsCssSelector": "",
  "keepUrlFragments": false,
  "maxCrawlDepth": 1,
  "maxCrawlPages": 3,
  "outputFormat": "markdown",
  "parsePdfText": true,
  "removeElementsCssSelector": "",
  "respectRobotsTxtFile": true,
  "startUrls": [
    {
      "url": "https://docs.apify.com"
    }
  ],
  "storeSkippedUrls": false,
  "takeScreenshots": false,
  "useLlmsTxt": false,
  "useSitemaps": false
}

Download input.example.json and save your reviewed copy as input.json.

Field	Type	Required	Description
`startUrls`	array	required	One or more URLs to start crawling from. The crawler follows links found on these pages and stays on the domain of the first start URL.
`maxCrawlDepth`	integer	optional	How many links deep to follow from the start URLs. 0 means only the start URLs are crawled.
`maxCrawlPages`	integer	optional	Maximum number of pages (and files) to store. The crawl stops when this limit is reached.
`useSitemaps`	boolean	optional	Look for sitemaps (from robots.txt and /sitemap.xml) and enqueue the URLs they list. Finds pages that are not linked from the start URLs.
`useLlmsTxt`	boolean	optional	Look for /llms.txt at the start URL domains and enqueue the files listed there (common on documentation sites).
`respectRobotsTxtFile`	boolean	optional	Skip discovered pages that the site's robots.txt disallows. Start URLs are always crawled.
`crawlOnlyPagesChangedSince`	string	optional	Optional date (YYYY-MM-DD or full ISO date-time). Crawls only pages that may have changed since then and outputs a delta. Leave empty to crawl everything.
`includeUrlGlobs`	array	optional	Only crawl discovered URLs matching these glob patterns (e.g. https://example.com/docs/*). Leave empty to allow everything on the domain.
`excludeUrlGlobs`	array	optional	Skip discovered URLs matching any of these glob patterns (e.g. login, *.pdf).
`includeUrlRegexps`	array	optional	Only crawl discovered URLs matching at least one of these regular expressions. Combined with glob patterns (both must pass when both are set).
`excludeUrlRegexps`	array	optional	Skip discovered URLs matching any of these regular expressions.
`keepUrlFragments`	boolean	optional	Treat URLs that differ only in #fragment as different pages. Enable for single-page sites that use fragments as routes.
`ignoreCanonicalUrl`	boolean	optional	When disabled, pages whose canonical URL points to a page that was already crawled are skipped as duplicates.
`outputFormat`	string	optional	How page content is returned. Markdown keeps headings, lists, tables, and links — best for LLMs and RAG.
`htmlTransformer`	string	optional	How the main content is isolated from navigation, footers, ads, and cookie banners.
`includeImages`	boolean	optional	Add image references to the markdown field. Image URLs are always collected in the imageUrls field.
`chunkMarkdown`	boolean	optional	Emit a chunks array per page — heading-aware Markdown pieces ready for embeddings. Code blocks and tables are never split, each piece carries its heading path and character count, and consecutive pieces overlap so context is kept at the boundaries.
`chunkMaxChars`	integer	optional	Target size of each chunk when chunkMarkdown is on. Code blocks and tables stay whole even when larger than this.
`chunkOverlapChars`	integer	optional	Approximate overlap between consecutive chunks when chunkMarkdown is on, so context is not lost at chunk boundaries.
`keepElementsCssSelector`	string	optional	Keep only the elements matching this CSS selector (e.g. article, main .content) before extraction. Leave empty to use the whole page.
`removeElementsCssSelector`	string	optional	Remove elements matching this CSS selector (e.g. .sidebar, .ad, nav) before extraction.
`parsePdfText`	boolean	optional	When a crawled URL serves a PDF, extract its text into the text field as a file record instead of skipping it.
`clickElementsCssSelector`	string	optional	Click elements matching this CSS selector (e.g. [aria-expanded="false"]) before extraction to expand collapsed content.
`takeScreenshots`	boolean	optional	Render every stored page in the browser and capture a full-page JPEG (quality 80) of it, putting its public URL in the screenshot field. Screenshots are held in memory until pushed and add roughly 0.3-1 MB and a fraction of a second per page; screenshot capture always runs after the page is rendered.
`customHttpHeaders`	array	optional	Extra request headers for all engines and discovery requests, as a list of {"key", "value"} pairs (e.g. an Authorization token or a specific User-Agent for sites that ask for one). Transport/framing and cookie headers (Host, Cookie, Content-Length, Connection, ...) are rejected so the actor stays cookieless.
`storeSkippedUrls`	boolean	optional	Save a SKIPPED_URLS record to the key-value store listing skipped URLs grouped by reason (robots, blocked, non-HTML, errors, duplicates).

3. Start an Actor run

Set APIFY_TOKEN in your environment. Python 3.10 or newer is sufficient; this recipe uses no third-party packages.

Execution may incur charges. Review current pricing on Apify. The request asks for a 120-second timeout and a charge limit; these settings are subject to Apify’s pricing and do not guarantee an all-inclusive spending cap.

import json
import os
import urllib.request

# Save the schema-derived example as input.json; review and edit it first.
with open("input.json", encoding="utf-8") as stream:
    actor_input = json.load(stream)

token = os.environ["APIFY_TOKEN"]
request = urllib.request.Request(
    "https://api.apify.com/v2/acts/daEeIAgzVqlS9oXl8/runs?timeout=120&maxTotalChargeUsd=0.50",
    data=json.dumps(actor_input).encode("utf-8"),
    headers={"Authorization": "Bearer " + token, "Content-Type": "application/json"},
    method="POST",
)
# This starts a potentially billable run. Do not automatically retry this POST.
with urllib.request.urlopen(request, timeout=40) as response:
    run = json.load(response)["data"]
print("Run ID:", run["id"])
print("Status:", run["status"])
# Inspect and download results in Apify Console once the run succeeds.

4. Inspect the actual results

Open the run in Apify Console and wait for a successful completion. Use the published output schema to locate results and the dataset schema to understand declared views. Missing or optional record fields must not be treated as guaranteed values.

One record per crawled page: Markdown/text/HTML content, title, description, author, dates, canonical URL, links, images, OpenGraph, JSON-LD, word count, approximate token count, optional RAG chunks, optional screenshot URL, and content hash.

Questions before you run

Which inputs are required?

The published required fields are startUrls

What does this guide verify?

The fields, prefills, and schema links are rendered from the published build. This is a usage recipe, not a paid live extraction test. Source availability and collection results can vary.

What if the run request times out?

Check Apify Console before retrying. The first request may already have created a run. Do not automatically repeat a run-creation POST.

Open on Apify Full Actor reference

Generated from source build mlbNAEkzbrejmXih5 and its README and schemas. Source changes regenerate this guide; no third-party editorial claims are inferred.