dify-datasource-litecrawl

A Dify datasource plugin that crawls a static documentation site or wiki and extracts
clean, per-page markdown — ready to feed a Dify Knowledge base.

It works with any site that exposes either a or navigable in-page links,
which covers the common static-site generators — VitePress, MkDocs, Docusaurus, Sphinx,
Hugo, Jekyll, GitBook, mdBook and similar — as well as MediaWiki wikis (crawled via
the MediaWiki API). Page content is converted to markdown with Trafilatura.

How it works

Choose a crawl :

auto (default) — detect the site type: if it is a MediaWiki, use its API; otherwise
use the sitemap when present, and fall back to following links breadth-first.
html — use the sitemap when present, otherwise a breadth-first link crawl.
mediawiki — enumerate pages via the MediaWiki API (, allpages) and render
each page via .

Parameters

(required) — documentation site URL, or a bare domain for auto-discovery.
— | | (default ).
— when a bare domain is given, probe common documentation paths (e.g. ).
— restrict the crawl to a path, e.g. .
/ — comma-separated glob patterns to include or skip.
— stop after N pages (default 50).
— link-follow depth from the start URL (default 5).

Development

The crawler logic (discovery, sitemap parsing, BFS link crawl, MediaWiki, and content
extraction) lives in the package, which is independent of the Dify SDK and
covered by unit tests with mocked network calls.

License

Repository