dify-datasource-ndjson

A Dify datasource plugin that loads a bulk NDJSON (JSON Lines) dump into a
Dify Knowledge base — one document per record — instead of crawling a live site.

It is generic: any HTTP-served NDJSON dump works — a bare /
, or a JSON manifest listing per-file compressed exports — via
configurable field mapping. See Sources for examples.

How it works

Give it a Source URL. The plugin auto-detects what it points at:

• A JSON manifest — a JSON object with an array. Each entry has a
  (relative or absolute) and optionally , , . The
  plugin filters by your / , downloads each selected
  export (gzip), optionally verifies its , decompresses it, and streams
  its records.
• A bare or a compressed — streamed
  directly. Compression is auto-detected (gzip / xz / bz2 / zstd) by magic bytes,
  falling back to the URL suffix.

Each NDJSON line is parsed as one JSON object and mapped to a document via the
parameters. Blank and malformed lines are skipped, never fatal. The
stream is decompressed lazily line-by-line, so a ~45 MB compressed export is
never fully materialised as decompressed text in memory.

Two-phase model ()

This is an -type datasource, which fits large bulk dumps far
better than (which would stream every record's full content into
the browser preview and time out on big sections). It works in two phases:

• List pages () — download and parse the source once, render each
  record, and return a light listing (page id + title, no content). Even
  20k records is a small message. The rendered content is written to a
  disk cache (SQLite, under the system temp dir) keyed by a hash of the
  datasource parameters.
• Get content () — Dify pulls content one page at a time
  during indexing; each call reads a single record from the disk cache.

The cache is on disk (not memory) because the plugin process may be recycled
between the two phases. The returned to Dify encodes the datasource
parameters, so on a cold cache (evicted / temp wiped) rebuilds the
cache once from the source, then serves every page from it — the dump is not
re-downloaded per page.

Metadata limitation

The ingested document exposes only a title and content — there is no free-form
metadata field. Any fields you list in are therefore inlined
as a compact one-line header at the top of the document content, e.g.:

This keeps the key metadata searchable/visible inside each document. A future
Dify SDK with a real metadata field would let this move out of the body.

Parameters

• (required) — URL of a manifest or an / .
• — manifest only: CSV of section values to keep (e.g. ).
• — manifest only: CSV of language values to keep (e.g. ).
• (default ) — record field → document source URL.
• (default ) — record field → document title.
• (default ) — record field → document body.
• — CSV of fields inlined as a header in content.
• + — ingest only records where the field equals
  the value (e.g. = , for deduping to canonical records).
  Booleans match / case-insensitively. Empty = no filter.
• — stop after N records. Empty = no cap.
• (default ) — manifest only: verify each export's
  before ingesting; a mismatch aborts with an error.

Credentials (all optional)

• — custom User-Agent header.
• — a Bearer token for a private dump (sent as ).
• — a full header value (overrides ).

The dump URL is a datasource parameter, not a credential, so credential
validation performs no network calls.

Sources

Anything that serves newline-delimited JSON over HTTP works. A few public examples:

• GH Archive — hourly gzipped NDJSON of every public GitHub event
  (); a bare , no manifest.
• OpenAlex — the scholarly-works snapshot ships as gzipped JSON Lines partitions
  listed by a manifest — the manifest mode below.
• Hugging Face datasets — many are published as / files at
  stable URLs.
• Your own exports — , Elasticsearch/OpenSearch /
  , BigQuery newline-delimited JSON, ClickHouse , or
  anything piped through all emit NDJSON.

Point at the file (or a manifest) and map its fields with the
parameters.

Example: manpages.altlinux.team

Development

The ingestion core (manifest parsing, gzip line-streaming, record→document
mapping, page-id assignment, and the disk cache) lives in the
package, which is independent of the Dify SDK so it can be unit-tested without
installed. is the thin SDK adapter.