Watercrawl Datasource Plugin for Dify

[blocked]

A powerful Dify datasource plugin that integrates with Watercrawl to recursively crawl websites and extract clean, LLM-ready content for AI applications.

🚀 Quick Start

Get API Key: Sign up at Watercrawl and get your API key
Install Plugin: Add the Watercrawl datasource in your Dify workspace
Configure: Enter your API key and start crawling websites
Extract: Get clean, structured content ready for your AI models

✨ Features

Feature	Description
🕷️ Recursive Crawling	Automatically discover and crawl linked pages
⚡ Fast Mode	Skip rendering for 3x faster crawling
📏 Depth Control	Limit crawl depth to control scope
🎯 Smart Extraction	Extract main content, skip navigation/ads
🔍 URL Filtering	Include/exclude specific URL patterns
📊 Progress Tracking	Real-time crawl status monitoring
📝 Clean Output	Markdown-formatted, LLM-optimized content
🏠 Self-Hosted	Use cloud service or your own instance

Setup

Prerequisites

Before using this plugin, you need:

A Watercrawl API key (for cloud service) or self-hosted Watercrawl instance
Target URLs ready for crawling
Understanding of your crawling requirements (depth, limits, patterns)

☁️ Cloud Setup (Recommended)

🏠 Self-Hosted Setup

📖 Usage Examples

📄 Single Page Extraction

Extract content from one specific page:

🌐 Full Website Crawl

Crawl an entire website systematically:

🎯 Targeted Section Crawl

Focus on specific website sections:

⚙️ Configuration Parameters

Parameter	Default	Description
Start URL	required	Base URL to begin crawling
Ignore rendering		Skip page rendering for faster crawling
URL patterns to exclude	-	Comma-separated exclude patterns:
URL patterns to include	-	Comma-separated include patterns:
Maximum crawl depth		How deep to crawl (1 = start URL only)
Maximum pages to crawl		Total page limit for the crawl job
Only main content		Extract main content, skip nav/footer
Proxy Server Slug	-	Proxy server identifier (optional)

🎯 Understanding Crawl Depth

🔍 URL Pattern Examples

Include Only Specific Sections:

Exclude Unwanted Areas:

📤 Output Format

Each crawled page returns structured data:

🔄 How It Works

Job Creation → Submit crawl request to Watercrawl API
Processing → Watercrawl crawls pages based on your parameters
Monitoring → Plugin polls job status every 5 seconds
Extraction → Content is cleaned and formatted as Markdown
Delivery → Structured results returned to Dify

🎯 Common Use Cases

📚 Documentation Indexing

Perfect for creating AI-powered knowledge bases:

📝 Content Analysis

Extract blog posts and articles for analysis:

🛍️ E-commerce Data

Gather product information systematically:

🔍 Competitor Research

Monitor competitor websites for changes:

💡 Best Practices

🧪 Start Small: Test with low limits first (, )
🎯 Use Filters: Focus crawling with include/exclude patterns
🤖 Respect Robots: Check before large crawls
⚡ Fast Mode: Use for speed
📄 Main Content: Enable for cleaner, AI-ready content
📊 Monitor: Watch crawl progress for large jobs
🔄 Iterate: Refine patterns based on results

⚡ Performance Notes

Scenario	Typical Time	Recommendation
Single page	< 5 seconds	Use for quick extractions
Small site (< 20 pages)	30-60 seconds	Good for testing patterns
Medium site (50-100 pages)	2-5 minutes	Monitor progress
Large site (> 200 pages)	5-15 minutes	Use during off-hours

💡 Tip: Deep crawling (depth > 3) can exponentially increase page count. Use cautiously!

🔧 Troubleshooting

❌ Authentication Errors

"API key is required" error:

✅ Verify API key in Dify plugin settings
✅ Check Base URL is correct ()
✅ Ensure API key starts with

🚫 Crawl Failures

"Failed to crawl" error:

✅ Test URL accessibility in browser
✅ Check if site blocks bots (robots.txt)
✅ Verify API key is valid and has credits
✅ Try with

📉 Incomplete Results

Missing expected pages:

✅ Verify URL patterns with pattern tester
✅ Check crawl depth covers target pages
✅ Increase page limit if needed
✅ Review site's internal linking structure

🐌 Performance Issues

Crawling too slow:

✅ Enable for 3x speed boost
✅ Use more specific URL patterns
✅ Reduce crawl depth and page limits
✅ Check your internet connection

📊 Rate Limits & Usage

☁️ Watercrawl Cloud

Free Plan: 1000 pages/month
Startup Plan: 10,000 pages/month
Pro Plan: 30,000 pages/month
Enterprise: Custom limits
Check pricing for more details
Monitor usage: Dashboard

🏠 Self-Hosted

No external rate limits
Performance depends on your server resources

🔒 Security & Privacy

🔐 Secure: All API calls use HTTPS encryption
🔑 API Keys: Store securely, never in client-side code
🏠 Self-Hosted: Full data control for sensitive content
📋 Review: Always check crawled content before use

📞 Support & Resources

🆘 Get Help

💬 Plugin Issues: support@watercrawl.dev
📖 Documentation: docs.watercrawl.dev
🐛 Bug Reports: GitHub Issues

📚 Learn More

Made with ❤️ by the Watercrawl Team

Website • Documentation • Dashboard