Web Crawler API

The Web Crawler API enables efficient web scraping and data extraction from websites, optimized for AI model training and automation workflows. It supports various output formats like Markdown and JSON, making it ideal for LLM-friendly data processing. 

Method: POST

URL: {{base_url}}/webcrawl

Authentication

Required Parameters:

Parameter Name	Value	Description
Parameter Name	Value	Description
client_id	your_client_id	API authentication ID

Required Headers:

Header Name	Value	Description
Header Name	Value	Description
client-secret	your_client_secret	API authentication secret
Content-Type	application/json	Data sent in JSON format

Input Formats

JSON Body:

1. Basic Crawl (Markdown Output, Default)

{
    "url": "https://example.com",
    "deep_crawl": "false",
    "max_pages": 10
}

2. Deep Crawl with BFS Strategy

{
    "url": "https://example.com",
    "deep_crawl": "true",
    "max_pages": 10
}

3. LLM Extraction with Specific Question

{
    "url": "https://example.com/products",
    "deep_crawl": "false",
    "question": "Extract all product prices"
}

Features of Web Crawler API

• Blazing Fast Performance: Outperforms many paid services.

• Output

  LLM-Friendly OutputsFormats: Supports JSON, cleaned HTML, and Markdown formats.

• Multi-URL Crawling: Crawl multiple URLs simultaneously.

• Media Extraction: Extracts images, audio, and video tags.

• Link Extraction: Captures internal and external links.

• Metadata Extraction: Retrieves metadata from web pages.

• Custom Hooks: Supports authentication, custom headers, and page modifications.

• User-Agent Customization: Allows custom user-agent settings.

• Screenshots: Takes screenshots of crawled pages.

• Custom JavaScript: Executes JavaScript before crawling.

Notes

• Basic Crawl: Performs a single-page crawl on the provided URL, ignoring max_pages.

• Deep Crawl: Crawls multiple pages using a BFS (Breadth-First Search) strategy,pages, respecting the max_pages limit.

• LLM Extraction: Use the question parameter to extract specific data, such as product prices, from the crawled content.

File Support

The Web Crawler API processes web pages and extracts data in the following formats:

• JSON

• Markdown

• Cleaned HTML

Contact

For Base URL, Client ID, Client Secret, or other queries related to the Web Crawler API, please reach out to our support team at:

Email: support@bizdata360.com