Skip to main content

Web Crawler API

The Web Crawler API enables efficient web scraping and data extraction from websites, optimized for automation workflows. It supports output formats like Markdown and JSON, making it ideal for data processing.

Method: POST

URL: {{base_url}}/webcrawl

Authentication

Required Parameters:

Parameter Name Value Description
client_id your_client_id API authentication ID

Required Headers:

Header Name Value Description
client-secret your_client_secret API authentication secret
Content-Type application/json Data sent in JSON format

Input Formats

JSON Body:

  1. Basic Crawl (Markdown Output, Default)
{
    "url": "https://example.com",
    "deep_crawl": "false",
    "max_pages": 10
}
  1. Deep Crawl
{
    "url": "https://example.com",
    "deep_crawl": "true",
    "max_pages": 10
}

Features of Web Crawler API

  • Blazing Fast Performance: Outperforms many paid services.
  • Output Formats: Supports JSON, cleaned HTML, and Markdown formats.
  • Multi-URL Crawling: Crawl multiple URLs simultaneously.
  • Media Extraction: Extracts images, audio, and video tags.
  • Link Extraction: Captures internal and external links.
  • Metadata Extraction: Retrieves metadata from web pages.

Notes

  • Basic Crawl: Performs a single-page crawl on the provided URL, ignoring max_pages.
  • Deep Crawl: Crawls multiple pages, respecting the max_pages limit.

File Support

The Web Crawler API processes web pages and extracts data in the following formats:

  • JSON
  • Markdown
  • Cleaned HTML

Contact

For Base URL, Client ID, Client Secret, or other queries related to the Web Crawler API, please reach out to our support team at:

Email: support@bizdata360.com

Back to top