n1 Web Crawler Worker API

This is the worker API for the n1 Web Crawler. It processes URLs from the queue and stores the results in KV storage.

How it works:

  1. Users submit a URL through the frontend at crawler-ui.nayaone.ai
  2. The URL is sent to a queue to be processed
  3. This worker processes the URL, capturing:
    • Screenshot of the page
    • Count of NayaOne.com links
    • Full HTML content
  4. Results are stored in KV storage and can be viewed at the results URL

API Usage:

To directly submit a URL to the queue, send a POST request with the URL as the body:

curl -X POST https://crawler.nayaone.ai -d "https://example.com"

For UI access, visit crawler-ui.nayaone.ai