Guides

Crawl jobs

A crawl job processes one or more websites. Create one from a URL list, a CSV upload, or search discovery.

Create from URLs

bash

curl https://api.orbicrawl.com/api/v1/jobs \
  -H "X-API-Key: oc_live_..." -H "Content-Type: application/json" \
  -d '{ "name": "Q3 prospects", "urls": ["https://acme.com"], "config": {} }'

Create from a CSV

POST /jobs/csv as multipart/form-data with a file field. Include a column named url, website or domain.

bash

curl https://api.orbicrawl.com/api/v1/jobs/csv \
  -H "X-API-Key: oc_live_..." \
  -F "file=@prospects.csv" -F "name=Imported list"

Crawl configuration

Pass a config object. Values are clamped to your plan's limits.

Field	Default	Notes
`max_depth`	`1`	Link levels to follow (0 = entry page only)
`use_browser_rendering`	`false`	Render JavaScript in a headless browser
`respect_robots_txt`	`true`	Honor robots.txt
`max_retries`	`2`	Retries on transient failures (capped at 5)
`rate_limit_per_host_ms`	`1000`	Minimum delay between requests to a host
`rotate_user_agents`	`true`	Rotate the User-Agent per request
`incremental`	`false`	Skip duplicate leads for unchanged sites on re-crawl

Note

Turn on use_browser_rendering for single-page apps and JS-heavy sites where contact data only appears after scripts run.

Track progress

GET /jobs/{id} — current status and counts.
GET /jobs/{id}/events — a Server-Sent Events stream of live progress.
GET /jobs/{id}/leads — the leads extracted so far.

Credits are reserved when a job is created and any unused remainder is refunded when it finishes.

Search discovery

Don't have the URLs yet? Turn a query into candidate sites, then crawl them:

bash

curl -X POST https://api.orbicrawl.com/api/v1/discovery \
  -H "X-API-Key: oc_live_..." -H "Content-Type: application/json" \
  -d '{ "query": "dental clinics in Austin" }'