Developers
Sign in
Developers

CrawlMe API

Submit URLs for indexing, check their status and track whole batches from your own code. One API key works with both API versions, against the same account, balance and submission history.

Choose a version

API v2 Recommended

The latest API, for new integrations. Clean URLs, one consistent response shape, machine-readable error codes, batch tracking and safe retries.

  • Batch summary and per-batch submission list
  • Optional Idempotency-Key header for retry-safe submissions
  • Specific error codes with HTTP statuses that match (402, 409, 422, 429, 503)
  • Rate-limit headers and Retry-After
  • ISO 8601 timestamps, accepted/invalid/duplicate counts, remaining balance
Read the v2 reference OpenAPI spec

API v1 Supported

The original API. Existing integrations keep working exactly as they do today, with the same keys, paths, fields and limits. Nothing needs to change.

  • Balance, submit, one submission, paginated list
  • Same account, balance and submission records as v2
  • Same 60 requests/minute account limit, shared with v2
Read the v1 reference Migration guide
Already using API v1? Your integration will continue working. Switching to v2 is completely optional, and the migration guide shows exactly what differs.

Quick start

Get an API key from your dashboard (API Access in the sidebar; an Admin may need to enable API access for your account first). Then:

1. Check your balance

cURL
curl -X GET "https://crawlme.net/api/v2/balance" \
  -H "Authorization: Bearer YOUR_API_KEY"
PHP
<?php
$ch = curl_init("https://crawlme.net/api/v2/balance");
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST  => 'GET',
    CURLOPT_HTTPHEADER     => [
    'Authorization: Bearer YOUR_API_KEY',
    ],
]);
$response = json_decode(curl_exec($ch), true);
$status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);

if ($response['success']) {
    print_r($response['data']);
} else {
    echo $status . ' ' . $response['error']['code'] . ': ' . $response['error']['message'];
}
Python
import requests

response = requests.get(
    "https://crawlme.net/api/v2/balance",
    headers={
    "Authorization": "Bearer YOUR_API_KEY",
    },
    timeout=30,
)
body = response.json()

if body["success"]:
    print(body["data"])
else:
    print(response.status_code, body["error"]["code"], body["error"]["message"])
JavaScript
const response = await fetch("https://crawlme.net/api/v2/balance", {
  method: "GET",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
  },
});
const body = await response.json();

if (body.success) {
  console.log(body.data);
} else {
  console.error(response.status, body.error.code, body.error.message);
}
JSON
{
    "success": true,
    "data": {
        "account_id": 1042,
        "balance": 150.0
    }
}

2. Submit URLs

Send at least 1 URLs. Add an Idempotency-Key so a retried request can never be charged twice.

cURL
curl -X POST "https://crawlme.net/api/v2/submissions" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Idempotency-Key: order-7f3a9c" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com/page-1","https://example.com/page-2"]}'
PHP
<?php
$ch = curl_init("https://crawlme.net/api/v2/submissions");
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST  => 'POST',
    CURLOPT_HTTPHEADER     => [
    'Authorization: Bearer YOUR_API_KEY',
    'Idempotency-Key: order-7f3a9c',
    'Content-Type: application/json',
    ],
    CURLOPT_POSTFIELDS     => json_encode(['urls' => ['https://example.com/page-1', 'https://example.com/page-2']]),
]);
$response = json_decode(curl_exec($ch), true);
$status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);

if ($response['success']) {
    print_r($response['data']);
} else {
    echo $status . ' ' . $response['error']['code'] . ': ' . $response['error']['message'];
}
Python
import requests

response = requests.post(
    "https://crawlme.net/api/v2/submissions",
    headers={
    "Authorization": "Bearer YOUR_API_KEY",
    "Idempotency-Key": "order-7f3a9c",
    },
    json={"urls":["https://example.com/page-1","https://example.com/page-2"]},
    timeout=30,
)
body = response.json()

if body["success"]:
    print(body["data"])
else:
    print(response.status_code, body["error"]["code"], body["error"]["message"])
JavaScript
const response = await fetch("https://crawlme.net/api/v2/submissions", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Idempotency-Key": "order-7f3a9c",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({"urls":["https://example.com/page-1","https://example.com/page-2"]}),
});
const body = await response.json();

if (body.success) {
  console.log(body.data);
} else {
  console.error(response.status, body.error.code, body.error.message);
}
JSON
{
    "success": true,
    "data": {
        "batch_id": 293244,
        "replayed": false,
        "accepted_count": 2,
        "invalid_count": 0,
        "duplicate_count": 0,
        "credit_cost": 2.0,
        "remaining_balance": 148.0,
        "invalid": [],
        "duplicates": [],
        "submission_ids": [
            8812001,
            8812002
        ],
        "batch": {
            "id": 293244,
            "account_id": 1042,
            "status": "PENDING",
            "total": 2,
            "successful": 0,
            "failed": 0,
            "pending": 2,
            "queued": 2,
            "processing": 0,
            "credit_cost": 2.0,
            "created_at": "2026-10-23T09:15:02Z",
            "completed_at": null
        }
    }
}

3. Track the batch

cURL
curl -X GET "https://crawlme.net/api/v2/batches/293244" \
  -H "Authorization: Bearer YOUR_API_KEY"
PHP
<?php
$ch = curl_init("https://crawlme.net/api/v2/batches/293244");
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST  => 'GET',
    CURLOPT_HTTPHEADER     => [
    'Authorization: Bearer YOUR_API_KEY',
    ],
]);
$response = json_decode(curl_exec($ch), true);
$status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);

if ($response['success']) {
    print_r($response['data']);
} else {
    echo $status . ' ' . $response['error']['code'] . ': ' . $response['error']['message'];
}
Python
import requests

response = requests.get(
    "https://crawlme.net/api/v2/batches/293244",
    headers={
    "Authorization": "Bearer YOUR_API_KEY",
    },
    timeout=30,
)
body = response.json()

if body["success"]:
    print(body["data"])
else:
    print(response.status_code, body["error"]["code"], body["error"]["message"])
JavaScript
const response = await fetch("https://crawlme.net/api/v2/batches/293244", {
  method: "GET",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
  },
});
const body = await response.json();

if (body.success) {
  console.log(body.data);
} else {
  console.error(response.status, body.error.code, body.error.message);
}

Poll every few minutes until pending reaches 0. The v2 reference explains every field and status.

How it works

  1. Validation. Each URL must be a well-formed http or https URL of at most 2048 characters. Invalid entries are returned to you and skipped.
  2. Duplicate detection. A URL repeated in the same request, or submitted by your account in the last 1 minutes, is skipped and listed as a duplicate. Nothing is charged for it.
  3. Charging. Accepted URLs cost 1 credit each, deducted up front in one atomic step. If your balance does not cover all accepted URLs, nothing is charged and nothing is submitted.
  4. Queueing. Accepted URLs are stored as PENDING submissions and handed to an indexing provider by a background worker within a few minutes. The API never waits on a provider.
  5. Status. Submissions move to PROCESSING once a provider accepts them, then SUCCESS or FAILED. Failed submissions are refunded automatically.

What SUCCESS means. The indexing provider accepted and processed the URL. CrawlMe does not verify whether Google has indexed the page; indexing is decided by search engines and can take time or not happen at all.