URL Indexing API v2 Reference | CrawlMe Developers

Overview

API v2 is a JSON-over-HTTPS API. Six endpoints:

Method	Path	Purpose
GET	/balance	Your credit balance
POST	/submissions	Submit URLs (creates a batch)
GET	/submissions	List submissions, paginated, optionally by status
GET	/submissions/{id}	One submission
GET	/batches/{id}	Batch summary with counts per state
GET	/batches/{id}/submissions	The submissions in a batch, paginated

Every response has one of two shapes:

JSON

{"success": true,  "data": { ... }}
{"success": false, "error": {"code": "ERROR_CODE", "message": "Human-readable explanation", "details": { ... }}}

error.details is present only when there is something actionable to add (the invalid URLs, the field at fault, the batch an idempotency key already belongs to).

Conventions: timestamps are UTC in ISO 8601 with a Z suffix (2026-10-23T09:15:02Z); money fields (balance, credit_cost, credits_used) are JSON numbers with up to two decimals; ids are integers; nullable fields are stated per endpoint. Responses never include which indexing provider handled a URL.

A machine-readable OpenAPI 3.1 description of everything on this page is available.

Base URL

https://crawlme.net/api/v2

Paths have no .php suffix. A trailing slash is tolerated. HTTPS only.

Authentication

Send your API key as a Bearer token on every request:

HTTP

Authorization: Bearer YOUR_API_KEY

Keys start with uidx_. They are the same keys API v1 uses. Never put a key in a query string. A request without a valid key gets 401 UNAUTHENTICATED; a valid key for an account whose API access an Admin has not enabled gets 403 FORBIDDEN.

API key management

Create or view your key on the API Access page of your dashboard. API access must be enabled for your account by an Admin; the page lets you request it.
One active key per account. Regenerating creates a new key and revokes the old one immediately; Revoke disables API access until you generate a new key.
The raw key is shown once, at creation. Store it in a secret store or environment variable, not in source code.
Revoking or regenerating affects API v1 and v2 at the same moment.

Quick start

Three calls: check your balance, submit two URLs with an idempotency key, then read the batch.

Balance

cURLPHPPythonJavaScript

cURL

curl -X GET "https://crawlme.net/api/v2/balance" \
  -H "Authorization: Bearer YOUR_API_KEY"

PHP

<?php
$ch = curl_init("https://crawlme.net/api/v2/balance");
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST  => 'GET',
    CURLOPT_HTTPHEADER     => [
    'Authorization: Bearer YOUR_API_KEY',
    ],
]);
$response = json_decode(curl_exec($ch), true);
$status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);

if ($response['success']) {
    print_r($response['data']);
} else {
    echo $status . ' ' . $response['error']['code'] . ': ' . $response['error']['message'];
}

Python

import requests

response = requests.get(
    "https://crawlme.net/api/v2/balance",
    headers={
    "Authorization": "Bearer YOUR_API_KEY",
    },
    timeout=30,
)
body = response.json()

if body["success"]:
    print(body["data"])
else:
    print(response.status_code, body["error"]["code"], body["error"]["message"])

JavaScript

const response = await fetch("https://crawlme.net/api/v2/balance", {
  method: "GET",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
  },
});
const body = await response.json();

if (body.success) {
  console.log(body.data);
} else {
  console.error(response.status, body.error.code, body.error.message);
}

Submit

cURLPHPPythonJavaScript

cURL

curl -X POST "https://crawlme.net/api/v2/submissions" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Idempotency-Key: order-7f3a9c" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com/page-1","https://example.com/page-2"]}'

PHP

<?php
$ch = curl_init("https://crawlme.net/api/v2/submissions");
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST  => 'POST',
    CURLOPT_HTTPHEADER     => [
    'Authorization: Bearer YOUR_API_KEY',
    'Idempotency-Key: order-7f3a9c',
    'Content-Type: application/json',
    ],
    CURLOPT_POSTFIELDS     => json_encode(['urls' => ['https://example.com/page-1', 'https://example.com/page-2']]),
]);
$response = json_decode(curl_exec($ch), true);
$status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);

if ($response['success']) {
    print_r($response['data']);
} else {
    echo $status . ' ' . $response['error']['code'] . ': ' . $response['error']['message'];
}

Python

import requests

response = requests.post(
    "https://crawlme.net/api/v2/submissions",
    headers={
    "Authorization": "Bearer YOUR_API_KEY",
    "Idempotency-Key": "order-7f3a9c",
    },
    json={"urls":["https://example.com/page-1","https://example.com/page-2"]},
    timeout=30,
)
body = response.json()

if body["success"]:
    print(body["data"])
else:
    print(response.status_code, body["error"]["code"], body["error"]["message"])

JavaScript

const response = await fetch("https://crawlme.net/api/v2/submissions", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Idempotency-Key": "order-7f3a9c",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({"urls":["https://example.com/page-1","https://example.com/page-2"]}),
});
const body = await response.json();

if (body.success) {
  console.log(body.data);
} else {
  console.error(response.status, body.error.code, body.error.message);
}

Batch status

cURLPHPPythonJavaScript

cURL

curl -X GET "https://crawlme.net/api/v2/batches/293244" \
  -H "Authorization: Bearer YOUR_API_KEY"

PHP

<?php
$ch = curl_init("https://crawlme.net/api/v2/batches/293244");
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST  => 'GET',
    CURLOPT_HTTPHEADER     => [
    'Authorization: Bearer YOUR_API_KEY',
    ],
]);
$response = json_decode(curl_exec($ch), true);
$status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);

if ($response['success']) {
    print_r($response['data']);
} else {
    echo $status . ' ' . $response['error']['code'] . ': ' . $response['error']['message'];
}

Python

import requests

response = requests.get(
    "https://crawlme.net/api/v2/batches/293244",
    headers={
    "Authorization": "Bearer YOUR_API_KEY",
    },
    timeout=30,
)
body = response.json()

if body["success"]:
    print(body["data"])
else:
    print(response.status_code, body["error"]["code"], body["error"]["message"])

JavaScript

const response = await fetch("https://crawlme.net/api/v2/batches/293244", {
  method: "GET",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
  },
});
const body = await response.json();

if (body.success) {
  console.log(body.data);
} else {
  console.error(response.status, body.error.code, body.error.message);
}

Error handling, rate limits and a complete retry loop are covered under Retry handling.

Account balance

GET`/api/v2/balance`#

Returns the current credit balance of the account that owns the key.

Headers: Authorization (required). No parameters, no body.

cURLPHPPythonJavaScript

cURL

curl -X GET "https://crawlme.net/api/v2/balance" \
  -H "Authorization: Bearer YOUR_API_KEY"

PHP

<?php
$ch = curl_init("https://crawlme.net/api/v2/balance");
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST  => 'GET',
    CURLOPT_HTTPHEADER     => [
    'Authorization: Bearer YOUR_API_KEY',
    ],
]);
$response = json_decode(curl_exec($ch), true);
$status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);

if ($response['success']) {
    print_r($response['data']);
} else {
    echo $status . ' ' . $response['error']['code'] . ': ' . $response['error']['message'];
}

Python

import requests

response = requests.get(
    "https://crawlme.net/api/v2/balance",
    headers={
    "Authorization": "Bearer YOUR_API_KEY",
    },
    timeout=30,
)
body = response.json()

if body["success"]:
    print(body["data"])
else:
    print(response.status_code, body["error"]["code"], body["error"]["message"])

JavaScript

const response = await fetch("https://crawlme.net/api/v2/balance", {
  method: "GET",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
  },
});
const body = await response.json();

if (body.success) {
  console.log(body.data);
} else {
  console.error(response.status, body.error.code, body.error.message);
}

Response 200:

JSON

{
    "success": true,
    "data": {
        "account_id": 1042,
        "balance": 150.0
    }
}

Field	Type	Meaning
account_id	integer	Your account id (useful when several keys or accounts are in play).
balance	number	Credits available right now, two decimals.

Errors: 401, 403, 429, 503 (see Error codes).

Submitting URLs

POST`/api/v2/submissions`#

Validates the URLs, skips invalid and duplicate ones, charges 1 credit per accepted URL in one atomic step, creates a batch and queues the accepted URLs for a provider. The call returns immediately; providers are contacted by a background worker within a few minutes.

Headers: Authorization (required), Content-Type: application/json (required), Idempotency-Key (optional, strongly recommended, see Idempotency).

Body:

JSON

{
    "urls": [
        "https://example.com/page-1",
        "https://example.com/page-2",
        "not-a-url"
    ]
}

Field	Type	Rules
urls	string[]	Required. A JSON array (not an object) of strings, one URL per entry (an entry containing a newline or other control character is rejected). Leading and trailing whitespace is trimmed; blank entries are ignored. At least 1 valid URLs must remain after validation and duplicate removal, and the array may hold at most 1,000 entries. Any non-string entry makes the whole request fail with 422.

cURLPHPPythonJavaScript

cURL

curl -X POST "https://crawlme.net/api/v2/submissions" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Idempotency-Key: order-7f3a9c" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com/page-1","https://example.com/page-2","not-a-url"]}'

PHP

<?php
$ch = curl_init("https://crawlme.net/api/v2/submissions");
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST  => 'POST',
    CURLOPT_HTTPHEADER     => [
    'Authorization: Bearer YOUR_API_KEY',
    'Idempotency-Key: order-7f3a9c',
    'Content-Type: application/json',
    ],
    CURLOPT_POSTFIELDS     => json_encode(['urls' => ['https://example.com/page-1', 'https://example.com/page-2', 'not-a-url']]),
]);
$response = json_decode(curl_exec($ch), true);
$status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);

if ($response['success']) {
    print_r($response['data']);
} else {
    echo $status . ' ' . $response['error']['code'] . ': ' . $response['error']['message'];
}

Python

import requests

response = requests.post(
    "https://crawlme.net/api/v2/submissions",
    headers={
    "Authorization": "Bearer YOUR_API_KEY",
    "Idempotency-Key": "order-7f3a9c",
    },
    json={"urls":["https://example.com/page-1","https://example.com/page-2","not-a-url"]},
    timeout=30,
)
body = response.json()

if body["success"]:
    print(body["data"])
else:
    print(response.status_code, body["error"]["code"], body["error"]["message"])

JavaScript

const response = await fetch("https://crawlme.net/api/v2/submissions", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Idempotency-Key": "order-7f3a9c",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({"urls":["https://example.com/page-1","https://example.com/page-2","not-a-url"]}),
});
const body = await response.json();

if (body.success) {
  console.log(body.data);
} else {
  console.error(response.status, body.error.code, body.error.message);
}

Response 201 Created (or 200 OK when an Idempotency-Key replays an earlier request):

JSON

{
    "success": true,
    "data": {
        "batch_id": 293244,
        "replayed": false,
        "accepted_count": 2,
        "invalid_count": 1,
        "duplicate_count": 0,
        "credit_cost": 2.0,
        "remaining_balance": 148.0,
        "invalid": [
            "not-a-url"
        ],
        "duplicates": [],
        "submission_ids": [
            8812001,
            8812002
        ],
        "batch": {
            "id": 293244,
            "account_id": 1042,
            "status": "PENDING",
            "total": 2,
            "successful": 0,
            "failed": 0,
            "pending": 2,
            "queued": 2,
            "processing": 0,
            "credit_cost": 2.0,
            "created_at": "2026-10-23T09:15:02Z",
            "completed_at": null
        }
    }
}

Field	Type	Meaning
batch_id	integer	The new batch. Pass it to /batches/{id}.
replayed	boolean	true when this response is a replay of an earlier request with the same Idempotency-Key. Nothing was charged or queued by this call.
accepted_count	integer	URLs accepted, charged and queued.
invalid_count	integer	Entries rejected as not well-formed URLs.
duplicate_count	integer	Entries skipped as duplicates (within the request or the 1-minute window).
credit_cost	number	Credits charged by this request: accepted_count × 1. 0 on replay only in the sense that nothing new was charged; the value shown is the original charge.
remaining_balance	number	Your balance after this request.
invalid	string[]	The rejected entries, exactly as you sent them.
duplicates	string[]	The skipped URLs in their normalized form. Empty on a replay (see Idempotency).
submission_ids	integer[]	One id per accepted URL, in the order they were accepted. Use with /submissions/{id}.
batch	object \| null	The batch summary, same shape as GET /batches/{id}.

Partial acceptance. If some entries are invalid or duplicates and at least 1 valid URLs remain, the request succeeds with 201 and tells you what was skipped. If fewer remain, it fails with 422 and nothing is charged.

Errors: 400 INVALID_JSON, 400 VALIDATION_ERROR (bad Idempotency-Key), 413 PAYLOAD_TOO_LARGE (body over 4 MB), 422 VALIDATION_ERROR, 422 BATCH_LIMIT_EXCEEDED, 422 NO_VALID_URLS, 402 INSUFFICIENT_CREDITS, 409 IDEMPOTENCY_CONFLICT, 503 PROVIDER_UNAVAILABLE, 500 INTERNAL_ERROR, plus 401/403/429/503 MAINTENANCE. On every error response from this endpoint nothing was charged and nothing was queued.

URL validation rules

Scheme must be http or https; a host is required.
At most 2048 characters after normalization.
Non-ASCII characters in the path or query are percent-encoded for you; the normalized URL is what gets stored, submitted and shown in duplicates.
Entries that fail are returned verbatim in invalid and never charged.

The rules are the same ones the dashboard and API v1 apply.

Duplicate detection

A URL is a duplicate when its normalized form matches another URL in the same request (the first occurrence is kept) or any URL your account submitted in the last 1 minutes, through any version of the API or the dashboard. Duplicates are skipped, listed in duplicates, and never charged. Different accounts never affect each other's duplicate detection.

The window is a platform setting and may change; the value shown here is the live one.

Credit deduction behavior

Cost is 1 credit per accepted URL, charged in a single database transaction together with the batch and submission records. Either everything is charged and queued, or nothing is.
If your balance is lower than the total for all accepted URLs, the request fails with 402 INSUFFICIENT_CREDITS. There is no partial charge for "as many as you can afford".
A submission that ends FAILED (or CANCELLED) is refunded automatically to the same account. The refund appears in your transaction history.
Every error response from POST /submissions means nothing was charged. The only way credits move is a 201.
If the connection drops after the server committed the charge but before you read the 201, you were charged. Use an Idempotency-Key so your retry returns the original result instead of charging again.

Batch status tracking

GET`/api/v2/batches/{id}`#

One summary for a batch you may see (your own, or a Customer's if you hold a Reseller key). Totals are read from durable counters that are kept after individual submissions age out, so a batch summary is available indefinitely.

cURLPHPPythonJavaScript

cURL

curl -X GET "https://crawlme.net/api/v2/batches/293244" \
  -H "Authorization: Bearer YOUR_API_KEY"

PHP

<?php
$ch = curl_init("https://crawlme.net/api/v2/batches/293244");
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST  => 'GET',
    CURLOPT_HTTPHEADER     => [
    'Authorization: Bearer YOUR_API_KEY',
    ],
]);
$response = json_decode(curl_exec($ch), true);
$status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);

if ($response['success']) {
    print_r($response['data']);
} else {
    echo $status . ' ' . $response['error']['code'] . ': ' . $response['error']['message'];
}

Python

import requests

response = requests.get(
    "https://crawlme.net/api/v2/batches/293244",
    headers={
    "Authorization": "Bearer YOUR_API_KEY",
    },
    timeout=30,
)
body = response.json()

if body["success"]:
    print(body["data"])
else:
    print(response.status_code, body["error"]["code"], body["error"]["message"])

JavaScript

const response = await fetch("https://crawlme.net/api/v2/batches/293244", {
  method: "GET",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
  },
});
const body = await response.json();

if (body.success) {
  console.log(body.data);
} else {
  console.error(response.status, body.error.code, body.error.message);
}

Response 200:

JSON

{
    "success": true,
    "data": {
        "id": 293244,
        "account_id": 1042,
        "status": "PROCESSING",
        "total": 500,
        "successful": 440,
        "failed": 10,
        "pending": 50,
        "queued": 20,
        "processing": 30,
        "credit_cost": 500.0,
        "created_at": "2026-10-23T09:15:02Z",
        "completed_at": null
    }
}

Field	Type	Meaning
id	integer	Batch id.
account_id	integer	Owning account.
status	string	PENDING (nothing sent yet), PROCESSING (some rows still queued or at a provider), COMPLETED (all SUCCESS), PARTIAL (some SUCCESS, some FAILED), FAILED (all FAILED), CANCELLED (reserved).
total	integer	Accepted URLs in the batch. Invalid and duplicate entries were never part of it.
successful	integer	Rows in SUCCESS.
failed	integer	Rows in FAILED or CANCELLED (both refunded).
pending	integer	Rows not finished yet: queued + processing.
queued	integer	Rows in PENDING (waiting for the background worker).
processing	integer	Rows in PROCESSING (accepted by a provider).
credit_cost	number	Credits charged when the batch was created. Refunds for failed rows are not subtracted here; see your balance or transaction history.
created_at	string	ISO 8601 UTC.
completed_at	string \| null	Set once pending reaches 0.

total = successful + failed + queued + processing always holds. Batch counters are refreshed by the background worker after each pass, so a figure can lag a status change by a few minutes.

Errors: 404 NOT_FOUND for an unknown id or one you may not see (same response on purpose), plus 401/403/429/503.

GET`/api/v2/batches/{id}/submissions`#

The submissions of a batch, oldest first, paginated. Same ownership rule as the summary.

Query parameter	Default	Meaning
page	1	Page number from 1.
per_page	25	1 to 100.

cURLPHPPythonJavaScript

cURL

curl -X GET "https://crawlme.net/api/v2/batches/293244/submissions?per_page=50&page=1" \
  -H "Authorization: Bearer YOUR_API_KEY"

PHP

<?php
$ch = curl_init("https://crawlme.net/api/v2/batches/293244/submissions?per_page=50&page=1");
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST  => 'GET',
    CURLOPT_HTTPHEADER     => [
    'Authorization: Bearer YOUR_API_KEY',
    ],
]);
$response = json_decode(curl_exec($ch), true);
$status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);

if ($response['success']) {
    print_r($response['data']);
} else {
    echo $status . ' ' . $response['error']['code'] . ': ' . $response['error']['message'];
}

Python

import requests

response = requests.get(
    "https://crawlme.net/api/v2/batches/293244/submissions?per_page=50&page=1",
    headers={
    "Authorization": "Bearer YOUR_API_KEY",
    },
    timeout=30,
)
body = response.json()

if body["success"]:
    print(body["data"])
else:
    print(response.status_code, body["error"]["code"], body["error"]["message"])

JavaScript

const response = await fetch("https://crawlme.net/api/v2/batches/293244/submissions?per_page=50&page=1", {
  method: "GET",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
  },
});
const body = await response.json();

if (body.success) {
  console.log(body.data);
} else {
  console.error(response.status, body.error.code, body.error.message);
}

JSON

{
    "success": true,
    "data": {
        "batch_id": 293244,
        "submissions": [
            {
                "id": 8812001,
                "batch_id": 293244,
                "account_id": 1042,
                "url": "https://example.com/page-1",
                "status": "PROCESSING",
                "credits_used": 1.0,
                "submitted_at": "2026-10-23T09:15:02Z",
                "updated_at": "2026-10-23T09:18:40Z",
                "completed_at": null
            }
        ],
        "pagination": {
            "page": 1,
            "per_page": 50,
            "total": 500,
            "total_pages": 10
        }
    }
}

Finished submissions (SUCCESS, FAILED, CANCELLED) are deleted after 14 days. After that the batch summary still answers, but this list and /submissions/{id} return fewer rows or 404 for those ids. Fetch per-URL results while the batch is fresh if you need to keep them.

Individual submission status

GET`/api/v2/submissions/{id}`#

cURLPHPPythonJavaScript

cURL

curl -X GET "https://crawlme.net/api/v2/submissions/8812001" \
  -H "Authorization: Bearer YOUR_API_KEY"

PHP

<?php
$ch = curl_init("https://crawlme.net/api/v2/submissions/8812001");
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST  => 'GET',
    CURLOPT_HTTPHEADER     => [
    'Authorization: Bearer YOUR_API_KEY',
    ],
]);
$response = json_decode(curl_exec($ch), true);
$status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);

if ($response['success']) {
    print_r($response['data']);
} else {
    echo $status . ' ' . $response['error']['code'] . ': ' . $response['error']['message'];
}

Python

import requests

response = requests.get(
    "https://crawlme.net/api/v2/submissions/8812001",
    headers={
    "Authorization": "Bearer YOUR_API_KEY",
    },
    timeout=30,
)
body = response.json()

if body["success"]:
    print(body["data"])
else:
    print(response.status_code, body["error"]["code"], body["error"]["message"])

JavaScript

const response = await fetch("https://crawlme.net/api/v2/submissions/8812001", {
  method: "GET",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
  },
});
const body = await response.json();

if (body.success) {
  console.log(body.data);
} else {
  console.error(response.status, body.error.code, body.error.message);
}

JSON

{
    "success": true,
    "data": {
        "id": 8812001,
        "batch_id": 293244,
        "account_id": 1042,
        "url": "https://example.com/page-1",
        "status": "PROCESSING",
        "credits_used": 1.0,
        "submitted_at": "2026-10-23T09:15:02Z",
        "updated_at": "2026-10-23T09:18:40Z",
        "completed_at": null
    }
}

Field	Type	Meaning
id	integer	Submission id.
batch_id	integer \| null	Batch the URL was submitted in.
account_id	integer	Owning account (your own, or your Customer's for Reseller keys).
url	string	The normalized URL that was submitted.
status	string	PENDING, PROCESSING, SUCCESS, FAILED or CANCELLED (below).
credits_used	number	Credits charged for this URL (refunded if it ends FAILED).
submitted_at	string	When you submitted it. ISO 8601 UTC.
updated_at	string	Last status change.
completed_at	string \| null	Set when the status becomes SUCCESS, FAILED or CANCELLED.

Status values

status	Meaning	Final?
PENDING	Accepted and charged. Waiting for the background worker to hand it to an indexing provider (normally within minutes). If the queue is stalled for longer than a platform limit, the row is failed and refunded instead of being sent late.	No
PROCESSING	An indexing provider has accepted the URL and is working on it. Can last from minutes to a day or more depending on the provider.	No
SUCCESS	The indexing provider reported that it processed the URL. CrawlMe does not check Google (or any search engine) itself: SUCCESS does not mean the page is indexed.	Yes
FAILED	The provider rejected it, or sending failed after several attempts. The credit was refunded.	Yes
CANCELLED	Reserved for a submission cancelled before processing. Treated like FAILED (refunded). Not produced by normal API use today.	Yes

Errors: 404 NOT_FOUND (unknown, purged, or not yours), plus 401/403/429/503.

Listing submissions

GET`/api/v2/submissions`#

Your submissions (and your Customers', for Reseller keys), newest first.

Query parameter	Default	Meaning
status	(none)	One of PENDING, PROCESSING, SUCCESS, FAILED, CANCELLED (case-insensitive). Any other value is 422 VALIDATION_ERROR.
page	1	Page number from 1.
per_page	25	1 to 100; values outside the range are clamped. page is clamped to 10,000 (use /batches/{id}/submissions to walk one batch).

cURLPHPPythonJavaScript

cURL

curl -X GET "https://crawlme.net/api/v2/submissions?status=FAILED&page=1&per_page=25" \
  -H "Authorization: Bearer YOUR_API_KEY"

PHP

<?php
$ch = curl_init("https://crawlme.net/api/v2/submissions?status=FAILED&page=1&per_page=25");
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST  => 'GET',
    CURLOPT_HTTPHEADER     => [
    'Authorization: Bearer YOUR_API_KEY',
    ],
]);
$response = json_decode(curl_exec($ch), true);
$status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);

if ($response['success']) {
    print_r($response['data']);
} else {
    echo $status . ' ' . $response['error']['code'] . ': ' . $response['error']['message'];
}

Python

import requests

response = requests.get(
    "https://crawlme.net/api/v2/submissions?status=FAILED&page=1&per_page=25",
    headers={
    "Authorization": "Bearer YOUR_API_KEY",
    },
    timeout=30,
)
body = response.json()

if body["success"]:
    print(body["data"])
else:
    print(response.status_code, body["error"]["code"], body["error"]["message"])

JavaScript

const response = await fetch("https://crawlme.net/api/v2/submissions?status=FAILED&page=1&per_page=25", {
  method: "GET",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
  },
});
const body = await response.json();

if (body.success) {
  console.log(body.data);
} else {
  console.error(response.status, body.error.code, body.error.message);
}

JSON

{
    "success": true,
    "data": {
        "submissions": [
            {
                "id": 8812001,
                "batch_id": 293244,
                "account_id": 1042,
                "url": "https://example.com/page-1",
                "status": "FAILED",
                "credits_used": 1.0,
                "submitted_at": "2026-10-23T09:15:02Z",
                "updated_at": "2026-10-23T09:18:40Z",
                "completed_at": "2026-10-23T12:00:00Z"
            }
        ],
        "pagination": {
            "page": 1,
            "per_page": 25,
            "total": 1,
            "total_pages": 1
        }
    }
}

Pagination

Every list carries a pagination object:

JSON

{
    "page": 2,
    "per_page": 25,
    "total": 137,
    "total_pages": 6
}

total_pages is never below 1. Asking for a page past the end returns an empty submissions array with status 200. New rows arriving while you page may shift results between pages; for a stable walk through one batch use /batches/{id}/submissions, which is ordered by id ascending.

Rate limiting

60 requests per minute per account, over a sliding 60-second window. Requests to API v1 and API v2 share this one allowance; regenerating a key does not reset it. Rejected requests do not count.

Every response that could be attributed to an account carries:

HTTP

X-RateLimit-Limit: 60
X-RateLimit-Remaining: 42
X-RateLimit-Reset: 1792345678

Header	Meaning
X-RateLimit-Limit	Requests allowed per 60-second window.
X-RateLimit-Remaining	Requests you can still make before the oldest counted request leaves the window.
X-RateLimit-Reset	Unix time (seconds) when the oldest counted request leaves the window, i.e. when Remaining increases by at least one.

When the limit is exceeded:

HTTP

HTTP/1.1 429 Too Many Requests
Retry-After: 17
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1792345695

{"success": false, "error": {"code": "RATE_LIMITED", "message": "Too many requests. Please slow down."}}

Retry-After is in seconds. A rate-limited request did nothing, so it is always safe to repeat it after waiting.

Idempotency

Add an Idempotency-Key header to POST /submissions to make the request safe to repeat. Choose a value that is unique per logical submission, for example a UUID or your own order id, and send the exact same value on every retry of that submission.

HTTP

POST /api/v2/submissions
Authorization: Bearer YOUR_API_KEY
Content-Type: application/json
Idempotency-Key: 4f6d81f0-8ab4-4aac-a723-9f1c2b3d4e5f

{"urls": ["https://example.com/page-1", "https://example.com/page-2"]}

Rules

Format. 1 to 128 characters from A-Z a-z 0-9 _ - . :. Anything else is 400 VALIDATION_ERROR before any work is done.
Scope. Keys are private to your account. Another account using the same string is unrelated to yours.
Binding. On the first successful (charged) request the key is stored together with a hash of the URL list exactly as you sent it (after trimming whitespace, in the same order, repeats included).
Replay. A later request with the same key and the same URL list returns 200 OK with replayed: true, the original batch_id, accepted_count, credit_cost and submission_ids, and the batch's current summary. Nothing is charged, nothing is queued. The invalid and duplicates lists are empty on a replay because only the original response knew them; invalid_count and duplicate_count are 0 for the same reason.
Conflict. The same key with a different URL list (different URLs, a different order, or a different number of repeats) returns 409 IDEMPOTENCY_CONFLICT with details.batch_id and details.original_request_at. Nothing is charged. Use a new key for a new request.
Failures are not stored. If the first attempt fails (422, 402, 503, …) nothing is recorded, so retrying the same key after fixing the problem works normally.
Concurrency. Two identical keyed requests arriving at the same time are serialized on the server; one creates the batch, the other replays it. Exactly one charge.
Retention. A used key stays bound to its batch for the life of the account; replays are answered from the batch record, which is kept permanently even after individual submission rows are purged. There is no expiry after which a reused key would silently create a second charge.

What idempotency does not cover

The key protects the API request: the charge and the creation of submissions. Delivery to the external indexing provider happens later, in the background, and is tracked per submission by status. The provider's own retry and timeout behaviour is independent of your key. Requests without a key are supported and behave as described under Duplicate detection: a repeat within 1 minutes is skipped as duplicates and not charged; a repeat after that window is a new submission and is charged.

Retry handling

Response	Retry?	How
Network error / timeout with no response	Yes	Same request, same Idempotency-Key, exponential backoff (2, 4, 8 s). Without a key you risk a second charge once the duplicate window has passed.
429 RATE_LIMITED	Yes	Wait Retry-After seconds, then repeat unchanged. Nothing happened.
500 INTERNAL_ERROR / 500 SERVER_ERROR	Yes, a few times	Nothing was charged. Same key.
503 MAINTENANCE / 503 PROVIDER_UNAVAILABLE	Yes, later	Wait minutes, not seconds. Nothing was charged.
402 INSUFFICIENT_CREDITS	After adding credits	Same key is fine: the failed attempt stored nothing.
409 IDEMPOTENCY_CONFLICT	No	You reused a key. Send a new key, or check details.batch_id if the original request is what you wanted.
400 / 422 validation errors	No	Fix the request. Repeating it unchanged will fail the same way.
401 / 403 / 404 / 405	No	Fix credentials, access, the id or the method.

Complete retry loops in PHP, Python and JavaScript are in the migration guide.

Error codes

HTTP	code	When	Charged?
400	INVALID_JSON	The body is not a JSON object.	No
400	VALIDATION_ERROR	Idempotency-Key header is malformed.	No
401	UNAUTHENTICATED	Missing/malformed Authorization header, or an unknown, revoked or regenerated key, or a disabled account.	No
402	INSUFFICIENT_CREDITS	Balance does not cover the accepted URLs. details.balance where applicable.	No
403	FORBIDDEN	Valid key, but API access is not enabled for the account.	No
404	NOT_FOUND	Unknown endpoint, or a submission/batch that does not exist or is not visible to you.	No
405	METHOD_NOT_ALLOWED	Known path, wrong method. The Allow header lists accepted methods.	No
413	PAYLOAD_TOO_LARGE	Request body over 4 MB. Split the submission.	No
409	IDEMPOTENCY_CONFLICT	Idempotency-Key reused with a different URL list. details.batch_id, details.original_request_at.	No
422	VALIDATION_ERROR	urls missing, not a list, contains a non-string, has no non-empty entry, or fewer than 1 valid URLs remain; or an unknown status filter. details.field / details.minimum_batch_size.	No
422	BATCH_LIMIT_EXCEEDED	More than 1,000 entries. details.maximum_batch_size.	No
422	NO_VALID_URLS	Every entry was invalid or a duplicate. details.invalid, details.duplicates.	No
429	RATE_LIMITED	Account over 60 requests/minute. Retry-After header.	No
500	INTERNAL_ERROR	The submission could not be stored because of a server fault.	No
500	SERVER_ERROR	Unexpected server fault elsewhere.	No
503	PROVIDER_UNAVAILABLE	No indexing provider is active right now.	No
503	MAINTENANCE	The platform is in maintenance mode.	No

Messages are meant for humans and may be reworded; branch on code and the HTTP status, never on message text.

Reseller permissions

A Reseller key reads its own submissions and batches and those of the Customer accounts it manages. It submits only under its own account and balance; there is no "submit on behalf of a Customer" parameter.
A Customer key reads and submits only its own data, including a Customer managed by a Reseller. It can never see the Reseller's data or another Customer's.
Ids you may not see return 404, not 403, so the existence of other accounts' records is never confirmed.
Customers managed by a Reseller are billed from their own balance. The "Insufficient credits" outcome for such accounts also covers the case where the Reseller's pooled balance cannot back the request.

Security best practices

Treat the key like a password: environment variables or a secret manager, never source control or client-side code.
Call the API from your server, not from a browser; a key in a web page is public.
Rotate the key from the API Access page if it may have leaked. The old key stops working instantly on both API versions.
Use HTTPS only (the base URL above). Never send the key in a URL.
Log batch_id, submission_ids and your Idempotency-Keys on your side; they make support questions and reconciliation straightforward.

Frequently asked questions

Does SUCCESS mean my page is indexed by Google?

No. SUCCESS means the indexing provider processed the URL. Whether and when a search engine indexes it is outside CrawlMe's control, and the API does not check it.

How long until a submission is PROCESSING / SUCCESS?

Queued URLs are usually handed to a provider within a few minutes. Provider processing varies from minutes to more than a day. Poll GET /batches/{id} every few minutes rather than every second; it costs one request against your rate limit.

Do I need a new key for v2?

No. Your existing key works with both versions.

Do v1 and v2 share the rate limit?

Yes, one allowance of 60 requests per minute per account across both.

What if I submit the same URL again later?

Within 1 minutes it is skipped as a duplicate for free. After that it is accepted and charged as a new submission.

Can I cancel a submission?

Not through the API. Accepted URLs are queued immediately.

Why is the batch's credit_cost higher than what I ended up paying?

credit_cost is what the batch was charged at creation. Failed URLs are refunded afterwards; the refunds show in your balance and transaction history, not in the batch figure.

Troubleshooting

Symptom	Likely cause	Fix
401 "Missing or malformed Authorization header"	Header absent, or not Bearer <key>; some proxies strip Authorization.	Send Authorization: Bearer uidx_…; check your HTTP client does not drop the header on redirects.
401 "Invalid or revoked API key"	Key was regenerated or revoked, or the account is disabled.	Copy the current key from API Access.
403 FORBIDDEN	API access not enabled for the account.	Request access from the API Access page; an Admin enables it.
404 NOT_FOUND on /api/v2/balance.php or a trailing slash	v2 paths have no .php suffix and no trailing slash.	Use https://crawlme.net/api/v2/balance.
422 VALIDATION_ERROR "urls[2] must be a string"	A nested array, number or null in the list.	Send plain strings only.
422 NO_VALID_URLS right after a successful request	You resubmitted the same URLs inside the duplicate window.	Expected; nothing was charged. Use an Idempotency-Key if this was a retry.
409 IDEMPOTENCY_CONFLICT	Key reused for a different URL list.	Generate a key per logical request.
Batch stays PENDING for a long time	The background worker runs every few minutes; a stalled queue fails and refunds rows after a platform limit.	Wait a few minutes; if rows end FAILED, the refund is automatic.
Submission returns 404 weeks later	Finished submissions are purged after the retention period.	Use GET /batches/{id} for the summary; store per-URL results when fresh.

Overview

Base URL

Authentication

API key management

Quick start

Balance

Submit

Batch status

Account balance

GET/api/v2/balance#

Submitting URLs

POST/api/v2/submissions#

URL validation rules

Duplicate detection

Credit deduction behavior

Batch status tracking

GET/api/v2/batches/{id}#

GET/api/v2/batches/{id}/submissions#

Individual submission status

GET/api/v2/submissions/{id}#

Status values

Listing submissions

GET/api/v2/submissions#

Pagination

Rate limiting

Idempotency

Rules

What idempotency does not cover

Retry handling

Error codes

Reseller permissions

Security best practices

Frequently asked questions

Does SUCCESS mean my page is indexed by Google?

How long until a submission is PROCESSING / SUCCESS?

Do I need a new key for v2?

Do v1 and v2 share the rate limit?

What if I submit the same URL again later?

Can I cancel a submission?

Why is the batch's credit_cost higher than what I ended up paying?

Troubleshooting

GET`/api/v2/balance`#

POST`/api/v2/submissions`#

GET`/api/v2/batches/{id}`#

GET`/api/v2/batches/{id}/submissions`#

GET`/api/v2/submissions/{id}`#

GET`/api/v2/submissions`#