Developers
Sign in
Guide

Migrating from API v1 to v2

Both versions talk to the same account, balance and submission records with the same API key. You can move one call at a time, run both side by side, and there is no deadline. API v1 stays supported.

Why migrate

  • Retry-safe submissions. An optional Idempotency-Key header guarantees a retried request is never charged or queued twice (details).
  • Batch tracking. GET /batches/{id} gives one summary with counts per state; GET /batches/{id}/submissions lists the rows.
  • Machine-readable errors. Insufficient credits is 402 INSUFFICIENT_CREDITS, a bad request is 422 VALIDATION_ERROR, and so on, instead of a shared 400 SUBMISSION_FAILED whose message you had to parse.
  • Rate-limit visibility. X-RateLimit-* headers on every response and Retry-After on 429.
  • Richer submit response. Accepted, invalid and duplicate counts, the credit cost, your remaining balance, the new submission ids and the batch summary in one reply.

Comparison

FeatureAPI v1API v2
Base URL/api/v1/api/v2
AuthenticationAuthorization: BearerAuthorization: Bearer (same key)
BalanceGET /balance.phpGET /balance
SubmitPOST /submit.phpPOST /submissions
Submission detailsGET /submission/{id}GET /submissions/{id}
Submission listGET /submissions.php?page=&limit=&status=GET /submissions?page=&per_page=&status=
Batch detailsNot availableGET /batches/{id}
Batch submissionsNot availableGET /batches/{id}/submissions
IdempotencyDuplicate-URL window only (1 min)Optional Idempotency-Key header, plus the same duplicate window
Envelope{success, data} / {success, error{code,message}}Same, plus optional error.details
Timestamps2026-10-23 09:15:02 (UTC)2026-10-23T09:15:02Z (UTC, ISO 8601)
Error statusesMostly 400 with SUBMISSION_FAILED402 / 404 / 405 / 409 / 422 / 429 / 503 with specific codes
Rate limit60/min per account, no headersSame shared limit, X-RateLimit-* and Retry-After headers
Unknown status filterIgnored (unfiltered list)422 VALIDATION_ERROR
Non-string entries in urlsSilently dropped422 VALIDATION_ERROR naming the entry
Request loggingPer requestPer request (same log)

Unchanged: API keys, account scope (Reseller keys see their Customers' data), URL validation rules, the duplicate window, batch size limits, credit pricing, charging and refund behaviour, and the background processing that sends URLs to providers. Nothing about how URLs are processed differs between versions.

Authentication

No change. Keep the same header and the same key.

HTTP
Authorization: Bearer YOUR_API_KEY

Balance

v1
cURL
curl "https://crawlme.net/api/v1/balance.php" \
  -H "Authorization: Bearer YOUR_API_KEY"
JSON
{
    "success": true,
    "data": {
        "balance": 150.0
    }
}
v2
cURL
curl "https://crawlme.net/api/v2/balance" \
  -H "Authorization: Bearer YOUR_API_KEY"
JSON
{
    "success": true,
    "data": {
        "account_id": 1042,
        "balance": 150.0
    }
}

v2 adds account_id. balance is identical.

Submitting URLs

The request body is the same. The response is richer, and the status code for a replayed request is 200 instead of 201.

v1
cURL
curl -X POST "https://crawlme.net/api/v1/submit.php" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com/a","https://example.com/b"]}'
JSON
{
    "success": true,
    "data": {
        "batch_id": 293244,
        "submitted_count": 2,
        "invalid": [],
        "duplicates": []
    }
}
v2
cURL
curl -X POST "https://crawlme.net/api/v2/submissions" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Idempotency-Key: order-7f3a9c" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com/a","https://example.com/b"]}'
JSON
{
    "success": true,
    "data": {
        "batch_id": 293244,
        "replayed": false,
        "accepted_count": 2,
        "invalid_count": 0,
        "duplicate_count": 0,
        "credit_cost": 2.0,
        "remaining_balance": 148.0,
        "invalid": [],
        "duplicates": [],
        "submission_ids": [
            8812001,
            8812002
        ],
        "batch": {
            "id": 293244,
            "account_id": 1042,
            "status": "PENDING",
            "total": 2,
            "successful": 0,
            "failed": 0,
            "pending": 2,
            "queued": 2,
            "processing": 0,
            "credit_cost": 2.0,
            "created_at": "2026-10-23T09:15:02Z",
            "completed_at": null
        }
    }
}
v1 fieldv2 field
batch_idbatch_id (same value)
submitted_countaccepted_count (same meaning)
invalidinvalid (same), plus invalid_count
duplicatesduplicates (same), plus duplicate_count
-credit_cost, remaining_balance, submission_ids, batch, replayed

Code to change: read accepted_count instead of submitted_count. Everything else you already read is still there under the same name.

Submission status

v1
cURL
curl "https://crawlme.net/api/v1/submission/8812001" \
  -H "Authorization: Bearer YOUR_API_KEY"
JSON
{
    "success": true,
    "data": {
        "id": 8812001,
        "batch_id": 293244,
        "url": "https://example.com/a",
        "status": "SUCCESS",
        "credits_used": 1.0,
        "submitted_at": "2026-10-23 09:15:02",
        "updated_at": "2026-10-23 11:02:11",
        "completed_at": "2026-10-23 11:02:11"
    }
}
v2
cURL
curl "https://crawlme.net/api/v2/submissions/8812001" \
  -H "Authorization: Bearer YOUR_API_KEY"
JSON
{
    "success": true,
    "data": {
        "id": 8812001,
        "batch_id": 293244,
        "account_id": 1042,
        "url": "https://example.com/a",
        "status": "SUCCESS",
        "credits_used": 1.0,
        "submitted_at": "2026-10-23T09:15:02Z",
        "updated_at": "2026-10-23T11:02:11Z",
        "completed_at": "2026-10-23T11:02:11Z"
    }
}

Same fields and status values; v2 adds account_id and formats timestamps as ISO 8601. If you parse dates, switch the format string. In v2 you can also stop polling individual submissions and watch GET /batches/{id} instead.

Listing submissions

Rename limit to per_page. The pagination object is identical. Rows have the v2 submission shape above. An unknown status value is an error in v2 rather than silently ignored.

v1
HTTP
GET https://crawlme.net/api/v1/submissions.php?page=2&limit=50&status=FAILED
v2
HTTP
GET https://crawlme.net/api/v2/submissions?page=2&per_page=50&status=FAILED

Error handling

In v1 almost every business failure is 400 SUBMISSION_FAILED and you must read the message. In v2 switch on the status code and error.code:

Situationv1v2
Bad JSON400 INVALID_JSON400 INVALID_JSON
Missing / malformed urls400 VALIDATION_ERROR422 VALIDATION_ERROR (with details.field)
Too many URLs400 SUBMISSION_FAILED422 BATCH_LIMIT_EXCEEDED
Too few valid URLs400 SUBMISSION_FAILED422 VALIDATION_ERROR (details.minimum_batch_size)
All invalid or duplicates400 SUBMISSION_FAILED422 NO_VALID_URLS (details.invalid, details.duplicates)
Not enough credits400 SUBMISSION_FAILED402 INSUFFICIENT_CREDITS
No provider available400 SUBMISSION_FAILED503 PROVIDER_UNAVAILABLE
Idempotency-Key reused with another payloadn/a409 IDEMPOTENCY_CONFLICT
Unknown endpointserver 404 page404 NOT_FOUND (JSON)
Wrong method405 METHOD_NOT_ALLOWED405 METHOD_NOT_ALLOWED (+ Allow header)
Rate limited429 RATE_LIMITED429 RATE_LIMITED (+ Retry-After)
Server error500 SERVER_ERROR / 400 "Submission failed. Please try again."500 SERVER_ERROR / 500 INTERNAL_ERROR

Retries

In v1, a retried submission is only safe inside the 1-minute duplicate window. In v2, send an Idempotency-Key that is unique per logical request (a UUID, or your own order id) and reuse it on every retry of that request:

PHP
<?php
function submit_with_retry(array $urls, string $idempotencyKey): array
{
    for ($attempt = 1; $attempt <= 5; $attempt++) {
        $ch = curl_init('https://crawlme.net/api/v2/submissions');
        curl_setopt_array($ch, [
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_POST           => true,
            CURLOPT_HTTPHEADER     => [
                'Authorization: Bearer YOUR_API_KEY',
                'Idempotency-Key: ' . $idempotencyKey,
                'Content-Type: application/json',
            ],
            CURLOPT_POSTFIELDS     => json_encode(['urls' => $urls]),
            CURLOPT_TIMEOUT        => 30,
        ]);
        $raw = curl_exec($ch);
        $status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
        curl_close($ch);

        if ($raw === false || $status >= 500) {
            sleep(2 ** $attempt); // network error or server error: retry, same key
            continue;
        }
        $body = json_decode($raw, true);
        if ($status === 429) {
            sleep(5); // Retry-After header carries the exact wait
            continue;
        }
        return $body; // 201 created, 200 replayed, or a 4xx you should not retry
    }
    throw new RuntimeException('Gave up after 5 attempts');
}
Python
import time, uuid, requests

def submit_with_retry(urls, idempotency_key=None):
    idempotency_key = idempotency_key or str(uuid.uuid4())
    for attempt in range(1, 6):
        try:
            r = requests.post(
                "https://crawlme.net/api/v2/submissions",
                headers={"Authorization": "Bearer YOUR_API_KEY", "Idempotency-Key": idempotency_key},
                json={"urls": urls},
                timeout=30,
            )
        except requests.RequestException:
            time.sleep(2 ** attempt)  # network error: retry with the same key
            continue
        if r.status_code == 429:
            time.sleep(int(r.headers.get("Retry-After", "5")))
            continue
        if r.status_code >= 500:
            time.sleep(2 ** attempt)
            continue
        return r.json()  # 201 created, 200 replayed, or a 4xx you should not retry
    raise RuntimeError("Gave up after 5 attempts")
JavaScript
async function submitWithRetry(urls, idempotencyKey = crypto.randomUUID()) {
  for (let attempt = 1; attempt <= 5; attempt++) {
    let response;
    try {
      response = await fetch("https://crawlme.net/api/v2/submissions", {
        method: "POST",
        headers: {
          "Authorization": "Bearer YOUR_API_KEY",
          "Idempotency-Key": idempotencyKey,
          "Content-Type": "application/json",
        },
        body: JSON.stringify({ urls }),
      });
    } catch (networkError) {
      await new Promise(r => setTimeout(r, 2 ** attempt * 1000)); // retry, same key
      continue;
    }
    if (response.status === 429) {
      const wait = Number(response.headers.get("Retry-After") || 5);
      await new Promise(r => setTimeout(r, wait * 1000));
      continue;
    }
    if (response.status >= 500) {
      await new Promise(r => setTimeout(r, 2 ** attempt * 1000));
      continue;
    }
    return response.json(); // 201 created, 200 replayed, or a 4xx you should not retry
  }
  throw new Error("Gave up after 5 attempts");
}

Checklist

  1. Change the base URL to https://crawlme.net/api/v2 and drop the .php suffixes; rename /submit.php to /submissions and /submission/{id} to /submissions/{id}.
  2. In the submit response, read accepted_count instead of submitted_count.
  3. Rename the list parameter limit to per_page.
  4. Parse ISO 8601 timestamps (2026-10-23T09:15:02Z).
  5. Handle errors by status code and error.code; stop matching on message text.
  6. Add an Idempotency-Key to submissions and retry 429 / 5xx / network errors with the same key.
  7. Optionally replace per-submission polling with GET /batches/{id}.
  8. Keep the same API key. Nothing to regenerate.

Both versions can run at the same time during the switch. Requests to either count against the same 60 requests/minute account limit.