Inaya Labs | Docs — Errors, retries & idempotency

Journey. As a developer, I want my integration to fail predictably, retry
only transient failures, and never corrupt state when a call is repeated, so
that my app survives the messy reality of networks and services.

Production integrations live or die by what they do when things go wrong. This guide shows the platform's failure vocabulary, a safe retry pattern, and how to make repeated writes harmless. It uses the Memcache Service (:40011) in the local environment — Sandbox accepts everything, which defeats the purpose.

Prerequisites

The Memcache Service is reachable in a non-Sandbox environment (local stack

:40011) — error statuses below assume real authentication is enforced.

Your tenant key pair in INAYA_API_KEY / INAYA_APP_KEY, and a user JWT in

TOKEN — Memcache endpoints require both (obtain one via guide G-01; the full example below fetches one via the QA sign-in shortcut).

curl, or any HTTP client. Snippets also show Python and Node.

Concepts (only what this journey needs)

Two credential tiers fail separately. A missing or rejected key pair or a

missing or expired JWT both produce 401 with an error message telling you which — {"error":"Missing Authorization token"} vs a key-pair rejection. Knowing which credential failed tells you whether to fix the pair (G-00) or re-authenticate (G-01).

The error envelope. Failure bodies are plain text or a JSON string:

{"error": "…"} — read the message before deciding what to do.

*4xx means you; 5xx/000 means them.* 4xx responses will fail

identically on retry — fix the request. 5xx and connection failures (000, timeouts) are the only statuses worth retrying.

Retry with backoff + jitter, and cap it. Two to four attempts, growing

waits, random jitter, and a hard ceiling. Then surface the failure.

Idempotency is a design choice, not a header. The platform does not issue

idempotency keys. Safe retries come from your identifiers: deterministic keys, upsert/overwrite semantics, or natural uniqueness in the payload.

Step 1 — Learn the failure vocabulary

Probe three representative failures and read the status line:

bash

echo "1. Missing key pair and JWT (expected 401):"
curl -s -o /tmp/last_error.json -w "   HTTP %{http_code}\n" \
  -X POST "${BASE_URL}/v3/mem-cache/add/demo-tenant" \
  -H "Content-Type: application/json" \
  -d '{"key": "k", "value": "v", "ttl_in_seconds": 60}'
cat /tmp/last_error.json; echo

echo "2. Unknown path, fully authenticated (expected 404):"
curl -s -o /dev/null -w "   HTTP %{http_code}\n" \
  "${BASE_URL}/v3/mem-cache/no-such-endpoint" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "x-api-key: ${INAYA_API_KEY}" \
  -H "x-client-app-key: ${INAYA_APP_KEY}"

echo "3. Malformed body, fully authenticated (expected 422):"
curl -s -o /dev/null -w "   HTTP %{http_code}\n" \
  -X POST "${BASE_URL}/v3/mem-cache/add/demo-tenant" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "x-api-key: ${INAYA_API_KEY}" \
  -H "x-client-app-key: ${INAYA_APP_KEY}" \
  -d '{"key": "k"}'

python

import json, os, urllib.request, urllib.error

base = os.environ["BASE_URL"]
headers = {
    "Content-Type": "application/json",
    "Authorization": f"Bearer {os.environ['TOKEN']}",
    "x-api-key": os.environ["INAYA_API_KEY"],
    "x-client-app-key": os.environ["INAYA_APP_KEY"],
}

def probe(label, req, expected):
    try:
        code = urllib.request.urlopen(req).status
    except urllib.error.HTTPError as e:
        code = e.code
    print(f"{label}: HTTP {code} (expected {expected})")

probe("1. Missing key pair and JWT",
      urllib.request.Request(f"{base}/v3/mem-cache/add/demo-tenant",
          data=json.dumps({"key": "k", "value": "v", "ttl_in_seconds": 60}).encode(),
          headers={"Content-Type": "application/json"}), 401)
probe("2. Unknown path",
      urllib.request.Request(f"{base}/v3/mem-cache/no-such-endpoint",
          headers=headers), 404)
probe("3. Malformed body",
      urllib.request.Request(f"{base}/v3/mem-cache/add/demo-tenant",
          data=json.dumps({"key": "k"}).encode(), headers=headers), 422)

javascript

const base = process.env.BASE_URL;
const headers = {
  "Content-Type": "application/json",
  Authorization: `Bearer ${process.env.TOKEN}`,
  "x-api-key": process.env.INAYA_API_KEY,
  "x-client-app-key": process.env.INAYA_APP_KEY,
};

const probe = async (label, input, expected) => {
  const res = await fetch(`${base}${input.path}`, input).catch(() => null);
  console.log(`${label}: HTTP ${res ? res.status : "000"} (expected ${expected})`);
};

await probe("1. Missing key pair and JWT", {
  path: "/v3/mem-cache/add/demo-tenant", method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ key: "k", value: "v", ttl_in_seconds: 60 }),
}, 401);
await probe("2. Unknown path", { path: "/v3/mem-cache/no-such-endpoint", headers }, 404);
await probe("3. Malformed body", {
  path: "/v3/mem-cache/add/demo-tenant", method: "POST",
  headers, body: JSON.stringify({ key: "k" }),
}, 422);

Expected output — probe 1's body names the missing credential; the rest are status codes:

1. Missing key pair and JWT (expected 401):
   HTTP 401
{"error": "Missing Authorization token"}
2. Unknown path, fully authenticated (expected 404):
   HTTP 404
3. Malformed body, fully authenticated (expected 422):
   HTTP 422

The mapping generalizes across every platform service:

Status	Meaning	Correct action
`400`	Business-rule rejection — the request was understood but refused	Read the message; change the request or treat as terminal
`401`	Key pair rejected, or missing JWT	Re-check credentials; re-authenticate via G-01 if token-based
`403`	Authenticated but not allowed (e.g. rotated refresh token)	Terminal — do not retry; re-authenticate
`404`	Wrong path or unknown resource id	Fix the URL/id; terminal
`409`	Conflict, e.g. CDN file exists without `overwrite=true`	Resolve the conflict deliberately (see G-04)
`422`	Body failed schema validation	Fix the payload; terminal
`429`	Rate limited	Back off longer than usual, then retry
`5xx` / `000`	Service or network failure	Retry with backoff (Step 2)

PITFALL

Never retry 4xx. A 422 retried ten times is ten guaranteed failures plus self-inflicted load. Retry logic must branch on the status class first.

Step 2 — Retry only what's retryable

A bounded retry helper: exponential backoff, jitter, and a small attempt cap. It retries connection failures and 5xx/429 only:

bash

#!/usr/bin/env bash
set -uo pipefail

call_with_retry() {
  local attempt=1 max=4 status
  while true; do
    status=$(curl -s -o /tmp/g03_last.json -w "%{http_code}" "$@")
    if [[ "$status" -lt 500 && "$status" -ne 000 && "$status" -ne 429 ]]; then
      echo "$status"; return 0
    fi
    if [[ "$attempt" -ge "$max" ]]; then
      echo "$status"; return 1
    fi
    sleep "$(( (2 ** attempt) + RANDOM % 2 ))"
    attempt=$((attempt + 1))
  done
}

code=$(call_with_retry -X GET "${BASE_URL}/v3/mem-cache/get/demo-tenant/some-key" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "x-api-key: ${INAYA_API_KEY}" \
  -H "x-client-app-key: ${INAYA_APP_KEY}")
echo "final status: $code"
cat /tmp/g03_last.json; echo

python

import json, os, random, time, urllib.request, urllib.error

def get_with_retry(path, attempts=4):
    url = f"{os.environ['BASE_URL']}{path}"
    headers = {
        "Authorization": f"Bearer {os.environ['TOKEN']}",
        "x-api-key": os.environ["INAYA_API_KEY"],
        "x-client-app-key": os.environ["INAYA_APP_KEY"],
    }
    for attempt in range(1, attempts + 1):
        try:
            req = urllib.request.Request(url, headers=headers)
            res = urllib.request.urlopen(req)
            return res.status, json.load(res)
        except urllib.error.HTTPError as e:
            if (e.code < 500 and e.code != 429) or attempt == attempts:
                return e.code, None
        except urllib.error.URLError:
            if attempt == attempts:
                return 0, None
        time.sleep(2 ** attempt + random.random())

status, _ = get_with_retry("/v3/mem-cache/get/demo-tenant/some-key")
print("final status:", status)

javascript

const getWithRetry = async (path, attempts = 4) => {
  for (let attempt = 1; ; attempt++) {
    try {
      const res = await fetch(`${process.env.BASE_URL}${path}`, {
        headers: {
          Authorization: `Bearer ${process.env.TOKEN}`,
          "x-api-key": process.env.INAYA_API_KEY,
          "x-client-app-key": process.env.INAYA_APP_KEY,
        },
      });
      if ((res.status < 500 && res.status !== 429) || attempt === attempts)
        return res.status;
    } catch {
      if (attempt === attempts) return 0;
    }
    await new Promise(r => setTimeout(r, 2 ** attempt * 1000 + Math.random() * 1000));
  }
};
console.log("final status:", await getWithRetry("/v3/mem-cache/get/demo-tenant/some-key"));

Expected output — succeeds on the first attempt against a healthy service. A cache miss on this REST endpoint is a 404 with an empty body — a terminal, non-retryable status, exactly what the helper is designed to stop on. (The MCP tool for the same service reports misses as {"found": false} instead — see G-02; conventions differ across surfaces, so always check the docs for the surface you use.)

final status: 404

NOTE

Design for parallel safety. If several of your instances can issue the same write, serialize retries per key — or make the write idempotent (Step 3) so racing retries converge instead of colliding.

Step 3 — Make retries harmless: idempotency by design

The platform has no idempotency-key header. Instead, choose identifiers and semantics that make repetition safe:

Deterministic keys + upsert semantics. Memcache add overwrites an

existing key (last write wins) and returns 201 whether or not the key existed. Derive the key from the business object (job-42-snapshot), not a random UUID: a retried store lands on the same key with the same value, and the state converges by construction. (Where a service offers an explicit overwrite switch instead — the CDN's overwrite=true, G-04 — use it when re-sending is expected.)

Read-modify-write with natural keys. GET then re-POST flows keyed by

a natural identifier survive repetition because the final state equals the last write.

Demonstrate convergence: two identical stores under a deterministic key both succeed, and the read-back shows a single entry with the expected value:

bash

#!/usr/bin/env bash
set -uo pipefail
HEADERS=(-H "Content-Type: application/json"
         -H "Authorization: Bearer ${TOKEN}"
         -H "x-api-key: ${INAYA_API_KEY}"
         -H "x-client-app-key: ${INAYA_APP_KEY}")
BODY='{"key": "job-42-snapshot", "value": "{\"status\": \"done\"}", "ttl_in_seconds": 300}'

# First store: 201
code1=$(curl -s -o /tmp/g03_add1.json -w "%{http_code}" -X POST \
  "${BASE_URL}/v3/mem-cache/add/demo-tenant" "${HEADERS[@]}" -d "$BODY")
# Retry (e.g. a client that never saw the response): 201 again — upsert
code2=$(curl -s -o /tmp/g03_add2.json -w "%{http_code}" -X POST \
  "${BASE_URL}/v3/mem-cache/add/demo-tenant" "${HEADERS[@]}" -d "$BODY")
# State check: the GET returns the stored value directly
final=$(curl -s "${BASE_URL}/v3/mem-cache/get/demo-tenant/job-42-snapshot" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "x-api-key: ${INAYA_API_KEY}" -H "x-client-app-key: ${INAYA_APP_KEY}")

echo "first: $code1, retry: $code2 (upsert — both succeed)"
echo "final state: $final"

Expected output:

first: 201, retry: 201 (upsert — both succeed)
final state: {"status": "done"}

PITFALL

Random IDs make retries dangerous. If your client generates a fresh UUID per attempt, a retried store creates orphan entries instead of converging. Generate business IDs once, at the source, and reuse them across retries.

Full working example

A resilient store helper — deterministic key, duplicate-tolerant, retried only on transient failures (retry_safe_store.sh). It first fetches a user JWT via the QA sign-in shortcut (local/QA environments only; in production, obtain the token through the passkey/federated flow from G-01):

bash

#!/usr/bin/env bash
set -euo pipefail

TENANT="demo-tenant"
KEY="job-42-snapshot"

# 0. Obtain a user JWT (QA shortcut — see G-01 for production flows)
TOKEN=$(curl -sf -X POST "${AUTH_URL}/v3/auth/qa/login" \
  -H "Content-Type: application/json" \
  -H "x-api-key: ${INAYA_API_KEY}" \
  -H "x-client-app-key: ${INAYA_APP_KEY}" \
  -d "{\"device_id\": \"retry-demo\", \"locale\": \"en-US\", \"tenant_id\": \"${TENANT_ID}\", \"username\": \"retry-demo\"}" \
  | python3 -c 'import json,sys; print(json.load(sys.stdin)["token"])')

HEADERS=(-H "Content-Type: application/json"
         -H "Authorization: Bearer ${TOKEN}"
         -H "x-api-key: ${INAYA_API_KEY}"
         -H "x-client-app-key: ${INAYA_APP_KEY}")

# 1. Store (upsert — safe to repeat)
code=$(curl -s -o /tmp/g03_full_last.json -w "%{http_code}" -X POST \
  "${BASE_URL}/v3/mem-cache/add/${TENANT}" "${HEADERS[@]}" \
  -d "{\"key\": \"${KEY}\", \"value\": \"done\", \"ttl_in_seconds\": 300}")
if [[ "$code" != "201" ]]; then
  echo "unexpected: $code"; cat /tmp/g03_full_last.json; exit 1
fi
echo "stored"

# 2. Read back
curl -sf "${BASE_URL}/v3/mem-cache/get/${TENANT}/${KEY}" "${HEADERS[@]}"
echo

(AUTH_URL is the Auth Service base URL — http://127.0.0.1:40004 on the local stack; TENANT_ID is your tenant's id.)

Try it live

Open the Playground → Infrastructure → Memcache console and repeat the same store twice — both attempts succeed, and the second simply overwrites: the upsert convergence from Step 3, live.

Copy agent prompt

Agent prompt — copy and hand to your agent

You are hardening an integration with the Inaya Labs Memcache Service at
${BASE_URL}. The tenant key pair is in INAYA_API_KEY / INAYA_APP_KEY env vars,
and a user JWT is in TOKEN (if empty, obtain one via the QA login described in
guide G-01, then send it as Authorization: Bearer on every call). Rules:
(1) Never retry 4xx statuses — report them as terminal with the error message.
(2) Retry only 5xx/429/connection failures, max 4 attempts, exponential
backoff 2s→4s→8s with jitter. (3) All keys must be deterministic: derive them
from the business object, never generate fresh UUIDs per attempt. (4) Memcache
add is an upsert (201, last write wins) — a repeated identical store converges;
verify with GET /v3/mem-cache/get/{tenant}/{key}, which returns the raw stored
value (404 with an empty body when missing). Demonstrate by storing under key
"job-42-snapshot" for tenant "demo-tenant", "retrying" the same store, and
reading the value back.

API reference

Memcache Service — Swagger UI · Scalar

Next journeys

Uploading files the platform way (G-04) — the 409/overwrite conflict pattern in action.
Authentication & sessions (G-01) — how to re-authenticate when 401/403 do occur.