Home Playground Docs hub

Guide G-12 · Applied Intelligence

Detecting objects and poses in images

Verified against local · 2026-09-23 · object-detection-service 0.1.0 (workspace HEAD)
Journey. As a developer, I want to submit an image and learn what's in it —
which objects, where, and how sure the model is — and for people, where their
body keypoints are. No model hosting, no GPU wrangling.

The Object Detection Service runs YOLO-family vision models behind two calls: POST /v3/detect labels every object in an image with a bounding box; POST /v3/pose finds every person and returns their 17 body keypoints. This guide walks both, explains the coordinate system, and shows client-side confidence filtering. Verified against the local stack (Object Detection Service :45091, YOLO26 small).

Prerequisites

shared dev subdomain).

TOKEN (obtain one via guide G-01; the full example below fetches one via the QA sign-in shortcut). On the local stack the REST routes have no auth middleware of their own — the header trio is enforced by the platform gateway in deployed environments, and sending it everywhere keeps your code environment-agnostic.

${IMAGE_PATH}.

Concepts (only what this journey needs)

the file in a field named image, and return JSON. detect covers the COCO object set (person, car, dog, chair, …); pose returns people only.

x1, y1 (top-left) → x2, y2 (bottom-right), each relative to the image width/height. Keypoint x/y work the same way. To draw on the image, multiply by its pixel dimensions — no image-size math needed to compare detections across images.

detection; you threshold confidence yourself (the MCP tools accept a min_confidence argument instead).

a one-time model-load cost (seconds), then each image costs roughly a second on the local stack.

tool get_recent_detections reads the latest ten. Useful for audit trails — be deliberate about what images you send.

Step 1 — Detect objects

bash
curl -s -X POST "${BASE_URL}/v3/detect" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "x-api-key: ${INAYA_API_KEY}" \
  -H "x-client-app-key: ${INAYA_APP_KEY}" \
  -F "image=@${IMAGE_PATH}"
python
import json, os, urllib.request

boundary = "----g12boundary"
image = open(os.environ["IMAGE_PATH"], "rb").read()
body = (
    f"--{boundary}\r\n"
    'Content-Disposition: form-data; name="image"; filename="image.png"\r\n'
    "Content-Type: image/png\r\n\r\n"
).encode() + image + f"\r\n--{boundary}--\r\n".encode()

req = urllib.request.Request(
    f"{os.environ['BASE_URL']}/v3/detect",
    data=body,
    headers={
        "Content-Type": f"multipart/form-data; boundary={boundary}",
        "Authorization": f"Bearer {os.environ['TOKEN']}",
        "x-api-key": os.environ["INAYA_API_KEY"],
        "x-client-app-key": os.environ["INAYA_APP_KEY"],
    },
)
print(json.dumps(json.load(urllib.request.urlopen(req, timeout=120)), indent=2))
javascript
import { readFileSync } from "node:fs";

const form = new FormData();
form.set("image", new Blob([readFileSync(process.env.IMAGE_PATH)]), "image.png");

const res = await fetch(`${process.env.BASE_URL}/v3/detect`, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.TOKEN}`,
    "x-api-key": process.env.INAYA_API_KEY,
    "x-client-app-key": process.env.INAYA_APP_KEY,
  },
  body: form,
});
console.log(await res.json());

Expected response — 200 with every detected object (verified against a 1280×720 street photo; trimmed):

json
{
  "detections": [
    { "bounding_box": { "x1": 0.155, "y1": 0.118, "x2": 0.325, "y2": 0.941 }, "confidence": 0.961, "object_instance_id": 0, "object_type": "person" },
    { "bounding_box": { "x1": 0.522, "y1": 0.359, "x2": 0.999, "y2": 0.912 }, "confidence": 0.968, "object_instance_id": 0, "object_type": "car" }
  ],
  "processing_time_ms": 1083
}

TIP

Pixel conversion is one multiply away — on the verified example, the person box lands at x 198–416, y 85–678 on the 1280×720 image: px_x1 = det["bounding_box"]["x1"] * img_width, likewise for the rest. object_instance_id is a per-response tracker id: REST calls are stateless, so don't expect ids to match between two uploads of the same scene.

PITFALL

Error bodies are { "error": "…" } with distinct codes: 400 — the multipart form has no image field; 422 — the bytes aren't a decodable image ("The image format could not be determined"); 503 — the model couldn't initialize (e.g. OBJECT_DETECTION_GENERAL_MODEL_PATH points at a missing file). All three were observed live; treat 503 as infrastructure, not input, and check the service logs.

Step 2 — Estimate poses

Same multipart contract, keypoints instead of labels — up to 17 COCO joints per person (nose, left_eye, …, left_ankle), each with its own confidence:

bash
curl -s -X POST "${BASE_URL}/v3/pose" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "x-api-key: ${INAYA_API_KEY}" \
  -H "x-client-app-key: ${INAYA_APP_KEY}" \
  -F "image=@${IMAGE_PATH}"
python
import json, os, urllib.request

boundary = "----g12boundary"
image = open(os.environ["IMAGE_PATH"], "rb").read()
body = (
    f"--{boundary}\r\n"
    'Content-Disposition: form-data; name="image"; filename="image.png"\r\n'
    "Content-Type: image/png\r\n\r\n"
).encode() + image + f"\r\n--{boundary}--\r\n".encode()

req = urllib.request.Request(
    f"{os.environ['BASE_URL']}/v3/pose",
    data=body,
    headers={
        "Content-Type": f"multipart/form-data; boundary={boundary}",
        "Authorization": f"Bearer {os.environ['TOKEN']}",
        "x-api-key": os.environ["INAYA_API_KEY"],
        "x-client-app-key": os.environ["INAYA_APP_KEY"],
    },
)
d = json.load(urllib.request.urlopen(req, timeout=120))
for pose in d["poses"]:
    visible = [k["part"] for k in pose["keypoints"] if k["confidence"] >= 0.5]
    print(f"person conf={pose['confidence']:.3f} visible keypoints: {', '.join(visible)}")
print("processing_time_ms:", d["processing_time_ms"])

Expected response — 200 with one pose per person (verified on the same street photo — one person, 17 keypoints returned, 16 at confidence ≥ 0.5; trimmed):

json
{
  "poses": [
    {
      "bounding_box": { "x1": 0.155, "y1": 0.120, "x2": 0.324, "y2": 0.939 },
      "confidence": 0.926,
      "object_instance_id": 0,
      "keypoints": [
        { "index": 0, "part": "nose", "x": 0.234, "y": 0.157, "confidence": 0.995 },
        { "index": 5, "part": "left_shoulder", "x": 0.177, "y": 0.276, "confidence": 0.98 }
      ]
    }
  ],
  "processing_time_ms": 1103
}

NOTE

All 17 keypoints are always present — there is no server-side visibility cut. A keypoint with very low confidence (below ~0.1) is probably not visible in the image; threshold before you draw a skeleton or compute angles.

Step 3 — Filter by confidence (client-side)

The REST response is unfiltered, so production UIs apply their own threshold. For example, keeping person and car detections at 0.5 or better:

python
import json, os, urllib.request

boundary = "----g12boundary"
image = open(os.environ["IMAGE_PATH"], "rb").read()
body = (
    f"--{boundary}\r\n"
    'Content-Disposition: form-data; name="image"; filename="image.png"\r\n'
    "Content-Type: image/png\r\n\r\n"
).encode() + image + f"\r\n--{boundary}--\r\n".encode()

req = urllib.request.Request(
    f"{os.environ['BASE_URL']}/v3/detect",
    data=body,
    headers={"Content-Type": f"multipart/form-data; boundary={boundary}"},
)
d = json.load(urllib.request.urlopen(req, timeout=120))
keep = [x for x in d["detections"] if x["confidence"] >= 0.5 and x["object_type"] in ("person", "car")]
print(f"kept {len(keep)} of {len(d['detections'])} detections at >= 0.5")
for x in keep:
    print(" ", x["object_type"], round(x["confidence"], 3))

Expected output (verified on the street photo):

kept 5 of 8 detections at >= 0.5
  car 0.968
  person 0.961
  car 0.884
  car 0.831
  car 0.814

TIP

If an AI agent drives this flow, prefer the service's MCP tools (detect_objects, estimate_pose): they take image_base64 plus a server-side min_confidence filter, and get_recent_detections replays the

latest snapshots. See guide G-02 for MCP mechanics.

Full working example

A complete detect → filter → pose cycle (vision_tour.sh) — it fetches a user JWT via the QA shortcut first (local/QA environments only; production uses the G-01 flows):

bash
#!/usr/bin/env bash
set -euo pipefail

# 0. Obtain a user JWT (QA shortcut — see G-01 for production flows)
TOKEN=$(curl -sf -X POST "${AUTH_URL}/v3/auth/qa/login" \
  -H "Content-Type: application/json" \
  -H "x-api-key: ${INAYA_API_KEY}" \
  -H "x-client-app-key: ${INAYA_APP_KEY}" \
  -d "{\"device_id\": \"vision-demo-device\", \"locale\": \"en-US\", \"tenant_id\": \"${TENANT_ID}\", \"username\": \"vision-demo\"}" \
  | python3 -c 'import json,sys; print(json.load(sys.stdin)["token"])')
AUTH=(-H "Authorization: Bearer ${TOKEN}"
      -H "x-api-key: ${INAYA_API_KEY}"
      -H "x-client-app-key: ${INAYA_APP_KEY}")

# 1. Detect
curl -sf -X POST "${BASE_URL}/v3/detect" "${AUTH[@]}" -F "image=@${IMAGE_PATH}" \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); print("detections:", len(d["detections"]), "in", d["processing_time_ms"], "ms")'

# 2. Pose
curl -sf -X POST "${BASE_URL}/v3/pose" "${AUTH[@]}" -F "image=@${IMAGE_PATH}" \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); print("poses:", len(d["poses"]), "| keypoints on first pose:", len(d["poses"][0]["keypoints"]) if d["poses"] else 0)'

(AUTH_URL is the Auth Service base URL — http://127.0.0.1:40004 on the local stack; BASE_URL here is the Object Detection Service — http://127.0.0.1:45091.)

Try it live

Open the Playground → Applied Intelligence → Object Detection — upload an image and see the boxes and keypoints rendered, running against a live service.

Copy agent prompt

Agent prompt — copy and hand to your agent
You are working with the Inaya Labs Object Detection Service at ${BASE_URL}
(local port 45091). The tenant key pair is in INAYA_API_KEY / INAYA_APP_KEY env
vars; obtain a user JWT via the QA login from guide G-01 if TOKEN is not
already set. Steps: (1) POST /v3/detect with multipart field "image" set to
the image at IMAGE_PATH; report the count of detections and each
object_type with its confidence. (2) Re-list only detections with
confidence >= 0.5. (3) POST /v3/pose on the same image; report the number of
people and, for the first person, how many of the 17 keypoints have
confidence >= 0.5. Remember coordinates are relative (0.0-1.0). Never log
the JWT.

API reference

Next journeys