Scenario. Dev closes the stockroom every evening: walk the aisles,
eyeball the shelves, write up what needs restocking. This tutorial replaces
the eyeball with an agent. You play the operator; the agent is any LLM
client that speaks MCP (Model Context Protocol). It discovers what each
service can do on its own — no API docs read, no SDK installed — counts the
objects in a shelf photo, writes a dated audit entry into the searchable
knowledge base, and answers follow-up questions from past audits. You
compose two services: Object Detection (YOLO26 vision, exposed as MCP
tools) and RAG (the audit log with citations).

The shape to internalize: services are tools, and discovery is the contract. Every service in this workspace mounts the same MCP route — POST /agent/v1/mcp (streamable HTTP, spec 2026-07-28, latest-only) — so an agent can walk from service to service asking "what can you do?" and call whatever it finds. The REST endpoints still exist for humans; the agent never needs them.

Time: ~45 min · Level: advanced (JSON-RPC, base64 payloads, failure injection)

Architecture

text

  you (the operator)
       │  "audit shelf A1 and log it"
       ▼
  ┌───────────── your MCP-capable agent (any LLM client) ─────────────┐
  │  1. discover:  POST /agent/v1/mcp  {method: tools/list}             │
  │     to every service it knows — no per-service code               │
  └───────┬──────────────────────────────────┬────────────────────────┘
          │ tools/call detect_objects        │ tools/call ingest_document
          ▼                                  ▼
  ┌─ Object Detection :45091 ──────┐  ┌─ RAG :40100 ──────────────────┐
  │ image_base64 ──► YOLO26 counts │  │ audit text ──► chunked, stored │
  │ {detections, count}            │  │ searchable by question         │
  └────────────────────────────────┘  └────────────────────────────────┘
          │                                  │
          └──────── audit entry ◄────────────┘
                     │
                     ▼
        tools/call search_documents ──► "what did last week's audit say?"

Hidden edge: an MCP tool result is a JSON string inside content[0].text — you parse twice (JSON-RPC envelope, then the tool's own payload). Checkpoint 2 makes that visible.

Prerequisites

The two services running: object-detection-service (:45091) and

rag-service (:40100), both booted with INAYA_LABS_ENVIRONMENT=sandbox so the MCP routes accept requests without tenant keys or JWTs. In production the same routes require x-api-key, x-client-app-key, and Authorization: Bearer <JWT> — the agent sends the same headers on every call.

Postgres running (object-detection persists detection snapshots there).
YOLO26 ONNX models. The service's default path expects

~/.cache/yolo_models/yolo26x-ONNX/model.onnx; if you only have a smaller variant, point the env vars at it:

``bash source /tmp/inaya_env.sh # provides INAYA_LABS_POSTGRES_URL export INAYA_LABS_ENVIRONMENT=sandbox export OBJECT_DETECTION_GENERAL_MODEL_PATH=$HOME/.cache/yolo_models/yolo26s-ONNX/model.onnx export OBJECT_DETECTION_POSE_MODEL_PATH=$HOME/.cache/yolo_models/yolo26s-pose-ONNX/model.onnx nohup target/debug/object-detection-service > /tmp/object-detection-service.log 2>&1 & RAG_CHECKPOINT_PATH=/tmp/rag-stockroom-checkpoint \ nohup target/debug/rag-service > /tmp/rag-service.log 2>&1 & ``

A shelf photo. The companion code ships stockroom.png (shelf A1:

cardboard boxes, shrink-wrapped bottled-water packs, a red toolbox) so the checkpoints are deterministic. Any photo with recognizable objects works — the assertions check structure, not specific inventory.

Patience: the first detect_objects call includes ONNX model load

(~1–5 s for yolo26s); subsequent calls are ~1 s.

PITFALL

Three RAG ingest gotchas. First: only .pdf, .docx, .pptx, .xlsx, and .txt are supported — a .md file is rejected with unsupported format, so audit entries must be named .txt even when they read like markdown. Second: the document id is derived — {collection_id}-{file_name} — so re-ingesting the same file name overwrites; design file names as keys. Third: embeddings are a placeholder engine in this build, so retrieval is effectively keyword (BM25) search — word the audit entry and the follow-up question with shared vocabulary, or the search comes back empty.

Companion code

_code/agentic-inventory-audit/ — agentic_audit.sh (all five checkpoints as one script, PHOTO_PATH overridable) and stockroom.png (the audit photo).

Checkpoint 1 — Discovery: ask every service what it can do

An agent doesn't hardcode endpoints. It walks its service list and calls server/discover, then tools/list. The MCP route is the same on every service in this workspace; only the port changes. Note the Accept header: the endpoint answers 406 unless you accept application/json and/or text/event-stream, and each request declares its protocol revision in params._meta.

bash

OD_MCP="http://localhost:45091/agent/v1/mcp"
RAG_MCP="http://localhost:40100/agent/v1/mcp"
HEADERS=(-H "Content-Type: application/json" -H "Accept: application/json")
META='"_meta": {"io.modelcontextprotocol/protocolVersion": "2026-07-28"}'
# Production: add -H "x-api-key: ..." -H "x-client-app-key: ..." \
#             -H "Authorization: Bearer <JWT>" to every call below.

DISCOVER=$(curl -sf -m 15 -X POST "${OD_MCP}" "${HEADERS[@]}" \
  -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"server/discover\",\"params\":{${META}}}")
echo "${DISCOVER}" | python3 -c 'import json,sys
d = json.load(sys.stdin)
result = d["result"]
info = result["_meta"]["io.modelcontextprotocol/serverInfo"]
assert info["name"] == "object-detection-service", info
assert "2026-07-28" in result["supportedVersions"], result
print("discover:", info["name"], info["version"], "speaks", ", ".join(result["supportedVersions"]))'

OD_TOOLS=$(curl -sf -m 15 -X POST "${OD_MCP}" "${HEADERS[@]}" \
  -d "{\"jsonrpc\":\"2.0\",\"id\":2,\"method\":\"tools/list\",\"params\":{${META}}}")
echo "${OD_TOOLS}" | python3 -c 'import json,sys
names = {t["name"] for t in json.load(sys.stdin)["result"]["tools"]}
assert {"detect_objects", "estimate_pose", "get_recent_detections"} <= names, names
print("object-detection tools:", sorted(names))'

RAG_TOOLS=$(curl -sf -m 15 -X POST "${RAG_MCP}" "${HEADERS[@]}" \
  -d "{\"jsonrpc\":\"2.0\",\"id\":3,\"method\":\"tools/list\",\"params\":{${META}}}")
echo "${RAG_TOOLS}" | python3 -c 'import json,sys
names = {t["name"] for t in json.load(sys.stdin)["result"]["tools"]}
assert {"ingest_document", "search_documents", "list_documents"} <= names, names
print("rag tools:", sorted(names))'
echo "checkpoint 1 passed: the agent learned both services' capabilities from server/discover + tools/list"

You should see — the discover line naming object-detection-service 0.1.0, both tool sets listed, and the checkpoint line. An LLM agent would feed exactly these description + inputSchema fields into its planner.

Checkpoint 2 — The audit: count what's on the shelf

The operator's instruction — "count the bottled water on A1-02" — becomes a tools/call. The image goes up base64-encoded (the tool also accepts a data: URI); min_confidence filters the noise floor.

bash

PHOTO="InayaLabs-website/docs-src/tutorials/_code/agentic-inventory-audit/stockroom.png"
[ -f "${PHOTO}" ] || PHOTO="/tmp/t05-stockroom.png"

# Build the JSON-RPC payload in Python: base64 of a 1.7 MB photo is not
# something to wrangle through shell quoting.
python3 - "${PHOTO}" <<'EOF' > /tmp/t05-detect-request.json
import base64, json, sys
img = base64.b64encode(open(sys.argv[1], "rb").read()).decode()
print(json.dumps({
    "jsonrpc": "2.0", "id": 10, "method": "tools/call",
    "params": {"name": "detect_objects",
               "arguments": {"image_base64": img, "min_confidence": 0.3}}}))
EOF

# The tool result is a JSON *string* inside content[0].text — parse twice.
RESP=$(curl -sf -m 120 -X POST "${OD_MCP}" "${HEADERS[@]}" \
  -d @/tmp/t05-detect-request.json)
python3 - "${RESP}" <<'EOF'
import json, sys
env = json.loads(sys.argv[1])
tool = json.loads(env["result"]["content"][0]["text"])   # <-- second parse
dets = tool["detections"]
assert tool["count"] == len(dets) >= 1, tool
assert any(d["object_type"] == "bottle" for d in dets), [d["object_type"] for d in dets]
for d in dets:
    assert 0.0 < d["confidence"] <= 1.0, d
    bb = d["bounding_box"]
    assert all(0.0 <= bb[k] <= 1.0 for k in ("x1", "y1", "x2", "y2")), bb
bottles = sum(1 for d in dets if d["object_type"] == "bottle")
print(f'{tool["count"]} detections in {tool["processing_time_ms"]} ms; {bottles} bottles (conf 0.30+)')
open("/tmp/t05-state.sh", "w").write(f"BOTTLE_COUNT={bottles}\n")
print("checkpoint 2 passed: shelf photo audited via tools/call detect_objects")
EOF

You should see — N detections in <ms> ms; 7 bottles (conf 0.30+) and the checkpoint line, with /tmp/t05-state.sh carrying the count forward. (The exact count can drift ±1 across ONNX runtimes; the assertions only require detections, ≥1 bottle, sane confidences, and relative 0–1 boxes.)

NOTE

YOLO is trained on COCO classes — bottle is one, cardboard box is not. The vision count is therefore a partial audit by construction: the model reliably sees the water packs and misses the boxes. A good audit entry records both what was counted and what wasn't.

Checkpoint 3 — The log: write the audit where memory lives

The agent now writes the entry. ingest_document takes the file name, the collection, and base64 content; the service chunks and indexes it. The document id is derived — {collection_id}-{file_name} — so the same nightly name overwrites, and list_documents is the existence check.

bash

source /tmp/t05-state.sh
TODAY=$(date +%F)
AUDIT_FILE="audit-${TODAY}.txt"

# Compose the audit entry: counted items from the model, uncounted noted.
python3 - "${BOTTLE_COUNT}" "${TODAY}" <<'EOF' > /tmp/t05-audit.txt
import sys
bottles, today = sys.argv[1], sys.argv[2]
print(f"""Stockroom audit {today}

Shelf A1-02 top: {bottles} bottles detected by vision audit (min confidence 0.30,
shrink-wrapped water packs on the top shelf).
Shelf A1-01 and A1 lower: cardboard boxes present but NOT counted
(cardboard box is not a COCO class the vision model reports).

Action: restock bottled water before Friday pickup.
Audited by: operator via MCP agent (object-detection-service tools/call).""")
EOF

python3 - "${AUDIT_FILE}" <<'EOF' > /tmp/t05-ingest-request.json
import base64, json, sys
content = base64.b64encode(open("/tmp/t05-audit.txt", "rb").read()).decode()
print(json.dumps({
    "jsonrpc": "2.0", "id": 20, "method": "tools/call",
    "params": {"name": "ingest_document",
               "arguments": {"collection_id": "stockroom-audits",
                             "file_name": sys.argv[1],
                             "content_base64": content}}}))
EOF

RESP3=$(curl -sf -m 60 -X POST "${RAG_MCP}" "${HEADERS[@]}" \
  -d @/tmp/t05-ingest-request.json)
python3 - "${RESP3}" "${AUDIT_FILE}" <<'EOF'
import json, sys
doc = json.loads(json.loads(sys.argv[1])["result"]["content"][0]["text"])
# rag-service namespaces document ids with the caller's tenant ("<tenant>::<collection>-<file>")
assert doc["id"].endswith("stockroom-audits-" + sys.argv[2]), doc
assert doc["chunk_count"] >= 1, doc
print("ingested:", doc["id"], "-", doc["chunk_count"], "chunk(s)")
EOF

RESP4=$(curl -sf -m 30 -X POST "${RAG_MCP}" "${HEADERS[@]}" \
  -d '{"jsonrpc":"2.0","id":21,"method":"tools/call","params":{"name":"list_documents","arguments":{"collection_id":"stockroom-audits"}}}')
python3 - "${RESP4}" "${AUDIT_FILE}" <<'EOF'
import json, sys
docs = json.loads(json.loads(sys.argv[1])["result"]["content"][0]["text"])["documents"]
names = {d["file_name"] for d in docs}
assert sys.argv[2] in names, names
print("checkpoint 3 passed: audit persisted — collection lists", len(docs), "document(s)")
EOF

You should see — the ingested: stockroom-audits-audit-<today>.txt line with a chunk count, then the checkpoint line listing the collection size. Re-running the same day overwrites the entry (same derived id) — the script stays rerun-safe.

Checkpoint 4 — Memory: ask yesterday's question in natural language

The payoff: next week, "what did the last audit say needs restocking?" is a search_documents call, not a file hunt. Retrieved chunks carry provenance (file_name, section_hierarchy, score) for citations.

bash

RESP5=$(curl -sf -m 60 -X POST "${RAG_MCP}" "${HEADERS[@]}" \
  -d '{"jsonrpc":"2.0","id":30,"method":"tools/call","params":{"name":"search_documents","arguments":{"collection_id":"stockroom-audits","question":"what restock action was recorded for bottled water?"}}}')
python3 - "${RESP5}" <<'EOF'
import json, sys
res = json.loads(json.loads(sys.argv[1])["result"]["content"][0]["text"])
assert res["chunks"], res
top = res["chunks"][0]
assert "restock" in top["content"].lower(), top["content"][:200]
line = next(l for l in top["content"].splitlines() if "restock" in l.lower())
print(f'top hit: {top["file_name"]} (score {top["score"]:.2f})')
print("  ", line.strip()[:90])
print("checkpoint 4 passed: natural-language question retrieved the audit entry with provenance")
EOF

You should see — top hit: audit-<today>.txt (score …) with the restock line quoted and the checkpoint line. Keep the question's vocabulary close to the entry's — this build's embeddings are placeholders, so matching is keyword-driven (see the PITFALL above).

Kill the vision service and watch the agent's world shrink: the tool it discovered in Checkpoint 1 is suddenly unreachable — the failure surfaces at call time, not at planning time. Then restart and confirm recovery.

bash

pkill -x object-detection-service && echo "object-detection-service killed"

curl -s -m 10 -X POST "${OD_MCP}" "${HEADERS[@]}" \
  -d '{"jsonrpc":"2.0","id":40,"method":"tools/list"}' \
  || echo "discovery during outage: connection refused (fails loudly)"

source /tmp/inaya_env.sh
export INAYA_LABS_ENVIRONMENT=sandbox
export OBJECT_DETECTION_GENERAL_MODEL_PATH=$HOME/.cache/yolo_models/yolo26s-ONNX/model.onnx
export OBJECT_DETECTION_POSE_MODEL_PATH=$HOME/.cache/yolo_models/yolo26s-pose-ONNX/model.onnx
nohup /Users/test/projects/Software/InayaLabs/target/debug/object-detection-service \
  > /tmp/object-detection-service.log 2>&1 &
sleep 4

RESP6=$(curl -sf -m 120 -X POST "${OD_MCP}" "${HEADERS[@]}" \
  -d @/tmp/t05-detect-request.json)
python3 - "${RESP6}" <<'EOF'
import json, sys
tool = json.loads(json.loads(sys.argv[1])["result"]["content"][0]["text"])
assert tool["count"] >= 1, tool
print(f'checkpoint 5 passed: vision restored — {tool["count"]} detections on the same photo (model reloaded lazily on first call)')
EOF

You should see — the killed line, the connection-refused line, then the checkpoint line with a fresh detection count. Note the restart carries the model-path env vars: without them the service falls back to the default yolo26x path and the first call 503s.

NOTE

The resilience lesson for an agent is different from a human client's: the agent re-plans. When tools/call fails, a well-prompted agent falls back to whatever else tools/list offered (here: nothing — which is itself the finding), or tells the operator plainly. What it must not do is fabricate a count. This is why the audit entry names its source (tools/call) — provenance is what makes a machine-written log auditable by humans.

Ship it

Production access: the same MCP route on the deployed hosts, now

with x-api-key / x-client-app-key / Authorization: Bearer <JWT> on every call; the agent sends the same headers it would to the REST API. No payload changes.

Continuous audit: pair the service with the media-ingester pipeline

and the get_recent_detections tool — snapshots land in Postgres without anyone photographing shelves; audits become queries.

Bigger brains in the loop: wire Evergreen AI between discovery and

action — the agent asks it to plan which tools to call, then calls them itself. Tutorials T-01 and T-04 show the session/chat mechanics.

Cleanup: tools/call delete_document with the derived document id

removes the entry; rm -rf /tmp/rag-stockroom-checkpoint resets the vector store.

Copy agent prompt

Follow tutorial T-05 "Build an agentic inventory audit" at
/Users/test/projects/Software/InayaLabs/InayaLabs-website/docs/tutorials/agentic-inventory-audit.html
end to end. Discover tools on object-detection (:45091) and rag (:40100)
via tools/list, audit the stockroom photo with detect_objects, ingest the
dated audit entry into collection stockroom-audits, verify it is
retrievable with a natural-language question, then kill and restart
object-detection-service. Stop if any checkpoint assertion fails and
report which one.

Architecture

Prerequisites

Companion code

Checkpoint 1 — Discovery: ask every service what it can do

Checkpoint 2 — The audit: count what's on the shelf

Checkpoint 3 — The log: write the audit where memory lives

Checkpoint 4 — Memory: ask yesterday's question in natural language

Checkpoint 5 — Failure injection: blind the agent, then restore it

Ship it

Copy agent prompt