Home Playground Docs hub

Tutorial T-04 · Multi-service

Build a multilingual voice assistant

Verified against local · 2026-09-26 · evergreen-ai, linga, speech services 0.1.0–3.0.0 (workspace HEAD)
Scenario. Amara runs a community garden whose volunteers speak
English, Spanish, and Arabic. She wants to ask a question out loud (well,
typed — this tutorial keeps it scriptable), get an answer from the AI
assistant, translate it for a volunteer, and **have it spoken back in her
own voice** — so a shared device at the garden gate sounds like her in
every language. You will compose three services: Evergreen AI (the
brain), Linga (the translator), and Speech (voice print + TTS).

The shape to internalize: the same text pipeline serves every language, and the voice is data, not configuration. A voice print registered once rides along as a CDN path on every synthesis call.

Time: ~60 min · Level: advanced (long-running AI calls, model warm-up, multipart audio)

Architecture

text
  you ──► ┌──────────────── Evergreen AI :40071 ────────────────┐
          │  session (criteria-picked model) + chat message      │
          └───────┬─────────────────────────────────────────────┘
                  │ answer text (JSON, ~20–60 s on local models)
                  ▼
          ┌──────────────── Linga :40065 ───────────────────────┐
          │  POST /v3/translation/translate                     │
          │  en-US ──► es-ES (via Evergreen, single-turn)       │
          └───────┬─────────────────────────────────────────────┘
                  │ translated text
                  ▼
  your voice ──► ┌────────────── Speech :40161 ──────────────────┐
  (WAV upload,   │  voice print  ──►  CDN path (registered once) │
   voice print)  │  text-to-speech {input, reference_voiceprint} │
                 └───────┬────────────────────────────────────────┘
                         │ audio/mpeg (your voice, Spanish words)
                         ▼
                       speaker

Hidden edge: Speech's TTS also calls Linga internally (language detection on the input text). One service, two consumers — Checkpoint 5 makes that visible.

Prerequisites

evergreen-ai-service (:40071), linga-service (:40065), speech-service (:40161), Ollama (Evergreen's inference backend — it must be up before evergreen boots, or evergreen's model inventory comes up empty), and the CDN (:40040) — Speech persists voice prints there.

(the 1.7B default exceeds the 120 s synthesis deadline on a loaded dev box; both knobs are env vars):

``bash source /tmp/inaya_env.sh export INAYA_LABS_ENVIRONMENT=local export INAYA_LABS_CDN_SHARED_STORAGE_PATH=/tmp/inaya-cdn-storage export QWEN_TTS_MODEL_DIR="$HOME/.InayaLabs/models/hf/Qwen3-TTS-12Hz-0.6B-Base" export TTS_SYNTH_DEADLINE_SECS=300 nohup target/debug/speech-service > /tmp/speech-service.log 2>&1 & ``

TENANT_ID (verification used qa-tenant-t02).

(to validate audio). Any WAV with intelligible speech works for the voice print — noise-only files are rejected with 422.

≈ 30–90 s, first TTS synthesis ≈ 60–120 s (model warm-up). Every snippet sets generous -m timeouts; do not shorten them.

PITFALL

Two selection_criteria mistakes bite in quick succession. First: an empty criteria list 404s with "No suitable AI model found" — by design. The criteria list is the contract: pick at least one MustHave criterion or the selector refuses to guess. *Second, subtler: a loose criterion such as {"Domain": "GeneralPurpose"} can match a cloud* model**, and the chat call then fails with a 500 like OPENROUTER_API_KEY is required for OpenRouter chat when no OpenRouter key is configured. For a local sandbox, pin a local model you know is loaded, e.g. {"Model": {"optionality": "MustHave", "name": "Llama3_2_1B_Q4"}}. This is the same lever the Playground's Session Console exposes. Also: if evergreen booted while Ollama was down, its model inventory is empty — restart evergreen after starting Ollama.

Companion code

text
InayaLabs-website/docs-src/tutorials/_code/multilingual-voice-assistant/
├── voice_assistant.sh   ← the assembled driver (all checkpoints)
└── README.md            ← how to run it standalone

Download links: voice_assistant.sh · README.md

Checkpoint 1 — Ask the assistant (Evergreen AI session + chat)

Create an impromptu session — every session here is created in place, no predefined templates — then ask the garden question. Chat messages are multipart (user_message text field), and the response is the assistant message resource:

bash
KEYS=(-H "x-api-key: ${INAYA_API_KEY}" -H "x-client-app-key: ${INAYA_APP_KEY}")
sign_in() {
  curl -sf -X POST "${AUTH_URL}/v3/auth/qa/login" "${KEYS[@]}" \
    -H "Content-Type: application/json" \
    -d "{\"device_id\": \"t04-device-01\", \"locale\": \"en-US\", \"tenant_id\": \"${TENANT_ID}\", \"username\": \"$1\"}" \
    | python3 -c 'import json,sys; print(json.load(sys.stdin)["token"])'
}
TOKEN=$(sign_in t04-amara)

SESS=$(curl -sf -m 180 -X POST "${EVG_URL}/v3/sessions" \
  -H "Authorization: Bearer ${TOKEN}" "${KEYS[@]}" \
  -H "Content-Type: application/json" \
  -d '{
        "system_prompt": "You are a concise community-garden assistant. Answer in one or two short sentences.",
        "temperature": 0.2,
        "required_modalities": ["Text"],
        "selection_criteria": [
          {"Model": {"optionality": "MustHave", "name": "Llama3_2_1B_Q4"}}
        ]
      }')
SESSION_ID=$(python3 -c 'import json,sys; print(json.load(sys.stdin)["ai_session_id"])' <<< "${SESS}")
echo "session ${SESSION_ID:0:16}… created"

REPLY=$(curl -sf -m 240 -X POST "${EVG_URL}/v3/sessions/${SESSION_ID}/chat-messages" \
  -H "Authorization: Bearer ${TOKEN}" "${KEYS[@]}" \
  -F "user_message=What should we plant this week? It is late September.")
ANSWER=$(python3 -c 'import json,sys
d = json.load(sys.stdin)
assert d["author"] == "Assistant", d
print(d["content"][0]["Text"])' <<< "${REPLY}")
test -n "${ANSWER}" && echo "assistant: ${ANSWER}"
echo "${ANSWER}" | grep -qi "plant\|crop\|seed\|frost\|season" \
  && echo "checkpoint 1 passed: session answered a gardening question"

You should see — the session id, a one-or-two-sentence gardening answer (local models vary; the assertion keys on gardening vocabulary, not exact wording), then the checkpoint line.

NOTE

The response content is a list of typed blocks — [{"Text": "…"}] — not a bare string. Vision modalities add other block types; always match on the block shape you asked for (required_modalities: ["Text"] here).

Checkpoint 2 — Translate the answer (Linga)

Linga is itself an Evergreen client: it opens a single-turn session per translation, which is why Ollama must be healthy for this call too. Source and target are LanguageDescription objects (locale, optional jargon or vernacular registers — a farmer can get Agriculture jargon, a legal team LegalFinancial):

bash
ES=$(curl -sf -m 180 -X POST "${LINGA_URL}/v3/translation/translate" \
  -H "Authorization: Bearer ${TOKEN}" "${KEYS[@]}" \
  -H "Content-Type: application/json" \
  -d "$(python3 - "$ANSWER" <<'EOF'
import json, sys
print(json.dumps({
  "source": {"locale": "en-US"},
  "target": {"locale": "es-ES"},
  "text": sys.argv[1]
}))
EOF
)")
TRANSLATED=$(python3 -c 'import json,sys
d = json.load(sys.stdin)
assert d["original_text"], d
print(d["translated_text"])' <<< "${ES}")
echo "es: ${TRANSLATED}"
python3 - "$TRANSLATED" <<'EOF'
import sys
t = sys.argv[1]
assert any(c in t.lower() for c in "áéíóúñ¿¡") or " " in t, t
print("checkpoint 2 passed: answer translated for the Spanish-speaking volunteers")
EOF

You should see — the Spanish sentence (diacritics included), then the checkpoint line. The assertion accepts any well-formed translation output; exact wording is model-dependent.

PITFALL

Do not assert on exact translated strings. Local models are deterministic-ish per session (temperature 0.2 + fixed seed on the Evergreen side) but wording still drifts across model refreshes. Assert on structure: non-empty, diacritic-bearing, correct target_language.

Checkpoint 3 — Register the voice (Speech voice print)

The voice print is what makes the assistant sound like Amara instead of a default narrator. Registration takes a WAV with intelligible speech, conforms it to 24 kHz mono, verifies it transcribes, and persists WAV + tensor to the CDN. Make a sample with the macOS speech synthesizer (any clear recording works), then register:

bash
say -o /tmp/t04-voice.aiff \
  "Hi, this is Amara. I keep the community garden notes, and I speak three languages."
afconvert -f WAVE -d LEI16@22050 /tmp/t04-voice.aiff /tmp/t04-voice.wav

VP=$(curl -sf -m 300 -X POST "${SPEECH_URL}/v3/voice-prints" \
  -H "Authorization: Bearer ${TOKEN}" "${KEYS[@]}" \
  -F "file=@/tmp/t04-voice.wav")
VOICEPRINT_PATH=$(python3 -c 'import json,sys; print(json.load(sys.stdin)["voice_print_wav_path"])' <<< "${VP}")
echo "voice print: ${VOICEPRINT_PATH}"
grep -q "^speech-service/voice-prints/" <<< "${VOICEPRINT_PATH}" \
  && echo "checkpoint 3 passed: voice print registered and persisted to the CDN"

You should see — voice print: speech-service/voice-prints/<hash>/voice_print.wav, then the checkpoint line.

NOTE

The response path is a CDN path, not a local file — voice prints are tenant-scoped objects like any other CDN asset, which is how the same print follows Amara across devices. Registration is one-per-user: re-registering overwrites (idempotent for this tutorial's re-runs).

Checkpoint 4 — Speak the translation in Amara's voice

Now the payoff: synthesize the Spanish text with the English-speaking user's voice print. The response is an MP3 stream (audio/mpeg), and the first synthesis after boot includes model warm-up — allow real time:

bash
curl -sf -m 320 -o /tmp/t04-es.mp3 \
  -X POST "${SPEECH_URL}/v3/text-to-speech" \
  -H "Authorization: Bearer ${TOKEN}" "${KEYS[@]}" \
  -H "Content-Type: application/json" \
  -d "$(python3 - "$TRANSLATED" "$VOICEPRINT_PATH" <<'EOF'
import json, sys
print(json.dumps({
  "input": sys.argv[1],
  "reference_voiceprint_path": sys.argv[2]
}))
EOF
)"

python3 - <<'EOF'
data = open("/tmp/t04-es.mp3", "rb").read()
assert data[:3] == b"ID3" or data[:2] in (b"\xff\xfb", b"\xff\xf3", b"\xff\xf2"), data[:16]
print(f"audio: {len(data)} bytes, MP3 magic confirmed")
EOF
DUR=$(ffprobe -v error -show_entries format=duration -of csv=p=0 /tmp/t04-es.mp3)
python3 -c "d=float('${DUR}'); assert d > 1.0, d; print(f'checkpoint 4 passed: {d:.1f}s of Spanish speech synthesized in the registered voice')"

You should see — the byte count with MP3 magic confirmed, then checkpoint 4 passed: X.Xs of Spanish speech synthesized in the registered voice.

PITFALL

If TTS 504s with tts_synthesis_failed: synthesis timed out, the synthesis hit TTS_SYNTH_DEADLINE_SECS (default 120 s) — almost always model warm-up on first call or the 1.7B model on a loaded machine. The fix is the boot env from Prerequisites (QWEN_TTS_MODEL_DIR → 0.6B, TTS_SYNTH_DEADLINE_SECS → 300), not retry logic in the caller. And note the response is MP3 even though the voice print was WAV — match on magic bytes, not extension.

Checkpoint 5 — Failure injection: kill the translator both pipelines share

Kill linga-service and watch the blast radius: the translate call you wrote fails and the TTS call you didn't know depended on Linga (input language detection) fails too. One dependency, two consumers — exactly the kind of coupling a composed tutorial surfaces:

bash
pkill -x linga-service && echo "linga-service killed"

curl -s -o /dev/null --max-time 5 \
  -X POST "${LINGA_URL}/v3/translation/translate" \
  -H "Authorization: Bearer ${TOKEN}" "${KEYS[@]}" \
  -H "Content-Type: application/json" \
  -d '{"source":{"locale":"en-US"},"target":{"locale":"ar-SA"},"text":"Water the tomatoes twice a week."}' \
  || echo "translate during outage: connection refused (fails loudly)"

CODE=$(curl -s -o /tmp/t04-tts-down.json --max-time 60 -w "%{http_code}" \
  -X POST "${SPEECH_URL}/v3/text-to-speech" \
  -H "Authorization: Bearer ${TOKEN}" "${KEYS[@]}" \
  -H "Content-Type: application/json" \
  -d "$(python3 - "$TRANSLATED" "$VOICEPRINT_PATH" <<'EOF'
import json, sys
print(json.dumps({"input": sys.argv[1], "reference_voiceprint_path": sys.argv[2]}))
EOF
)")
echo "tts during outage: HTTP ${CODE} $(head -c 80 /tmp/t04-tts-down.json 2>/dev/null)"
test "${CODE}" != "200" \
  && echo "checkpoint 5a passed: one killed service broke two pipelines — Linga is a shared dependency"

source /tmp/inaya_env.sh
export INAYA_LABS_ENVIRONMENT=local
nohup /Users/test/projects/Software/InayaLabs/target/debug/linga-service > /tmp/linga-service.log 2>&1 &
sleep 3

ES2=$(curl -sf -m 180 -X POST "${LINGA_URL}/v3/translation/translate" \
  -H "Authorization: Bearer ${TOKEN}" "${KEYS[@]}" \
  -H "Content-Type: application/json" \
  -d '{"source":{"locale":"en-US"},"target":{"locale":"ar-SA"},"text":"Water the tomatoes twice a week."}')
python3 -c 'import json,sys
d = json.load(sys.stdin)
assert d["translated_text"], d
print("checkpoint 5b passed: translator restarted — Arabic translation flowing again")' <<< "${ES2}"

You should see — linga-service killed, the connection-refused line, a non-200 TTS code during the outage, both checkpoint lines, and finally an Arabic translation proving recovery.

NOTE

The correct resilience shape here is a caller-side fallback, not silent degradation: if language detection is unreachable, speak the text with a default voice rather than failing the request — or fail fast and let the UI say why. What a service must never do is hang: both failures above surface in seconds.

Ship it

the payloads are unchanged. Deployed Speech uses the 1.7B model with GPU backends — drop the QWEN_TTS_MODEL_DIR override.

POST /v3/transcriptions (audio file → text) and the loop is fully spoken: speech in → answer → translated → speech out in the caller's voice.

vernacular registers ({"locale":"en-US","jargon":"Agriculture"}) — the garden's irrigation schedule can be translated with the right vocabulary.

speech-service/voice-prints/ — DELETE /v3/cdn/files/{path} removes them.

Copy agent prompt

Follow tutorial T-04 "Build a multilingual voice assistant" at
/Users/test/projects/Software/InayaLabs/InayaLabs-website/docs/tutorials/multilingual-voice-assistant.html
end to end. Ask the garden question, translate the answer, register the
voice print, synthesize the translated answer, then kill and restart
linga-service. Stop if any checkpoint assertion fails and report which
one.