vibes.diy/docs Make an app → Discord
creator docs · reference · generated

Speech — the app speaks out loud and listens to a recording

This is the same document the Vibes code generator reads when it builds apps — rendered from prompts/pkg/llms/speech.md on every deploy, so it can't drift from what the platform actually does. Read it from your terminal any time with npx vibes-diy skills.

speak reads text aloud in a natural voice, and transcribe turns a recording into words. Both come from call-ai, beside callAI, and run on the platform's AI the same way: on the viewer's credits, with free guest credits on a visitor's first press. Call them from a press (a button, a submit), and show their errors like any callAI error, including reason: "sign-in".

jsimport { callAI, speak, transcribe, SPEECH_VOICES } from "call-ai";

Speaking out loud

speak(text, { voice, instructions, model }) resolves { blob, url, mimeType, durationMs }: a playable audio/wav Blob, an object URL for it, and how long it plays. <audio controls src={url} /> gives the person a player to press. To play it straight after the press that asked for it, the sound core's unlockAudio() runs first, synchronously, in that press: iOS Safari plays only sound a gesture started, and speak resolves after an await, too late for new Audio(url).play(). The unlocked context then plays the clip whenever it arrives:

js// lib/speech-player.js — plays a speak() clip on the context the press unlocked.
import { unlockAudio } from "./sfx.js";

export async function playClip(blob) {
  const ctx = unlockAudio(); // already running: the press called it first
  const buffer = await ctx.decodeAudioData(await blob.arrayBuffer());
  const source = ctx.createBufferSource();
  source.buffer = buffer;
  source.connect(ctx.destination);
  source.start();
  return new Promise((resolve) => (source.onended = resolve));
}
  • voice — one of SPEECH_VOICES: Kore (the default), Puck, Charon, Zephyr, Fenrir, Leda, Aoede, Sulafat and more. Offer a few in a Select when the person picks the voice.
  • instructions — how to say it, in plain words: "Say warmly", "Read slowly, like a bedtime story", "Whisper", "Sound curious".
  • model — "google/gemini-2.5-pro-tts" (the default) or "google/gemini-2.5-flash-tts", which answers sooner.

One call takes up to 4000 characters. Split a longer script into paragraphs and speak them in order, starting the next while the current one plays. Keep the url of a line the person may replay instead of speaking it again.

Listening to a recording

transcribe(blob, { instructions, model }) resolves the words spoken in the recording, as a string. Record with the browser's MediaRecorder on the stream openMic() opens inside the press (mic-capture's lib/media.js), and hand the Blob it produces straight over: Chrome records audio/webm, Safari audio/mp4, and both go to the model as they are, with nothing to convert. instructions is a line of guidance such as the names the speaker will say; model is an audio-capable id from listModels({ kind: "text" }), Gemini 3.5 Flash when absent.

A recording can be up to 10 MB in one call, about half an hour of speech; a longer one is answered at once with an error whose reason is "audio-too-large", so ask for a shorter take.

App.jsx — an interview: ask aloud, record the answer, read it back

jsximport React, { useEffect, useRef, useState } from "react";
import { callAI, speak, transcribe } from "call-ai";
import { Button, Stack, Text } from "@vibes.diy/ui";
import { openMic, stopStream, streamHelp } from "./lib/media.js";
import { unlockAudio } from "./lib/sfx.js";
import { playClip } from "./lib/speech-player.js";

export default function App() {
  const [question, setQuestion] = useState("What did you love most about this week?");
  const [answer, setAnswer] = useState("");
  const [recording, setRecording] = useState(false);
  const [busy, setBusy] = useState(false);
  const [message, setMessage] = useState("");
  const recorder = useRef(null);
  const streamRef = useRef(null);

  // The mic light goes out when the person leaves: unmount, another tab, a locked phone.
  useEffect(() => {
    const release = () => {
      if (recorder.current?.state === "recording") recorder.current.stop();
      stopStream(streamRef.current);
      streamRef.current = null;
      setRecording(false);
    };
    const hide = () => document.visibilityState === "hidden" && release();
    document.addEventListener("visibilitychange", hide);
    window.addEventListener("pagehide", release);
    return () => {
      document.removeEventListener("visibilitychange", hide);
      window.removeEventListener("pagehide", release);
      release();
    };
  }, []);

  async function ask() {
    unlockAudio(); // first line of the press, before any await
    setBusy(true);
    setMessage("");
    try {
      const { blob } = await speak(question, { voice: "Puck", instructions: "Ask warmly, like a curious friend" });
      await playClip(blob);
    } catch (error) {
      setMessage(error.message);
    } finally {
      setBusy(false);
    }
  }

  async function record() {
    setMessage("");
    const opened = await openMic();
    if (!opened.ok) return setMessage(streamHelp(opened.reason));
    streamRef.current = opened.stream;
    const chunks = [];
    const rec = new MediaRecorder(opened.stream);
    rec.ondataavailable = (e) => chunks.push(e.data);
    rec.onstop = async () => {
      stopStream(opened.stream);
      streamRef.current = null;
      setBusy(true);
      try {
        const words = await transcribe(new Blob(chunks, { type: rec.mimeType }));
        setAnswer(words);
        const next = await callAI(`The person answered "${words}" to "${question}". Ask one short follow-up question.`, {
          schema: { properties: { question: { type: "string" } }, required: ["question"] },
        });
        setQuestion(JSON.parse(next).question);
      } catch (error) {
        setMessage(error.message);
      } finally {
        setBusy(false);
      }
    };
    rec.start();
    recorder.current = rec;
    setRecording(true);
  }

  function stop() {
    recorder.current?.stop();
    setRecording(false);
  }

  return (
    <Stack>
      <Text>{question}</Text>
      <Button onClick={ask} loading={busy && !recording}>Ask me</Button>
      <Button onClick={recording ? stop : record} disabled={busy}>{recording ? "Done answering" : "Answer out loud"}</Button>
      {answer && <Text>You said: {answer}</Text>}
      {message && <Text>{message}</Text>}
    </Stack>
  );
}

A recording inside a callAI prompt

To ask the model about a recording in one call, put the audio in the prompt as an input_audio part beside a text part, and name an audio-capable model:

jsconst response = await callAI(
  [
    { type: "text", text: "Summarize what the speaker decided." },
    { type: "input_audio", input_audio: { data: base64, format: "webm" } },
  ],
  { model: "google/gemini-3.5-flash", schema: { properties: { summary: { type: "string" } } } }
);

data is the recording's bytes in base64 with no data: prefix (read a FileReader data URL and keep what follows the comma), and format is its container: "webm", "m4a" for Safari's audio/mp4, "wav", "mp3" or "ogg".