Speech — the app speaks out loud and listens to a recording
prompts/pkg/llms/speech.md on every deploy, so it
can't drift from what the platform actually does. Read it from your terminal any
time with npx vibes-diy skills.
speak reads text aloud in a natural voice, and transcribe turns a recording
into words. Both come from call-ai, beside callAI, and run on the
platform's AI the same way: on the viewer's credits, with free guest credits on
a visitor's first press. Call them from a press (a button, a submit), and show
their errors like any callAI error, including reason: "sign-in".
jsimport { callAI, speak, transcribe, SPEECH_VOICES } from "call-ai";
Speaking out loud
speak(text, { voice, instructions, model }) resolves
{ blob, url, mimeType, durationMs }: a playable audio/wav Blob, an object URL
for it, and how long it plays. <audio controls src={url} /> gives the person
a player to press. To play it straight after the press that asked for it, the
sound core's unlockAudio() runs first, synchronously, in that press: iOS
Safari plays only sound a gesture started, and speak resolves after an
await, too late for new Audio(url).play(). The unlocked context then plays
the clip whenever it arrives:
js// lib/speech-player.js — plays a speak() clip on the context the press unlocked.
import { unlockAudio } from "./sfx.js";
export async function playClip(blob) {
const ctx = unlockAudio(); // already running: the press called it first
const buffer = await ctx.decodeAudioData(await blob.arrayBuffer());
const source = ctx.createBufferSource();
source.buffer = buffer;
source.connect(ctx.destination);
source.start();
return new Promise((resolve) => (source.onended = resolve));
}
voice— one ofSPEECH_VOICES: Kore (the default), Puck, Charon, Zephyr, Fenrir, Leda, Aoede, Sulafat and more. Offer a few in aSelectwhen the person picks the voice.instructions— how to say it, in plain words:"Say warmly","Read slowly, like a bedtime story","Whisper","Sound curious".model—"google/gemini-2.5-pro-tts"(the default) or"google/gemini-2.5-flash-tts", which answers sooner.
One call takes up to 4000 characters. Split a longer script into paragraphs
and speak them in order, starting the next while the current one plays. Keep
the url of a line the person may replay instead of speaking it again.
Listening to a recording
transcribe(blob, { instructions, model }) resolves the words spoken in the
recording, as a string. Record with the browser's MediaRecorder on the stream
openMic() opens inside the press (mic-capture's lib/media.js), and hand the
Blob it produces straight over: Chrome records audio/webm, Safari
audio/mp4, and both go to the model as they are, with nothing to convert.
instructions is a line of guidance such as the names the speaker will say;
model is an audio-capable id from listModels({ kind: "text" }), Gemini 3.5
Flash when absent.
A recording can be up to 10 MB in one call, about half an hour of speech; a
longer one is answered at once with an error whose reason is
"audio-too-large", so ask for a shorter take.
App.jsx — an interview: ask aloud, record the answer, read it back
jsximport React, { useEffect, useRef, useState } from "react";
import { callAI, speak, transcribe } from "call-ai";
import { Button, Stack, Text } from "@vibes.diy/ui";
import { openMic, stopStream, streamHelp } from "./lib/media.js";
import { unlockAudio } from "./lib/sfx.js";
import { playClip } from "./lib/speech-player.js";
export default function App() {
const [question, setQuestion] = useState("What did you love most about this week?");
const [answer, setAnswer] = useState("");
const [recording, setRecording] = useState(false);
const [busy, setBusy] = useState(false);
const [message, setMessage] = useState("");
const recorder = useRef(null);
const streamRef = useRef(null);
// The mic light goes out when the person leaves: unmount, another tab, a locked phone.
useEffect(() => {
const release = () => {
if (recorder.current?.state === "recording") recorder.current.stop();
stopStream(streamRef.current);
streamRef.current = null;
setRecording(false);
};
const hide = () => document.visibilityState === "hidden" && release();
document.addEventListener("visibilitychange", hide);
window.addEventListener("pagehide", release);
return () => {
document.removeEventListener("visibilitychange", hide);
window.removeEventListener("pagehide", release);
release();
};
}, []);
async function ask() {
unlockAudio(); // first line of the press, before any await
setBusy(true);
setMessage("");
try {
const { blob } = await speak(question, { voice: "Puck", instructions: "Ask warmly, like a curious friend" });
await playClip(blob);
} catch (error) {
setMessage(error.message);
} finally {
setBusy(false);
}
}
async function record() {
setMessage("");
const opened = await openMic();
if (!opened.ok) return setMessage(streamHelp(opened.reason));
streamRef.current = opened.stream;
const chunks = [];
const rec = new MediaRecorder(opened.stream);
rec.ondataavailable = (e) => chunks.push(e.data);
rec.onstop = async () => {
stopStream(opened.stream);
streamRef.current = null;
setBusy(true);
try {
const words = await transcribe(new Blob(chunks, { type: rec.mimeType }));
setAnswer(words);
const next = await callAI(`The person answered "${words}" to "${question}". Ask one short follow-up question.`, {
schema: { properties: { question: { type: "string" } }, required: ["question"] },
});
setQuestion(JSON.parse(next).question);
} catch (error) {
setMessage(error.message);
} finally {
setBusy(false);
}
};
rec.start();
recorder.current = rec;
setRecording(true);
}
function stop() {
recorder.current?.stop();
setRecording(false);
}
return (
<Stack>
<Text>{question}</Text>
<Button onClick={ask} loading={busy && !recording}>Ask me</Button>
<Button onClick={recording ? stop : record} disabled={busy}>{recording ? "Done answering" : "Answer out loud"}</Button>
{answer && <Text>You said: {answer}</Text>}
{message && <Text>{message}</Text>}
</Stack>
);
}
A recording inside a callAI prompt
To ask the model about a recording in one call, put the audio in the prompt as
an input_audio part beside a text part, and name an audio-capable model:
jsconst response = await callAI(
[
{ type: "text", text: "Summarize what the speaker decided." },
{ type: "input_audio", input_audio: { data: base64, format: "webm" } },
],
{ model: "google/gemini-3.5-flash", schema: { properties: { summary: { type: "string" } } } }
);
data is the recording's bytes in base64 with no data: prefix (read a
FileReader data URL and keep what follows the comma), and format is its
container: "webm", "m4a" for Safari's audio/mp4, "wav", "mp3" or
"ogg".