Transcribe

transcribe() is an input-phase Official Capability for audio. It turns audio message parts into transcript text before the Agent Driver receives the final input.

The Capability finds audio parts in input messages, transcribes them, appends transcript text to the message, and records transcription results in invocation context. It can also persist transcript and source-audio artifacts into a writable Workspace.

Configure transcription

Provide either an AI SDK transcription model configuration or an execute() function. The example keeps artifacts off, so it does not require a writable Workspace.

import { defineAgent } from 'vite-hub/agent'
import { transcribe } from 'vite-hub/agent/capabilities'

export default defineAgent({
  driver: { model },
  capabilities: [
    transcribe({
      model: transcriptionModel,
    }),
  ],
})

OpenRouter

Use openRouterTranscriptionModel() to keep OpenRouter authentication, audio encoding, request shape, and provider errors behind the AI SDK transcription model interface.

import { defineAgent } from 'vite-hub/agent'
import {
  openRouterTranscriptionModel,
  transcribe,
} from 'vite-hub/agent/capabilities'

export default defineAgent({
  driver: { model },
  capabilities: [
    transcribe({
      model: openRouterTranscriptionModel({
        apiKey: () => env.OPENROUTER_API_KEY,
        model: 'openai/gpt-4o-transcribe',
      }),
    }),
  ],
})

The adapter uses OpenRouter's base64 JSON request with response_format: 'json'. This avoids provider-specific multipart handling in Agent Definitions and does not request verbose_json, which OpenRouter only supports for some upstream providers.

AI SDK providerOptions.openrouter values for language, temperature, and provider routing are forwarded to OpenRouter. Unsupported OpenRouter transcription options fail explicitly instead of being silently ignored.

How transcription works

transcribe() runs before model execution. It enforces the configured maximum audio size, resolves audio data from direct data, fetchData, or URL, and replaces the consumed audio parts with transcript text in the user message.

When artifacts are enabled, it writes sanitized transcript and optional audio files to the Agent's writable Workspace and exposes results as a finish extension.

Use artifacts.directory to keep the transcript and source audio together. The generated paths share a sanitized timestamp/message stem.

transcribe({
  model: transcriptionModel,
  artifacts: {
    directory: 'inputs/voice-notes',
    transcript: { format: 'markdown' },
  },
})

Streaming transcription

Use streamTranscription() for live raw audio. It wraps AI SDK streaming transcription and exposes textStream, an append-only text stream that can be passed directly to event.reply().

import type { AgentFinishHookEvent } from 'vite-hub/agent'
import { streamTranscription } from 'vite-hub/agent/capabilities'

export async function liveTranscriptReply(
  event: AgentFinishHookEvent,
  audio: ReadableStream<Uint8Array>,
) {
  const transcription = await streamTranscription({
    model: 'openai/gpt-realtime-whisper',
    audio,
    inputAudioFormat: {
      type: 'audio/pcm',
      rate: 24_000,
    },
  })

  return event.reply(transcription.textStream)
}

Return the reply intent without awaiting transcription.text; consuming the reply drives the provider stream and resolves the final text promise. Chat-backed Channels use native streaming when the adapter supports it and Chat SDK's post-and-edit fallback otherwise. Other message Channels use their native stream method when available and fall back to one final reply.

streamTranscription() emits provider transcript-delta events as reply chunks. Providers that only emit partial and final snapshots produce one final reply, which avoids duplicating corrected partial text.

Asynchronous remote transcription

Use createTranscription() to submit a private remote object from a durable workflow and resume after the provider completes it. The client returns the provider operation ID used for acknowledgement and idempotency, then normalizes an authenticated completion payload into a provider-neutral transcript or failure.

import {
  createTranscription,
  elevenLabsScribe,
} from 'vite-hub/agent/capabilities'

const transcription = createTranscription({
  driver: elevenLabsScribe({
    apiKey: () => env.ELEVENLABS_API_KEY,
    diarize: true,
    tagAudioEvents: true,
    timestampsGranularity: 'word',
    webhookId: env.ELEVENLABS_WEBHOOK_ID,
  }),
})

const submission = await transcription.submit({
  metadata: { attemptId, jobId },
  source: { url: signedAudioUrl },
})

const completion = await transcription.receive(authenticatedProviderPayload)

if (completion.status === 'failed') {
  console.error(completion.error.code, completion.error.message)
}

submit() never downloads the remote object into the application process. Use a signed HTTPS URL for private Blob objects, with an expiry long enough for the provider to fetch it.

The caller still owns callback authentication before receive(), durable operation state, duplicate-delivery handling, timeouts, and workflow resumption. Compose those concerns with the Workflow primitive; correlation metadata is untrusted until it matches the stored workflow attempt. Provider callback payloads and SDK types do not cross the transcription client interface. Failed completions contain a ViteHubError with a fixed TRANSCRIPTION_* code and message. Raw provider diagnostics stay behind the in-memory cause and are omitted when the completion is serialized.

Requirements

Basic transcription requires a model or custom executor. Artifact persistence requires an explicit writable Workspace.

Streaming transcription requires an AI SDK streaming transcription model, a ReadableStream<Uint8Array | string> of raw audio, and its input audio format.

Asynchronous remote transcription requires a TranscriptionDriver. The built-in ElevenLabs Scribe driver requires an API key and an explicitly configured speech-to-text webhook ID.

Audio data must stay within maxBytes. Artifact paths must stay inside the Workspace and cannot target reserved .git or .vitehub paths.

Driver support

Asynchronous transcription is independent of the Agent Driver because the caller composes its submitted operation and completion result into a durable Workflow.

Verify transcription

Run an invocation with one audio part and inspect the final message text. Confirm that the transcript appears before the Agent Driver runs.

When artifacts are enabled, inspect the Workspace for transcript files and the finish extension for transcription metadata.

Options

Asynchronous client

openRouterTranscriptionModel({ apiKey, model }) returns an AI SDK transcription model for synchronous transcribe({ model }) usage.

Without artifacts.directory, transcripts use transcripts/<date>/<stem>.txt and audio is placed beside the transcript. If transcripts are disabled, audio uses audio/<date>/<stem>.<extension>.

Public helpers

Import these helpers from @vite-hub/agent/capabilities when custom hooks or executors need the same normalized data as the Capability.

Each TranscriptionResult contains createdAt, date, messageId, stem, and transcript, plus audioPath or transcriptPath when those artifacts were written.

Related pages

• AI Gateway streaming transcription
• Workspace primitive
• Agent invocations