Interactive Avatar: Pipecat Plugin reference
Reference for pipecat-synthesia, the Synthesia plugin for Pipecat pipelines, for Python.
Use this page to add an Interactive Avatar to a Pipecat bot. For the mental model, see Concepts. The plugin’s README is the full guide, with every event, error and edge case, and its examples/ folder has a runnable bot for each join mode.
SynthesiaVideoService, SynthesiaError and ErrorType live in pipecat_synthesia. The room adapters live in pipecat_synthesia.livekit.
Requirements
- Python 3.11 or later.
pipecat-ai1.14 or later, below 2.0. Tested with 1.14, 1.15 and 1.16.- A LiveKit project, for both join modes. LiveKit Cloud’s free tier is fine.
- A Synthesia API key with Interactive Avatar access, and an Interactive Avatar id (see Choosing an avatar).
Install the plugin from PyPI with its [livekit] extra, which both join modes need:
On a Pipecat minor newer than the tested ones, the plugin logs one warning at startup. A new Pipecat minor can break the plugin before it has been tested, so for stability pin pipecat-ai to a tested minor (for example pipecat-ai==1.15.*). The plugin follows SemVer: pin it with pipecat-synthesia[livekit]>=1.0,<2.
Choosing a join mode
The avatar always runs in a LiveKit room. Which room depends on how your users connect to your bot.
Use a direct join if your users connect over LiveKit, or if you haven’t chosen a transport yet. A relayed join over Pipecat’s LiveKit transport is untested and not supported.
Environment variables
Read when the corresponding argument isn’t passed explicitly.
Keep the keys and secrets in your bot’s environment or a secret manager, never in frontend code. The plugin mints the avatar’s LiveKit token inside your process, so your LiveKit secret never leaves it. LIVEKIT_URL accepts https:// as well as wss://.
On a direct join, the LiveKit key and secret must come from the project your bot’s room is in.
Direct join
Place the avatar straight after your TTS (or your speech-to-speech model), and the input filter straight after transport.input():
A direct join needs three things:
room.input_filter()straight aftertransport.input(). The avatar is a participant in your room, so its voice reaches your bot’s input. Without the filter, the bot transcribes the avatar and replies to itself.- A bot token with the
kind="agent"claim. Without it, the avatar worker counts your bot as a human, so the session doesn’t end when the real user leaves, and keeps billing.LiveKitTransportRoom(transport)raisesValueErrorat startup for a token without it. - LiveKit credentials from the project your bot’s room is in.
kind is a top-level claim, not the agent grant:
Pipecat’s development runner (python bot.py -t livekit) mints a bot token without this claim, so a direct join can’t run under it. Mint the token yourself and start the bot directly, as the repository’s examples/bot_direct_join.py does. LiveKit Cloud’s agent-dispatch tokens already carry the claim.
The avatar is a room participant too, and usually the first to arrive, so on_first_participant_joined fires for it. Greet the first participant that isn’t avatar.avatar_identity instead. Don’t end the call the moment the last human leaves: a dropped connection that reconnects leaves and rejoins moments apart, and ending in between takes the avatar away. The avatar worker waits about 5 seconds before ending the session itself, so end your call when on_session_ended reports no_humans_left, or wait at least that long. Waiting can’t help when the human’s app leaves on purpose (for example by calling disconnect()): the avatar worker then ends the session at once. The repository’s examples/bot_direct_join.py shows both.
Relayed join
The avatar renders at 512×512 and 25 fps. Set all five of these on your transport’s params (TransportParams, or DailyParams for Daily):
Without video_out_enabled=True your users see no avatar. With another size or frame rate, Pipecat resizes every frame: the avatar looks stretched, a CPU core is pinned, and latency grows over the call. The plugin logs a warning at startup if the settings don’t match.
Speech-to-speech models (for example OpenAI Realtime) go in the same slot: replace stt, llm and tts with the model and put avatar straight after it. Voice activity detection belongs on the user aggregator, LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()). The transport params silently ignore a vad_analyzer, and the bot then never detects speech.
SynthesiaVideoService
Invalid arguments, such as a malformed avatar id, a missing API key, an API URL outside synthesia.io, or an unknown keyword like the LiveKit plugin’s join_timeout, raise ValueError or TypeError at once.
If the avatar leaves mid-call (the worker failed, the session reached its length cap, or credits ran out), the bot carries on voice-only whatever required says, and you get on_avatar_disconnected. To end the call instead, push an EndWorkerFrame upstream from that handler.
await avatar.swap_avatar(avatar_id, *, timeout=None)
The session always starts on the first id in avatar_ids, so list the one you want first. swap_avatar() switches to another id from avatar_ids, or back to the first with "default". timeout defaults to 15 seconds. It needs an active session: called before on_avatar_connected, it raises. A failed swap raises, reports through on_error, and keeps the previous avatar. swap_avatar() hasn’t yet been tested against the live Synthesia API.
Room adapters
Events
The plugin also logs metrics.summary() once per session, in a line containing synthesia session summary. Once the Synthesia API has created the session, it carries the session and request ids; quote it when contacting support.
Errors
Once the service is running, every failure is a SynthesiaError with a type from ErrorType. In an on_error handler, read it from frame.exception, not from the ErrorFrame.
For the HTTP-level error codes, see Errors.
Troubleshooting
Choosing an avatar
avatar_ids takes Interactive Avatar ids. A Studio avatar id is not one: the two look alike, and a Studio id fails with UNKNOWN_AVATAR about four seconds into the call. To get an Interactive Avatar id, see Finding your avatar ID.
The plugin doesn’t choose a voice. Pick a TTS voice in your pipeline that fits the avatar.
Limits
- Concurrent sessions: your plan sets how many sessions can run at once; see Operational trust. Going over it fails with
429 concurrency_limit(CONCURRENCY_LIMIT). - Ending a session: there’s no end-session API. The session ends, and billing stops, about 5 seconds after your bot leaves the room or, on a direct join, after the last human leaves (at once if their app calls
disconnect()). - A human who leaves during the launch isn’t noticed. On a direct join, if the last human leaves before the avatar worker has joined, the session runs, and bills, until your bot leaves or the session reaches its length cap. End the call yourself when nobody is left, as the repository’s
examples/_presence.pydoes. - Session length cap: set by your plan.
- Pipecat ends an idle call after 300 seconds without speech. Change it with
PipelineWorker(idle_timeout_secs=...).
The README’s known limitations cover rarer cases.
Deploying
- Declare the dependency. Container and cloud builds install only what your project declares, so use
uv add(or your requirements file) rather than an environment-only install. - Handle SIGTERM. Use
WorkerRunner(handle_sigterm=True), thenawait runner.add_workers(worker)andawait runner.run(). Without it, a container stop kills the bot before the avatar session ends. Under Pipecat’s development runner the runner owns the signals instead; see the deployment notes. - Network: outbound only. HTTPS to
developers.synthesia.io, and WSS plus UDP (with a TCP/443 fallback) to your LiveKit project.
The repository’s deployment notes cover containers, Pipecat Cloud, egress rules and more troubleshooting.
What the plugin sends
Every request the plugin makes to the Synthesia API identifies it in its User-Agent and X-Synthesia-Client headers: the plugin’s, Pipecat’s and Python’s versions, the operating system and processor architecture, the join mode, and the kind of user transport (such as livekit or daily). Synthesia uses them to see which plugin versions are in use before changing anything those versions depend on.
An avatar token the plugin mints carries the plugin’s name and version, its protocol level, the join mode and a random id for each launch, as participant attributes that everyone in the room can see. An avatar token you provide carries only what you put in it. The full list is in the repository’s protocol notes, sections 4 and 6.
Additional resources
pipecat-synthesiaon GitHub: the README, examples and changelog. MIT licensed.pipecat-synthesiaon PyPI- Pipecat documentation
- Support: GitHub issues are turned off. Email support@synthesia.io and say it’s about
pipecat-synthesia; the repository’s SUPPORT.md lists what to include. To report a security issue, see SECURITY.md.