Skip to navigation

Interactive Avatar: Pipecat Plugin reference

Reference for pipecat-synthesia, the Synthesia plugin for Pipecat pipelines, for Python.

Use this page to add an Interactive Avatar to a Pipecat bot. For the mental model, see Concepts. The plugin’s README is the full guide, with every event, error and edge case, and its examples/ folder has a runnable bot for each join mode.

SynthesiaVideoService, SynthesiaError and ErrorType live in pipecat_synthesia. The room adapters live in pipecat_synthesia.livekit.

Requirements

  • Python 3.11 or later.
  • pipecat-ai 1.14 or later, below 2.0. Tested with 1.14, 1.15 and 1.16.
  • A LiveKit project, for both join modes. LiveKit Cloud’s free tier is fine.
  • A Synthesia API key with Interactive Avatar access, and an Interactive Avatar id (see Choosing an avatar).

Install the plugin from PyPI with its [livekit] extra, which both join modes need:

uv add "pipecat-synthesia[livekit]"
# or
pip install "pipecat-synthesia[livekit]"

On a Pipecat minor newer than the tested ones, the plugin logs one warning at startup. A new Pipecat minor can break the plugin before it has been tested, so for stability pin pipecat-ai to a tested minor (for example pipecat-ai==1.15.*). The plugin follows SemVer: pin it with pipecat-synthesia[livekit]>=1.0,<2.

Choosing a join mode

The avatar always runs in a LiveKit room. Which room depends on how your users connect to your bot.

Direct joinRelayed join
Your users connect overLiveKitDaily or SmallWebRTC (other transports are untested)
Room adapterLiveKitTransportRoom(transport)PrivateLiveKitRoom()
How the video reaches usersThe avatar publishes it straight into your roomThrough your bot, as Pipecat frames
LiveKit projectThe one your users’ room is inAny project of yours; it only hosts the private rooms
Extra wiringroom.input_filter(), and a bot token you mintFive video_out_* transport settings

Use a direct join if your users connect over LiveKit, or if you haven’t chosen a transport yet. A relayed join over Pipecat’s LiveKit transport is untested and not supported.

Environment variables

Read when the corresponding argument isn’t passed explicitly.

VariableUsed for
SYNTHESIA_API_KEYYour Synthesia workspace API key
SYNTHESIA_API_URLAPI base URL. Defaults to https://developers.synthesia.io; leave it unset
SYNTHESIA_ALLOW_ANY_API_HOSTSet to 1 to let SYNTHESIA_API_URL point outside synthesia.io, for example at your own proxy
LIVEKIT_URLYour LiveKit project URL, on a relayed join. A direct join uses the URL your LiveKitTransport was given
LIVEKIT_API_KEYLiveKit API key
LIVEKIT_API_SECRETLiveKit API secret

Keep the keys and secrets in your bot’s environment or a secret manager, never in frontend code. The plugin mints the avatar’s LiveKit token inside your process, so your LiveKit secret never leaves it. LIVEKIT_URL accepts https:// as well as wss://.

On a direct join, the LiveKit key and secret must come from the project your bot’s room is in.

Direct join

Place the avatar straight after your TTS (or your speech-to-speech model), and the input filter straight after transport.input():

from pipecat.pipeline.pipeline import Pipeline
from pipecat.transports.livekit.transport import LiveKitParams, LiveKitTransport
from pipecat_synthesia import SynthesiaVideoService
from pipecat_synthesia.livekit import LiveKitTransportRoom
transport = LiveKitTransport(
url, bot_token, room_name, LiveKitParams(audio_in_enabled=True, audio_out_enabled=True)
)
room = LiveKitTransportRoom(transport)
avatar = SynthesiaVideoService(avatar_ids=["<avatar-id>"], room=room)
pipeline = Pipeline(
[
transport.input(),
room.input_filter(), # keeps the avatar's voice off the bot's input
stt,
context_aggregator.user(),
llm,
tts,
avatar, # straight after the TTS
transport.output(),
context_aggregator.assistant(),
]
)

A direct join needs three things:

  1. room.input_filter() straight after transport.input(). The avatar is a participant in your room, so its voice reaches your bot’s input. Without the filter, the bot transcribes the avatar and replies to itself.
  2. A bot token with the kind="agent" claim. Without it, the avatar worker counts your bot as a human, so the session doesn’t end when the real user leaves, and keeps billing. LiveKitTransportRoom(transport) raises ValueError at startup for a token without it.
  3. LiveKit credentials from the project your bot’s room is in.

kind is a top-level claim, not the agent grant:

from livekit.api import AccessToken, VideoGrants
bot_token = (
AccessToken(api_key, api_secret)
.with_identity("pipecat-bot")
.with_kind("agent")
.with_grants(VideoGrants(room_join=True, room=room_name, agent=True))
.to_jwt()
)

Pipecat’s development runner (python bot.py -t livekit) mints a bot token without this claim, so a direct join can’t run under it. Mint the token yourself and start the bot directly, as the repository’s examples/bot_direct_join.py does. LiveKit Cloud’s agent-dispatch tokens already carry the claim.

The avatar is a room participant too, and usually the first to arrive, so on_first_participant_joined fires for it. Greet the first participant that isn’t avatar.avatar_identity instead. Don’t end the call the moment the last human leaves: a dropped connection that reconnects leaves and rejoins moments apart, and ending in between takes the avatar away. The avatar worker waits about 5 seconds before ending the session itself, so end your call when on_session_ended reports no_humans_left, or wait at least that long. Waiting can’t help when the human’s app leaves on purpose (for example by calling disconnect()): the avatar worker then ends the session at once. The repository’s examples/bot_direct_join.py shows both.

Relayed join

from pipecat.pipeline.pipeline import Pipeline
from pipecat_synthesia import SynthesiaVideoService
from pipecat_synthesia.livekit import PrivateLiveKitRoom
avatar = SynthesiaVideoService(
avatar_ids=["<avatar-id>"],
room=PrivateLiveKitRoom(), # reads LIVEKIT_URL, LIVEKIT_API_KEY and LIVEKIT_API_SECRET
)
pipeline = Pipeline(
[transport.input(), stt, context_aggregator.user(), llm, tts, avatar, transport.output(), context_aggregator.assistant()]
)

The avatar renders at 512×512 and 25 fps. Set all five of these on your transport’s params (TransportParams, or DailyParams for Daily):

video_out_enabled=True,
video_out_is_live=True,
video_out_width=512,
video_out_height=512,
video_out_framerate=25,

Without video_out_enabled=True your users see no avatar. With another size or frame rate, Pipecat resizes every frame: the avatar looks stretched, a CPU core is pinned, and latency grows over the call. The plugin logs a warning at startup if the settings don’t match.

Speech-to-speech models (for example OpenAI Realtime) go in the same slot: replace stt, llm and tts with the model and put avatar straight after it. Voice activity detection belongs on the user aggregator, LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()). The transport params silently ignore a vad_analyzer, and the bot then never detects speech.

SynthesiaVideoService

SynthesiaVideoService(
*,
avatar_ids,
room,
api_key=None,
api_url=None,
allow_any_api_host=None,
launch_timeout=90.0,
required=True,
session=None,
settings=None,
)
ParameterTypeDefaultDescription
avatar_idsSequence[str]requiredOne to five Interactive Avatar ids. The first is shown; the others can be switched to with swap_avatar(). The av_ prefix is added for you.
roomroom adapterrequiredLiveKitTransportRoom(transport) for a direct join, or PrivateLiveKitRoom() for a relayed join.
api_keystr | NoneNoneFalls back to SYNTHESIA_API_KEY.
api_urlstr | NoneNoneFalls back to SYNTHESIA_API_URL, then https://developers.synthesia.io. Must be https:// to synthesia.io or a host under it, unless allow_any_api_host is set.
allow_any_api_hostbool | NoneNoneFalls back to SYNTHESIA_ALLOW_ANY_API_HOST, then False. Lets api_url be an https:// host outside synthesia.io, such as your own proxy. Your API key and a LiveKit token for the session go to that host, and the plugin logs a warning.
launch_timeoutfloat90.0Seconds for the whole launch: connecting, creating the session (with retries) and the avatar joining.
requiredboolTrueWhether the avatar is required to start. True: if it can’t start, the service ends the call, and none of the bot’s speech plays from then on. False: the bot carries on voice-only.
sessionaiohttp.ClientSession | NoneNoneBring your own HTTP session.
settingsSynthesiaVideoSettings | NoneNoneInitial settings.

Invalid arguments, such as a malformed avatar id, a missing API key, an API URL outside synthesia.io, or an unknown keyword like the LiveKit plugin’s join_timeout, raise ValueError or TypeError at once.

If the avatar leaves mid-call (the worker failed, the session reached its length cap, or credits ran out), the bot carries on voice-only whatever required says, and you get on_avatar_disconnected. To end the call instead, push an EndWorkerFrame upstream from that handler.

await avatar.swap_avatar(avatar_id, *, timeout=None)

The session always starts on the first id in avatar_ids, so list the one you want first. swap_avatar() switches to another id from avatar_ids, or back to the first with "default". timeout defaults to 15 seconds. It needs an active session: called before on_avatar_connected, it raises. A failed swap raises, reports through on_error, and keeps the previous avatar. swap_avatar() hasn’t yet been tested against the live Synthesia API.

Room adapters

AdapterSignatureNotes
LiveKitTransportRoom(transport, *, avatar_token=None, credentials=None, livekit_url=None, avatar_identity=None, avatar_name="Synthesia avatar", connect_timeout=30.0)Direct join. Mints the avatar’s token from credentials (a LiveKitCredentials), else LIVEKIT_API_KEY and LIVEKIT_API_SECRET. room.input_filter() returns the filter for your pipeline.
PrivateLiveKitRoom(*, url=None, api_key=None, api_secret=None, room_name=None, bot_identity="pipecat-bot", avatar_identity="synthesia-avatar-agent", avatar_name="Synthesia avatar", empty_timeout_secs=120, departure_timeout_secs=20, delete_room_on_close=True, connect_timeout=15.0)Relayed join. Reads LIVEKIT_URL, LIVEKIT_API_KEY and LIVEKIT_API_SECRET when not passed. Creates a private room for each session and deletes it at the end.

Events

@avatar.event_handler("on_avatar_connected")
async def on_avatar_connected(service, info):
logger.info(f"avatar is live: session {info.session_id}")
@avatar.event_handler("on_metrics")
async def on_metrics(service, metrics):
logger.info(f"avatar metrics: {metrics.summary()}")
EventFiresHandler arguments
on_avatar_connectedAt most once, when the Synthesia API has created the session. The avatar’s audio can start up to about half a second earlier.service, info (session_id, avatar_identity, active_avatar_id, room_name)
on_avatar_readyOnce, when every avatar in avatar_ids is readyservice, info (session_id, status)
on_avatar_disconnectedAt most once, when the avatar leaves an active sessionservice, reason: avatar_left_room, avatar_video_stopped or room_connection_lost
on_session_endedExactly onceservice, reason: an end reason, below
on_metricsOnce, right after on_session_endedservice, metrics: call metrics.summary()
on_errorPer error. Pipecat’s own eventservice, frame: an ErrorFrame whose .exception is a SynthesiaError
End reasonMeaning
pipeline_endedYour pipeline ended normally.
pipeline_cancelledYour pipeline was cancelled, including a stop by signal.
pipeline_cleanupPipecat cleaned up the service without ending or cancelling it first.
no_humans_leftDirect join: the last human left, so the avatar worker ended the session. A normal end, with no error.
avatar_leftThe avatar left unexpectedly: on a direct join, while humans were still in the room 2 seconds later (at once if the bot lost its own connection to the room); on a relayed join, any leave. The bot continues voice-only.
launch_failedThe avatar couldn’t start, with required=True. The service ends the call.
launch_timeoutThe launch took longer than launch_timeout, or a step in it timed out (an API request, or the avatar not joining within 30 seconds of the session being created), with required=True. The service ends the call.
degradedThe avatar couldn’t start, with required=False. The bot continues voice-only.

The plugin also logs metrics.summary() once per session, in a line containing synthesia session summary. Once the Synthesia API has created the session, it carries the session and request ids; quote it when contacting support.

Errors

Once the service is running, every failure is a SynthesiaError with a type from ErrorType. In an on_error handler, read it from frame.exception, not from the ErrorFrame.

ErrorTypeMeaningRetryable
AUTHThe API key is invalid, or isn’t allowed to use Interactive Avatars.no
FEATURE_NOT_IN_PLANYour workspace’s plan doesn’t include Interactive Avatars.no
QUOTA_EXCEEDEDYour workspace is out of quota or credits.no
UNKNOWN_AVATARAn avatar id isn’t available to your workspace, or is a Studio avatar id.no
INVALID_ROOM_TOKENA LiveKit token expired before the launch, or doesn’t name the bot’s room.no
LIVEKIT_CREDENTIALS_REJECTEDRelayed join: your LiveKit project refused to create the private room.no
INVALID_SESSION_REQUESTThe API rejected the session request.no
RATE_LIMITEDThrottled. Carries retry_after when the API supplied one.yes
CONCURRENCY_LIMITEvery concurrent-session slot for your plan is in use (429 concurrency_limit). See Limits.yes
TIMEOUTA step timed out.yes
CONNECTIONA network failure, a 5xx, or the avatar dropped.yes
CLIENT_UPGRADE_REQUIREDThe API says this plugin version is too old.no
AVATAR_SWAP_FAILEDThe avatar worker refused a swap.no
UNKNOWNAnything else, with the API’s message kept as it was.no

For the HTTP-level error codes, see Errors.

Troubleshooting

SymptomLikely causeFix
ValueError at startup mentioning kind="agent"Direct join with a bot token that lacks the claim, for example from the -t livekit development runner.Mint the bot token with .with_kind("agent"). See Direct join.
ValueError at startup: the API URL “must be an https:// URL to synthesia.io or a host under it”SYNTHESIA_API_URL or api_url points outside synthesia.io.Leave it unset. To go through a proxy of your own, use an https:// URL and set allow_any_api_host=True or SYNTHESIA_ALLOW_ANY_API_HOST=1.
The call ends about 4 seconds in, before the avatar says anythingAn avatar id your workspace can’t use, often a Studio avatar id.Look for UNKNOWN_AVATAR; its message lists the ids sent. See Choosing an avatar.
The bot replies to itselfDirect join without room.input_filter().Put room.input_filter() straight after transport.input().
Direct join fails with 503 service_unavailableThe LiveKit key is from a different project than your bot’s room, or the LiveKit URL isn’t publicly reachable.Take the key, secret and URL from the project your bot’s room is in.
No avatar video (relayed join)video_out_enabled isn’t set on the transport.Set all five video_out_* values.
The avatar looks stretched, a CPU core is pinned, latency growsThe transport’s video size or frame rate isn’t 512×512 at 25 fps.Set all five video_out_* values.
The bot never detects speech and can’t be interruptedVAD passed to the transport params, which ignore it.Put VAD on the user aggregator.
The call ends at almost exactly 5 minutesPipecat’s 300-second idle timeout.PipelineWorker(idle_timeout_secs=...).
UNKNOWN error “with no Synthesia error code”Something other than the Synthesia API answered: a proxy, or a redirect, which the plugin doesn’t follow.Leave SYNTHESIA_API_URL unset, or point it at an address that answers directly.
AssertionError from FfiHandle.__del__ at shutdownLiveKit’s Python SDK cleaning up.Ignore it. On Linux this can, rarely, end with exit code 134; see the deployment notes.

Choosing an avatar

avatar_ids takes Interactive Avatar ids. A Studio avatar id is not one: the two look alike, and a Studio id fails with UNKNOWN_AVATAR about four seconds into the call. To get an Interactive Avatar id, see Finding your avatar ID.

The plugin doesn’t choose a voice. Pick a TTS voice in your pipeline that fits the avatar.

Limits

  • Concurrent sessions: your plan sets how many sessions can run at once; see Operational trust. Going over it fails with 429 concurrency_limit (CONCURRENCY_LIMIT).
  • Ending a session: there’s no end-session API. The session ends, and billing stops, about 5 seconds after your bot leaves the room or, on a direct join, after the last human leaves (at once if their app calls disconnect()).
  • A human who leaves during the launch isn’t noticed. On a direct join, if the last human leaves before the avatar worker has joined, the session runs, and bills, until your bot leaves or the session reaches its length cap. End the call yourself when nobody is left, as the repository’s examples/_presence.py does.
  • Session length cap: set by your plan.
  • Pipecat ends an idle call after 300 seconds without speech. Change it with PipelineWorker(idle_timeout_secs=...).

The README’s known limitations cover rarer cases.

Deploying

  • Declare the dependency. Container and cloud builds install only what your project declares, so use uv add (or your requirements file) rather than an environment-only install.
  • Handle SIGTERM. Use WorkerRunner(handle_sigterm=True), then await runner.add_workers(worker) and await runner.run(). Without it, a container stop kills the bot before the avatar session ends. Under Pipecat’s development runner the runner owns the signals instead; see the deployment notes.
  • Network: outbound only. HTTPS to developers.synthesia.io, and WSS plus UDP (with a TCP/443 fallback) to your LiveKit project.

The repository’s deployment notes cover containers, Pipecat Cloud, egress rules and more troubleshooting.

What the plugin sends

Every request the plugin makes to the Synthesia API identifies it in its User-Agent and X-Synthesia-Client headers: the plugin’s, Pipecat’s and Python’s versions, the operating system and processor architecture, the join mode, and the kind of user transport (such as livekit or daily). Synthesia uses them to see which plugin versions are in use before changing anything those versions depend on.

An avatar token the plugin mints carries the plugin’s name and version, its protocol level, the join mode and a random id for each launch, as participant attributes that everyone in the room can see. An avatar token you provide carries only what you put in it. The full list is in the repository’s protocol notes, sections 4 and 6.

Additional resources