> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://developers.synthesia.io/docs/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://developers.synthesia.io/docs/_mcp/server.

# Interactive Avatar: Pipecat Plugin reference

Use this page to add an Interactive Avatar to a [Pipecat](https://docs.pipecat.ai) bot. For the mental model, see [Concepts](/docs/ia-concepts). The plugin's [README](https://github.com/synthesia-ai/pipecat-synthesia#readme) is the full guide, with every event, error and edge case, and its [`examples/`](https://github.com/synthesia-ai/pipecat-synthesia/tree/main/examples) folder has a runnable bot for each join mode.

`SynthesiaVideoService`, `SynthesiaError` and `ErrorType` live in `pipecat_synthesia`. The room adapters live in `pipecat_synthesia.livekit`.

## Requirements

* Python 3.11 or later.
* `pipecat-ai` 1.14 or later, below 2.0. Tested with 1.14, 1.15 and 1.16.
* A LiveKit project, for both join modes. LiveKit Cloud's free tier is fine.
* A Synthesia API key with [Interactive Avatar access](https://www.synthesia.io/features/avatars/interactive-avatars), and an Interactive Avatar id (see [Choosing an avatar](#choosing-an-avatar)).

Install the plugin from PyPI with its `[livekit]` extra, which both join modes need:

```bash
uv add "pipecat-synthesia[livekit]"
# or
pip install "pipecat-synthesia[livekit]"
```

On a Pipecat minor newer than the tested ones, the plugin logs one warning at startup. A new Pipecat minor can break the plugin before it has been tested, so for stability pin `pipecat-ai` to a tested minor (for example `pipecat-ai==1.15.*`). The plugin follows SemVer: pin it with `pipecat-synthesia[livekit]>=1.0,<2`.

## Choosing a join mode

The avatar always runs in a LiveKit room. Which room depends on how your users connect to your bot.

|                             | Direct join                                     | Relayed join                                          |
| --------------------------- | ----------------------------------------------- | ----------------------------------------------------- |
| Your users connect over     | LiveKit                                         | Daily or SmallWebRTC (other transports are untested)  |
| Room adapter                | `LiveKitTransportRoom(transport)`               | `PrivateLiveKitRoom()`                                |
| How the video reaches users | The avatar publishes it straight into your room | Through your bot, as Pipecat frames                   |
| LiveKit project             | The one your users' room is in                  | Any project of yours; it only hosts the private rooms |
| Extra wiring                | `room.input_filter()`, and a bot token you mint | Five `video_out_*` transport settings                 |

Use a direct join if your users connect over LiveKit, or if you haven't chosen a transport yet. A relayed join over Pipecat's LiveKit transport is untested and not supported.

## Environment variables

Read when the corresponding argument isn't passed explicitly.

| Variable                       | Used for                                                                                                  |
| ------------------------------ | --------------------------------------------------------------------------------------------------------- |
| `SYNTHESIA_API_KEY`            | Your Synthesia workspace API key                                                                          |
| `SYNTHESIA_API_URL`            | API base URL. Defaults to `https://developers.synthesia.io`; leave it unset                               |
| `SYNTHESIA_ALLOW_ANY_API_HOST` | Set to `1` to let `SYNTHESIA_API_URL` point outside synthesia.io, for example at your own proxy           |
| `LIVEKIT_URL`                  | Your LiveKit project URL, on a relayed join. A direct join uses the URL your `LiveKitTransport` was given |
| `LIVEKIT_API_KEY`              | LiveKit API key                                                                                           |
| `LIVEKIT_API_SECRET`           | LiveKit API secret                                                                                        |

Keep the keys and secrets in your bot's environment or a secret manager, never in frontend code. The plugin mints the avatar's LiveKit token inside your process, so your LiveKit secret never leaves it. `LIVEKIT_URL` accepts `https://` as well as `wss://`.

On a direct join, the LiveKit key and secret must come from the project your bot's room is in.

## Direct join

Place the avatar straight after your TTS (or your speech-to-speech model), and the input filter straight after `transport.input()`:

```python
from pipecat.pipeline.pipeline import Pipeline
from pipecat.transports.livekit.transport import LiveKitParams, LiveKitTransport
from pipecat_synthesia import SynthesiaVideoService
from pipecat_synthesia.livekit import LiveKitTransportRoom

transport = LiveKitTransport(
    url, bot_token, room_name, LiveKitParams(audio_in_enabled=True, audio_out_enabled=True)
)
room = LiveKitTransportRoom(transport)
avatar = SynthesiaVideoService(avatar_ids=["<avatar-id>"], room=room)

pipeline = Pipeline(
    [
        transport.input(),
        room.input_filter(),  # keeps the avatar's voice off the bot's input
        stt,
        context_aggregator.user(),
        llm,
        tts,
        avatar,               # straight after the TTS
        transport.output(),
        context_aggregator.assistant(),
    ]
)
```

A direct join needs three things:

1. **`room.input_filter()` straight after `transport.input()`.** The avatar is a participant in your room, so its voice reaches your bot's input. Without the filter, the bot transcribes the avatar and replies to itself.
2. **A bot token with the `kind="agent"` claim.** Without it, the avatar worker counts your bot as a human, so the session doesn't end when the real user leaves, and keeps billing. `LiveKitTransportRoom(transport)` raises `ValueError` at startup for a token without it.
3. **LiveKit credentials from the project your bot's room is in.**

`kind` is a top-level claim, not the `agent` grant:

```python
from livekit.api import AccessToken, VideoGrants

bot_token = (
    AccessToken(api_key, api_secret)
    .with_identity("pipecat-bot")
    .with_kind("agent")
    .with_grants(VideoGrants(room_join=True, room=room_name, agent=True))
    .to_jwt()
)
```

> **Warning**
>
> Pipecat's development runner (`python bot.py -t livekit`) mints a bot token without this claim, so a direct join can't run under it. Mint the token yourself and start the bot directly, as the repository's `examples/bot_direct_join.py` does. LiveKit Cloud's agent-dispatch tokens already carry the claim.

The avatar is a room participant too, and usually the first to arrive, so `on_first_participant_joined` fires for it. Greet the first participant that isn't `avatar.avatar_identity` instead. Don't end the call the moment the last human leaves: a dropped connection that reconnects leaves and rejoins moments apart, and ending in between takes the avatar away. The avatar worker waits about 5 seconds before ending the session itself, so end your call when `on_session_ended` reports `no_humans_left`, or wait at least that long. Waiting can't help when the human's app leaves on purpose (for example by calling `disconnect()`): the avatar worker then ends the session at once. The repository's `examples/bot_direct_join.py` shows both.

## Relayed join

```python
from pipecat.pipeline.pipeline import Pipeline
from pipecat_synthesia import SynthesiaVideoService
from pipecat_synthesia.livekit import PrivateLiveKitRoom

avatar = SynthesiaVideoService(
    avatar_ids=["<avatar-id>"],
    room=PrivateLiveKitRoom(),  # reads LIVEKIT_URL, LIVEKIT_API_KEY and LIVEKIT_API_SECRET
)

pipeline = Pipeline(
    [transport.input(), stt, context_aggregator.user(), llm, tts, avatar, transport.output(), context_aggregator.assistant()]
)
```

The avatar renders at 512×512 and 25 fps. Set all five of these on your transport's params (`TransportParams`, or `DailyParams` for Daily):

```python
video_out_enabled=True,
video_out_is_live=True,
video_out_width=512,
video_out_height=512,
video_out_framerate=25,
```

Without `video_out_enabled=True` your users see no avatar. With another size or frame rate, Pipecat resizes every frame: the avatar looks stretched, a CPU core is pinned, and latency grows over the call. The plugin logs a warning at startup if the settings don't match.

> **Note**
>
> **Speech-to-speech models** (for example OpenAI Realtime) go in the same slot: replace `stt`, `llm` and `tts` with the model and put `avatar` straight after it. **Voice activity detection** belongs on the user aggregator, `LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer())`. The transport params silently ignore a `vad_analyzer`, and the bot then never detects speech.

## `SynthesiaVideoService`

```python
SynthesiaVideoService(
    *,
    avatar_ids,
    room,
    api_key=None,
    api_url=None,
    allow_any_api_host=None,
    launch_timeout=90.0,
    required=True,
    session=None,
    settings=None,
)
```

| Parameter            | Type                             | Default      | Description                                                                                                                                                                                                                                     |
| -------------------- | -------------------------------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `avatar_ids`         | `Sequence[str]`                  | **required** | One to five Interactive Avatar ids. The first is shown; the others can be switched to with `swap_avatar()`. The `av_` prefix is added for you.                                                                                                  |
| `room`               | room adapter                     | **required** | `LiveKitTransportRoom(transport)` for a direct join, or `PrivateLiveKitRoom()` for a relayed join.                                                                                                                                              |
| `api_key`            | `str \| None`                    | `None`       | Falls back to `SYNTHESIA_API_KEY`.                                                                                                                                                                                                              |
| `api_url`            | `str \| None`                    | `None`       | Falls back to `SYNTHESIA_API_URL`, then `https://developers.synthesia.io`. Must be `https://` to synthesia.io or a host under it, unless `allow_any_api_host` is set.                                                                           |
| `allow_any_api_host` | `bool \| None`                   | `None`       | Falls back to `SYNTHESIA_ALLOW_ANY_API_HOST`, then `False`. Lets `api_url` be an `https://` host outside synthesia.io, such as your own proxy. Your API key and a LiveKit token for the session go to that host, and the plugin logs a warning. |
| `launch_timeout`     | `float`                          | `90.0`       | Seconds for the whole launch: connecting, creating the session (with retries) and the avatar joining.                                                                                                                                           |
| `required`           | `bool`                           | `True`       | Whether the avatar is required to start. `True`: if it can't start, the service ends the call, and none of the bot's speech plays from then on. `False`: the bot carries on voice-only.                                                         |
| `session`            | `aiohttp.ClientSession \| None`  | `None`       | Bring your own HTTP session.                                                                                                                                                                                                                    |
| `settings`           | `SynthesiaVideoSettings \| None` | `None`       | Initial settings.                                                                                                                                                                                                                               |

Invalid arguments, such as a malformed avatar id, a missing API key, an API URL outside synthesia.io, or an unknown keyword like the LiveKit plugin's `join_timeout`, raise `ValueError` or `TypeError` at once.

If the avatar leaves mid-call (the worker failed, the session reached its length cap, or credits ran out), the bot carries on voice-only whatever `required` says, and you get `on_avatar_disconnected`. To end the call instead, push an `EndWorkerFrame` upstream from that handler.

### `await avatar.swap_avatar(avatar_id, *, timeout=None)`

The session always starts on the first id in `avatar_ids`, so list the one you want first. `swap_avatar()` switches to another id from `avatar_ids`, or back to the first with `"default"`. `timeout` defaults to 15 seconds. It needs an active session: called before `on_avatar_connected`, it raises. A failed swap raises, reports through `on_error`, and keeps the previous avatar. `swap_avatar()` hasn't yet been tested against the live Synthesia API.

## Room adapters

| Adapter                | Signature                                                                                                                                                                                                                                                                | Notes                                                                                                                                                                                           |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `LiveKitTransportRoom` | `(transport, *, avatar_token=None, credentials=None, livekit_url=None, avatar_identity=None, avatar_name="Synthesia avatar", connect_timeout=30.0)`                                                                                                                      | Direct join. Mints the avatar's token from `credentials` (a `LiveKitCredentials`), else `LIVEKIT_API_KEY` and `LIVEKIT_API_SECRET`. `room.input_filter()` returns the filter for your pipeline. |
| `PrivateLiveKitRoom`   | `(*, url=None, api_key=None, api_secret=None, room_name=None, bot_identity="pipecat-bot", avatar_identity="synthesia-avatar-agent", avatar_name="Synthesia avatar", empty_timeout_secs=120, departure_timeout_secs=20, delete_room_on_close=True, connect_timeout=15.0)` | Relayed join. Reads `LIVEKIT_URL`, `LIVEKIT_API_KEY` and `LIVEKIT_API_SECRET` when not passed. Creates a private room for each session and deletes it at the end.                               |

## Events

```python
@avatar.event_handler("on_avatar_connected")
async def on_avatar_connected(service, info):
    logger.info(f"avatar is live: session {info.session_id}")

@avatar.event_handler("on_metrics")
async def on_metrics(service, metrics):
    logger.info(f"avatar metrics: {metrics.summary()}")
```

| Event                    | Fires                                                                                                                         | Handler arguments                                                                       |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| `on_avatar_connected`    | At most once, when the Synthesia API has created the session. The avatar's audio can start up to about half a second earlier. | `service, info` (`session_id`, `avatar_identity`, `active_avatar_id`, `room_name`)      |
| `on_avatar_ready`        | Once, when every avatar in `avatar_ids` is ready                                                                              | `service, info` (`session_id`, `status`)                                                |
| `on_avatar_disconnected` | At most once, when the avatar leaves an active session                                                                        | `service, reason`: `avatar_left_room`, `avatar_video_stopped` or `room_connection_lost` |
| `on_session_ended`       | Exactly once                                                                                                                  | `service, reason`: an end reason, below                                                 |
| `on_metrics`             | Once, right after `on_session_ended`                                                                                          | `service, metrics`: call `metrics.summary()`                                            |
| `on_error`               | Per error. Pipecat's own event                                                                                                | `service, frame`: an `ErrorFrame` whose `.exception` is a `SynthesiaError`              |

| End reason           | Meaning                                                                                                                                                                                                                   |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pipeline_ended`     | Your pipeline ended normally.                                                                                                                                                                                             |
| `pipeline_cancelled` | Your pipeline was cancelled, including a stop by signal.                                                                                                                                                                  |
| `pipeline_cleanup`   | Pipecat cleaned up the service without ending or cancelling it first.                                                                                                                                                     |
| `no_humans_left`     | Direct join: the last human left, so the avatar worker ended the session. A normal end, with no error.                                                                                                                    |
| `avatar_left`        | The avatar left unexpectedly: on a direct join, while humans were still in the room 2 seconds later (at once if the bot lost its own connection to the room); on a relayed join, any leave. The bot continues voice-only. |
| `launch_failed`      | The avatar couldn't start, with `required=True`. The service ends the call.                                                                                                                                               |
| `launch_timeout`     | The launch took longer than `launch_timeout`, or a step in it timed out (an API request, or the avatar not joining within 30 seconds of the session being created), with `required=True`. The service ends the call.      |
| `degraded`           | The avatar couldn't start, with `required=False`. The bot continues voice-only.                                                                                                                                           |

The plugin also logs `metrics.summary()` once per session, in a line containing `synthesia session summary`. Once the Synthesia API has created the session, it carries the session and request ids; quote it when contacting support.

## Errors

Once the service is running, every failure is a `SynthesiaError` with a `type` from `ErrorType`. In an `on_error` handler, read it from `frame.exception`, not from the `ErrorFrame`.

| `ErrorType`                    | Meaning                                                                                                 | Retryable |
| ------------------------------ | ------------------------------------------------------------------------------------------------------- | --------- |
| `AUTH`                         | The API key is invalid, or isn't allowed to use Interactive Avatars.                                    | no        |
| `FEATURE_NOT_IN_PLAN`          | Your workspace's plan doesn't include Interactive Avatars.                                              | no        |
| `QUOTA_EXCEEDED`               | Your workspace is out of quota or credits.                                                              | no        |
| `UNKNOWN_AVATAR`               | An avatar id isn't available to your workspace, or is a Studio avatar id.                               | no        |
| `INVALID_ROOM_TOKEN`           | A LiveKit token expired before the launch, or doesn't name the bot's room.                              | no        |
| `LIVEKIT_CREDENTIALS_REJECTED` | Relayed join: your LiveKit project refused to create the private room.                                  | no        |
| `INVALID_SESSION_REQUEST`      | The API rejected the session request.                                                                   | no        |
| `RATE_LIMITED`                 | Throttled. Carries `retry_after` when the API supplied one.                                             | yes       |
| `CONCURRENCY_LIMIT`            | Every concurrent-session slot for your plan is in use (`429 concurrency_limit`). See [Limits](#limits). | yes       |
| `TIMEOUT`                      | A step timed out.                                                                                       | yes       |
| `CONNECTION`                   | A network failure, a 5xx, or the avatar dropped.                                                        | yes       |
| `CLIENT_UPGRADE_REQUIRED`      | The API says this plugin version is too old.                                                            | no        |
| `AVATAR_SWAP_FAILED`           | The avatar worker refused a swap.                                                                       | no        |
| `UNKNOWN`                      | Anything else, with the API's message kept as it was.                                                   | no        |

For the HTTP-level error codes, see [Errors](/docs/errors).

## Troubleshooting

| Symptom                                                                                            | Likely cause                                                                                                   | Fix                                                                                                                                             |
| -------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `ValueError` at startup mentioning `kind="agent"`                                                  | Direct join with a bot token that lacks the claim, for example from the `-t livekit` development runner.       | Mint the bot token with `.with_kind("agent")`. See [Direct join](#direct-join).                                                                 |
| `ValueError` at startup: the API URL "must be an https\:// URL to synthesia.io or a host under it" | `SYNTHESIA_API_URL` or `api_url` points outside synthesia.io.                                                  | Leave it unset. To go through a proxy of your own, use an `https://` URL and set `allow_any_api_host=True` or `SYNTHESIA_ALLOW_ANY_API_HOST=1`. |
| The call ends about 4 seconds in, before the avatar says anything                                  | An avatar id your workspace can't use, often a Studio avatar id.                                               | Look for `UNKNOWN_AVATAR`; its message lists the ids sent. See [Choosing an avatar](#choosing-an-avatar).                                       |
| The bot replies to itself                                                                          | Direct join without `room.input_filter()`.                                                                     | Put `room.input_filter()` straight after `transport.input()`.                                                                                   |
| Direct join fails with `503 service_unavailable`                                                   | The LiveKit key is from a different project than your bot's room, or the LiveKit URL isn't publicly reachable. | Take the key, secret and URL from the project your bot's room is in.                                                                            |
| No avatar video (relayed join)                                                                     | `video_out_enabled` isn't set on the transport.                                                                | Set all five `video_out_*` values.                                                                                                              |
| The avatar looks stretched, a CPU core is pinned, latency grows                                    | The transport's video size or frame rate isn't 512×512 at 25 fps.                                              | Set all five `video_out_*` values.                                                                                                              |
| The bot never detects speech and can't be interrupted                                              | VAD passed to the transport params, which ignore it.                                                           | Put VAD on the user aggregator.                                                                                                                 |
| The call ends at almost exactly 5 minutes                                                          | Pipecat's 300-second idle timeout.                                                                             | `PipelineWorker(idle_timeout_secs=...)`.                                                                                                        |
| `UNKNOWN` error "with no Synthesia error code"                                                     | Something other than the Synthesia API answered: a proxy, or a redirect, which the plugin doesn't follow.      | Leave `SYNTHESIA_API_URL` unset, or point it at an address that answers directly.                                                               |
| `AssertionError` from `FfiHandle.__del__` at shutdown                                              | LiveKit's Python SDK cleaning up.                                                                              | Ignore it. On Linux this can, rarely, end with exit code 134; see the deployment notes.                                                         |

## Choosing an avatar

`avatar_ids` takes Interactive Avatar ids. A Studio avatar id is not one: the two look alike, and a Studio id fails with `UNKNOWN_AVATAR` about four seconds into the call. To get an Interactive Avatar id, see [Finding your avatar ID](/docs/ia-concepts#finding-your-avatar-id).

The plugin doesn't choose a voice. Pick a TTS voice in your pipeline that fits the avatar.

## Limits

* **Concurrent sessions:** your plan sets how many sessions can run at once; see [Operational trust](/docs/ia-operational-trust#limits). Going over it fails with `429 concurrency_limit` (`CONCURRENCY_LIMIT`).
* **Ending a session:** there's no end-session API. The session ends, and billing stops, about 5 seconds after your bot leaves the room or, on a direct join, after the last human leaves (at once if their app calls `disconnect()`).
* **A human who leaves during the launch isn't noticed.** On a direct join, if the last human leaves before the avatar worker has joined, the session runs, and bills, until your bot leaves or the session reaches its length cap. End the call yourself when nobody is left, as the repository's `examples/_presence.py` does.
* **Session length cap:** set by your plan.
* **Pipecat ends an idle call after 300 seconds** without speech. Change it with `PipelineWorker(idle_timeout_secs=...)`.

The README's [known limitations](https://github.com/synthesia-ai/pipecat-synthesia#known-limitations) cover rarer cases.

## Deploying

* **Declare the dependency.** Container and cloud builds install only what your project declares, so use `uv add` (or your requirements file) rather than an environment-only install.
* **Handle SIGTERM.** Use `WorkerRunner(handle_sigterm=True)`, then `await runner.add_workers(worker)` and `await runner.run()`. Without it, a container stop kills the bot before the avatar session ends. Under Pipecat's development runner the runner owns the signals instead; see the deployment notes.
* **Network:** outbound only. HTTPS to `developers.synthesia.io`, and WSS plus UDP (with a TCP/443 fallback) to your LiveKit project.

The repository's [deployment notes](https://github.com/synthesia-ai/pipecat-synthesia/blob/main/docs/deployment.md) cover containers, Pipecat Cloud, egress rules and more troubleshooting.

## What the plugin sends

Every request the plugin makes to the Synthesia API identifies it in its `User-Agent` and `X-Synthesia-Client` headers: the plugin's, Pipecat's and Python's versions, the operating system and processor architecture, the join mode, and the kind of user transport (such as `livekit` or `daily`). Synthesia uses them to see which plugin versions are in use before changing anything those versions depend on.

An avatar token the plugin mints carries the plugin's name and version, its protocol level, the join mode and a random id for each launch, as participant attributes that everyone in the room can see. An avatar token you provide carries only what you put in it. The full list is in the repository's [protocol notes](https://github.com/synthesia-ai/pipecat-synthesia/blob/main/docs/protocol-compatibility.md), sections 4 and 6.

## Additional resources

* [`pipecat-synthesia` on GitHub](https://github.com/synthesia-ai/pipecat-synthesia): the README, examples and changelog. MIT licensed.
* [`pipecat-synthesia` on PyPI](https://pypi.org/project/pipecat-synthesia/)
* [Pipecat documentation](https://docs.pipecat.ai)
* **Support:** GitHub issues are turned off. Email [support@synthesia.io](mailto:support@synthesia.io) and say it's about `pipecat-synthesia`; the repository's [SUPPORT.md](https://github.com/synthesia-ai/pipecat-synthesia/blob/main/SUPPORT.md) lists what to include. To report a security issue, see [SECURITY.md](https://github.com/synthesia-ai/pipecat-synthesia/blob/main/SECURITY.md).