Speaking Bot

This repository expands upon Pipecat's Python framework for building voice and multimodal conversational agents. Our implementation creates AI meeting agents that can join and participate in Google Meet and Microsoft Teams meetings with distinct personalities and capabilities defined in Markdown files.

Overview

This project extends Pipecat's WebSocket server implementation to create:

• Meeting agents that can join Google Meet or Microsoft Teams through the MeetingBaas API
• Customizable personas with unique context
• Support for running multiple instances via a simple API
• WebSocket-based communication for real-time interaction

Architecture

Core Framework: Pipecat Integration

Pipecat provides the foundational framework with:

• Real-time audio processing pipeline
• WebSocket communication
• Voice activity detection
• Message context management

In this implementation, Pipecat is integrated with Cartesia for speech generation (text-to-speech), Gladia or Deepgram for speech-to-text conversion, and OpenAI's GPT-4 as the underlying LLM.

API-First Architecture

The project follows a streamlined API-first approach with:

• A lightweight FastAPI server that handles bot management via direct MeetingBaas API calls
• WebSocket server for real-time communication between MeetingBaas and Pipecat
• Properly typed Pydantic models for request/response validation
• Clean separation of concerns with modular components

API Endpoints

1. Root endpoint (GET /):

   • Health check endpoint
   • Returns: {"message": "MeetingBaas Bot API is running"}

2. Run Bots (POST /run-bots):

   {
     "meeting_url": "https://meet.google.com/xxx-yyyy-zzz",
     "personas": ["interviewer"],
     "meeting_baas_api_key": "your-api-key",
     "bot_image": "https://example.com/avatar.jpg",
     "entry_message": "Hello, I'm here to help!"
   }

   • Required field: meeting_url
   • Authentication: send x-meeting-baas-api-key as a request header
   • Optional override: websocket_url
   • Returns: MeetingBaas bot_id

3. WebSocket endpoint (/ws/{client_id}):

   • Real-time communication channel for audio streaming
   • Binary audio data and control messages

4. Pipecat WebSocket endpoint (/pipecat/{client_id}):

   • Connection point for Pipecat services
   • Bidirectional conversion between raw audio and Protobuf frames

WebSocket URL Resolution

The server determines the WebSocket URL to use in the following priority order:

1. User-provided URL in the request (if specified in the websocket_url field)
2. BASE_URL environment variable (recommended for production)
3. ngrok URL in local development mode
4. Auto-detection from request headers (fallback, not reliable in production)

For production deployments, it's strongly recommended to set the BASE_URL environment variable to your server's public domain (e.g., https://your-server-domain.com).

Project Extensions

Building upon Pipecat, we've added:

• Persona system with Markdown-based configuration for:
  • Core personality traits and behaviors
  • Knowledge base and domain expertise
  • Additional contextual information (websites formatted to MD, technical documentation, etc.)
• AI image generation via Replicate
• Image hosting through UploadThing (UTFS)
• MeetingBaas integration for video meeting platform support
• Multi-agent orchestration via API

Required API Keys

For Pipecat-related Services

• OpenAI (LLM)
• Cartesia (text-to-speech)
• Gladia or Deepgram (speech-to-text)
• MeetingBaas (video meeting platform integration)

For Project-specific Add-ons

• OpenAI (LLM to complete the user prompt and match to a Cartesia Voice ID)
• Replicate (AI image generation)
• UploadThing (UTFS) (image hosting)

For speech-related services (TTS/STT) and LLM choice (like Claude, GPT-4, etc), you can freely choose and swap between any of the integrations available in Pipecat's supported services.

Important Note

OpenAI's GPT-4, UploadThing (UTFS), and Replicate are currently hard-coded specifically for the CLI-based persona generation features: matching personas to available voices from Cartesia, generating AI avatars, and creating initial personality descriptions and knowledge bases. You do not need a Replicat or UTFS API key to run the project if you're not using the CLI-based persona creation feature and edit Markdowns manually.

Persona System

Bot Service

• Real-time audio processing pipeline

• WebSocket-based communication

• Tool integration (weather, time)

• Voice activity detection

• Message context management

• Dynamic persona loading from markdown files

• Customizable personality traits and behaviors

• Support for multiple languages

• Voice characteristic customization

• Image generation for persona avatars

• Metadata management for each persona

Persona Structure

Each persona is defined in the @personas directory with:

• A README.md defining their personality
• Space for additional markdown files to expand knowledge and behaviour

Example Persona Structure

@personas/
└── quantum_physicist/
    ├── README.md
    └── (additional beVhavior files)

Prerequisites

• Python 3.x
• grpc_tools for protocol buffer compilation
• Ngrok (for local deployment)
• Poetry for dependency management

Installation

1. Set Up Poetry Environment

# Install Poetry (Unix/macOS)
curl -sSL https://install.python-poetry.org | python3 -

# Install Poetry (Windows)
(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py -

2. Install System Dependencies

The project requires certain system dependencies for scientific libraries:

# macOS (using Homebrew)
brew install llvm cython

# Ubuntu/Debian
sudo apt-get install llvm python3-dev cython

# Fedora/RHEL
sudo dnf install llvm-devel python3-devel Cython

3. Set up Project with Poetry

# Clone the repository (if you haven't already)
git clone https://github.com/yourusername/speaking-meeting-bot.git
cd speaking-meeting-bot

# Configure Poetry to use Python 3.11+
poetry env use python3.11

# Install dependencies with LLVM config path
# On macOS:
LLVM_CONFIG=$(brew --prefix llvm)/bin/llvm-config poetry install

# On Linux (path may vary):
# LLVM_CONFIG=/usr/bin/llvm-config poetry install

# Activate virtual environment
poetry env activate

4. Compile Protocol Buffers

poetry run python -m grpc_tools.protoc --proto_path=./protobufs --python_out=./protobufs frames.proto

5. Configure Environment

cp env.example .env

Edit .env with your MeetingBaas credentials and add the runtime settings needed for your environment.

Example .env file:

MEETING_BAAS_API_KEY=your_api_key_here
BASE_URL=https://your-server-domain.com  # For production
PORT=7014
CORS_ALLOW_ORIGINS=https://your-frontend.example.com

Running Meeting Agents

API Server Setup

There are two ways to run the server:

# Standard mode
poetry run api --host 0.0.0.0 --port ${PORT}

# Local development mode with ngrok auto-configuration
poetry run python app/main.py --local-dev

The local development mode simplifies WebSocket setup by:

• Automatically detecting ngrok tunnels
• Handling WebSocket URL configuration for MeetingBaas
• Supporting up to 2 bots (limited by free ngrok tunnels)
• Providing clear warnings about limitations

Setting Up ngrok for Local Development

1. Install ngrok if you haven't already:

   brew install ngrok  # macOS

2. Sign up for an ngrok account at https://dashboard.ngrok.com/signup and get your authtoken.

3. Configure your ngrok authtoken:

   ngrok config add-authtoken YOUR_AUTHTOKEN_HERE

4. Start ngrok tunnels for your bot connections:

   # Start ngrok with the provided configuration
   ngrok start --all --config config/ngrok/config.yml

   This will create two tunnels (ports 7014 and 7015) for running multiple bots.

5. Copy the ngrok HTTPS URL (e.g., https://xxxx.ngrok-free.app) and set it as BASE_URL in your .env file:

   BASE_URL=https://xxxx.ngrok-free.app

6. Start the server:

   poetry run uvicorn app:app --reload --host 0.0.0.0 --port 7014

Creating Bots via API

The WebSocket URL is optional in all cases. The server determines the appropriate URL based on the priority list described in the WebSocket URL Resolution section:

curl -X POST http://localhost:${PORT}/bots \
  -H "Content-Type: application/json" \
  -H "x-meeting-baas-api-key: your-api-key" \
  -d '{
    "meeting_url": "https://meet.google.com/xxx-yyyy-zzz",
    "personas": ["interviewer"]
  }'

You can attach external prompt context and MCP servers per bot. External context is loaded before the Pipecat process starts and capped by prompt_data_token_limit using an approximate token budget. URL sources block localhost and private-network targets by default; set PROMPT_DATA_ALLOW_PRIVATE_URLS=true only in trusted deployments.

You can also select the bot LLM per request with llm_provider and llm_model. Supported providers are openai, anthropic, and zai. Provider credentials and base URLs are server-side environment variables only: OPENAI_API_KEY, ANTHROPIC_API_KEY, ZAI_API_KEY, and optional ZAI_BASE_URL. OpenAI defaults to Pipecat's Responses API service for newest models; set OPENAI_API_SURFACE=chat to use the older Chat Completions bridge. The Z.ai integration uses the OpenAI-compatible Chat Completions bridge; Anthropic uses Pipecat's native Claude bridge. The API rejects unconfigured providers before creating the upstream MeetingBaaS bot.

MCP servers are live-query capable only when their config is connectable. http, streamable_http, and sse servers require transport, url, and optional headers. For trusted local mcpproxy groups, prefer mcp_profile instead of hand-writing the loopback server config:

• mcp_profile: "professional" connects to MCP_PROXY_PROFESSIONAL_URL or http://127.0.0.1:8111/mcp.
• mcp_profile: "personal" connects to MCP_PROXY_PERSONAL_URL or http://127.0.0.1:8110/mcp.
• mcp_profile: "all" connects to MCP_PROXY_ALL_URL or http://127.0.0.1:8109/mcp.
• mcp_profile_tool_access: "read_only" exposes only retrieve_tools, call_tool_read, read_cache, and set_profile.
• mcp_profile_tool_access: "read_write" additionally exposes call_tool_write; presets never expose upstream_servers, call_tool_destructive, code_execution, registry, or quarantine tools.

You can combine mcp_profile with explicit mcp.servers; the preset is prepended, and request MCP instructions are appended after the safety instructions. Local process stdio MCP is intentionally not accepted by the public API because it would execute caller-supplied commands. Remote MCP URLs also block localhost and private-network targets by default; production deployments should use MCP_ALLOWED_PRIVATE_URLS with exact http://host:port/path entries for trusted loopback MCPs instead of the broad MCP_ALLOW_PRIVATE_URLS=true development bypass. If transport is omitted, the server is treated as metadata-only and MCP tools are not executed. Secrets are not required in the request, but headers are available for deployments that need them. Use tool_allowlist to constrain which server tools the bot may call, and set enabled: false to document a server without connecting to it.

curl -X POST http://localhost:${PORT}/bots \
  -H "Content-Type: application/json" \
  -H "x-meeting-baas-api-key: your-api-key" \
  -d '{
    "meeting_url": "https://meet.google.com/xxx-yyyy-zzz",
    "personas": ["account_executive"],
    "llm_provider": "anthropic",
    "llm_model": "claude-opus-4-8",
    "mcp_profile": "professional",
    "mcp_profile_tool_access": "read_only",
    "prompt_data_token_limit": 4000,
    "prompt_data_sources": [
      {
        "name": "CRM account notes",
        "type": "url",
        "url": "https://example.com/account-notes.md"
      },
      {
        "name": "Call objective",
        "type": "text",
        "text": "Confirm timeline, budget, and integration constraints."
      }
    ],
    "mcp": {
      "instructions": "Use CRM context only when relevant. Google Drive is available through the professional mcpproxy profile.",
      "servers": [
        {
          "name": "remote-crm",
          "enabled": true,
          "transport": "streamable_http",
          "url": "https://mcp.example.com/mcp",
          "headers": {
            "Authorization": "Bearer optional-token"
          },
          "tools": ["get_account", "list_recent_calls"],
          "tool_allowlist": ["get_account", "list_recent_calls"],
          "timeout_seconds": 15
        }
      ]
    },
    "speech_speed": 1.25
  }'

speech_speed overrides CARTESIA_TTS_SPEED, TTS_SPEED, or SPEECH_SPEED. The Cartesia runner clamps speed to 0.6..1.5.

LLM defaults are resolved as request value, then provider-specific env, then generic LLM_MODEL, then service default:

• OpenAI: OPENAI_MODEL, default gpt-5.5. OPENAI_API_SURFACE defaults to responses; set it to chat for compatibility with older OpenAI-compatible paths. OPENAI_SERVICE_TIER is passed through when set.
• Anthropic: ANTHROPIC_MODEL, default claude-opus-4-8. Low-latency example: claude-haiku-4-5.
• Z.ai: ZAI_MODEL, default glm-5.2; ZAI_BASE_URL defaults to https://api.z.ai/api/paas/v4/.

OpenAPI Snapshots

This repo contains four OpenAPI files with different roles:

• openapi.json is this FastAPI service snapshot for generic tooling.
• speaking-bot-openapi.json is the same service snapshot, named explicitly for the speaking-bots MCP sync.
• meeting-baas-openapi-v1.json is the upstream MeetingBaaS v1 API snapshot.
• openapi-v2.json is the upstream MeetingBaaS v2 API snapshot.

The service snapshots include /bots, /bots/{bot_id}, /personas/generate-image, /health, /ready, /webhook, and the current BotRequest fields for prompt_data_sources, prompt_data_token_limit, mcp, mcp_profile, mcp_profile_tool_access, and speech_speed.

Regenerate the service snapshot after API model changes:

poetry run python scripts/export_openapi.py

You can still manually specify a WebSocket URL if needed:

curl -X POST http://localhost:${PORT}/bots \
  -H "Content-Type: application/json" \
  -H "x-meeting-baas-api-key: your-api-key" \
  -d '{
    "meeting_url": "https://meet.google.com/xxx-yyyy-zzz",
    "personas": ["interviewer"],
    "websocket_url": "wss://your-custom-websocket-url.example.com"
  }'

Production Deployment Considerations

When deploying to production, always set the BASE_URL environment variable to ensure reliable WebSocket connections:

1. Set BASE_URL to your server's public domain:

   export BASE_URL=https://your-server-domain.com
   

2. Ensure your server is accessible on the public internet

3. Consider using HTTPS/WSS for secure connections in production

Troubleshooting Local Development

If you encounter issues with the local development mode:

1. Make sure ngrok is running with the correct configuration
2. Verify that you've entered the correct ngrok URLs when prompted
3. Check that your ngrok URLs are accessible (try opening in a browser)
4. Remember that the free tier of ngrok limits you to 2 concurrent tunnels

Future Extensibility

The persona architecture is designed to support:

• Scrapping the websites given by the user to MD for the bot knowledge base
• Containerizing this nicely

Troubleshooting

• Verify Poetry environment is activated
• Check Ngrok connection status
• Validate environment variables
• Ensure unique Ngrok URLs for multiple agents

For more detailed information about specific personas or deployment options, check the respective documentation in the @personas directory.

Troubleshooting WebSocket Connections

Handling Timing Issues with ngrok and Meeting Baas Bots

Sometimes, due to WebSocket connection delays through ngrok, the Meeting Baas bots may join the meeting before your local bot connects. If this happens:

• Simply press Enter to respawn your bot
• This will reinitiate the connection and allow your bot to join the meeting

This is a normal occurrence and can be easily resolved with a quick bot respawn.

Running the API Server

Local Development

# Install dependencies
poetry install

# Compile Protocol Buffers
poetry run python -m grpc_tools.protoc --proto_path=./protobufs --python_out=./protobufs frames.proto

# Run the API server with hot reload
poetry run uvicorn app:app --reload --host 0.0.0.0 --port ${PORT}

Local Testing with Multiple Bots

For local development and testing with multiple bots, you'll need two terminals:

# Terminal 1: Start the API server
poetry run uvicorn app:app --reload --host 0.0.0.0 --port ${PORT}

# Terminal 2: Start ngrok to expose your local server
ngrok http ${PORT}

Once ngrok is running, it will provide you with a public URL that the server will use for WebSocket connections in local development mode.

API Improvements

The API has been completely redesigned for simplicity and reliability:

• Direct integration with the MeetingBaas API without subprocess management
• Strongly typed Pydantic models with proper validation
• Cleaner WebSocket handling with better error management
• Improved logging with better visibility into the system
• Enhanced JSON message processing for debugging
• Intelligent WebSocket URL resolution with multiple fallback methods
• Support for explicit BASE_URL configuration for production environments

The direct API integration provides several benefits:

# Direct API call to MeetingBaas
meetingbaas_bot_id = create_meeting_bot(
    meeting_url=request.meeting_url,
    websocket_url=websocket_url,  # Determined by the server via multiple methods
    bot_id=bot_client_id,
    persona_name=persona_name,
    api_key=request.meeting_baas_api_key,
    # Additional parameters
    bot_image=request.bot_image,
    entry_message=request.entry_message,
    extra=request.extra,
)

This approach eliminates the complexity of subprocess management, provides immediate feedback on bot creation, and returns both the MeetingBaas bot ID and client ID for WebSocket connections.

Production Deployment

For production deployment, always set the BASE_URL environment variable:

# Set the BASE_URL for WebSocket connections
export BASE_URL=https://your-server-domain.com

# Run the API server in production mode
poetry run api --host 0.0.0.0 --port ${PORT}

API Documentation

Once the server is running, you can access:

• Interactive API docs: http://localhost:${PORT}/docs
• OpenAPI specification: http://localhost:${PORT}/openapi.json
• Committed service OpenAPI snapshots: openapi.json, speaking-bot-openapi.json
• Health endpoint: http://localhost:${PORT}/health
• Readiness endpoint: http://localhost:${PORT}/ready

Future Development

The API-first approach enables several planned features:

1. Parent API Integration:

   • Authentication and authorization
   • Rate limiting
   • User management
   • Billing integration

2. Enhanced Bot Management:

   • Real-time bot status monitoring
   • Dynamic persona loading
   • Bot lifecycle management
   • Meeting recording and transcription

3. WebSocket Features:

   • Real-time bot control
   • Live transcription streaming
   • Meeting analytics
   • Multi-bot coordination

4. Persona Management:

   • Dynamic persona creation via API
   • Persona validation and testing
   • Knowledge base expansion
   • Voice characteristic customization