elevenlabs-agent-simulator

ElevenLabs Agent Simulator - Chrome Extension Implementation Plan

Project Overview

ElevenLabs Agent Simulator is a Chrome extension that brings SDK-level conversation testing capabilities directly into the ElevenLabs web interface. It enables developers to simulate realistic dialogues between their conversational AI agent and an automated user agent, allowing them to test prompts, refine settings, and identify edge cases before production deployment.

Architecture Overview

Extension Structure

manifest.json - Chrome extension manifest (V3)
src/popup/ - Extension popup UI (HTML, TS, CSS)
src/utils/ - URL parser utility
public/ - Icons and assets
Configuration files - package.json, tsconfig.json, webpack.config.js

Note: No background service worker or content scripts needed - the popup handles everything directly.

Implementation Phases

Phase 1: Project Setup & Foundation

1.1 Initialize Project Structure

Set up TypeScript configuration
Configure Webpack for Chrome extension bundling
Set up ESLint and Prettier
Create basic folder structure
Initialize package.json with required dependencies

1.2 Dependencies

Runtime: None (will use direct REST API calls to ElevenLabs)
Dev: TypeScript, Webpack + loaders, ESLint, Prettier, Chrome types

Note: The ElevenLabs Node.js SDK (@elevenlabs/elevenlabs-js) cannot be used in browser environments. The browser SDK (@elevenlabs/client) is designed for real-time voice conversations via WebSocket/WebRTC, not for the simulateConversation API. We’ll make direct HTTPS REST API calls to https://api.elevenlabs.io/v1/convai/agents/{agent_id}/simulate-conversation using the Fetch API.

1.3 Create manifest.json

Manifest V3 with permissions: activeTab
Host permissions: elevenlabs.io/*, api.elevenlabs.io/*
Popup action with icons (16px, 48px, 128px)
No background service worker or content scripts needed

2.1 URL Parser Utility (`src/utils/url-parser.ts`)

Create function to extract Agent ID from URL pattern: elevenlabs.io/app/conversational-ai/[agent-id]/*
Handle invalid URLs gracefully
Return null if pattern doesn’t match

2.2 Popup Interface (`src/popup/popup.html`)

API Key Input - Simple text input field (no validation, no storage)
Agent ID Input - Text input field, auto-populated from current tab URL
Simulated User Prompt - Multi-line textarea for user prompt (no templates)
Start Button - Trigger simulation
Output Section - Scrollable text area to display JSON response

2.3 Popup Styling (`src/popup/popup.css`)

Clean, simple layout
Proper spacing between sections
Scrollable output area with monospace font for JSON
Responsive design for popup window

2.4 Popup Logic (`src/popup/popup.ts`)

On load: Get current tab URL and auto-populate Agent ID field
On “Start” button click: Validate inputs (non-empty)
Show loading state during simulation
Display error messages if simulation fails
Display JSON response in output section

Phase 3: Simulation Logic

Key Architecture: The ElevenLabs REST API’s simulate-conversation endpoint is extremely simple:

✅ Single HTTPS POST request - Returns complete transcript
✅ No state management needed
✅ No background processing required
✅ Runs directly in popup context using Fetch API
✅ No SDK dependencies needed

3.1 Simulation Function (`src/popup/popup.ts`)

Create async function runSimulation(apiKey: string, agentId: string, userPrompt: string)

Make REST API call using Fetch API:

const response = await fetch(
  `https://api.elevenlabs.io/v1/convai/agents/${agentId}/simulate-conversation`,
  {
    method: 'POST',
    headers: {
      'xi-api-key': apiKey,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      simulation_specification: {
        simulated_user_config: {
          prompt: {
            prompt: userPrompt,
            llm: 'gpt-4o',        // hardcoded for now
            temperature: 0.7,      // hardcoded for now
          },
        },
      },
    }),
  }
);

if (!response.ok) {
  throw new Error(`API Error: ${response.status} ${response.statusText}`);
}

return await response.json();

Return response as JSON
Handle errors with try-catch and display in UI

3.2 Response Display

Format JSON response with JSON.stringify(response, null, 2)
Display in scrollable output area
Show “Simulation running…” during API call
Clear previous results when starting new simulation

Phase 4: Dynamic Variables Support ✅ COMPLETED

Problem: Some agents require dynamic variables (e.g., user_name, order_id) to be provided in the simulation. Without these, the API returns a 400 error:

{
  "detail": {
    "status": "missing_dynamic_variables",
    "message": "Missing required dynamic variables in first message: {'user_name'}"
  }
}

4.1 Dynamic Variables UI (`src/popup/popup.html`)

Add Dynamic Variables section between “Simulated User Prompt” and “Start Button”
Create a container for variable key-value pairs
Each variable row should contain:
- Text input for variable key (e.g., “user_name”)
- Text input for variable value (e.g., “Javier Perez”)
- Remove button (X) to delete the variable pair
Add “Add Variable” button to create new key-value pair rows
Start with one empty variable row by default
Support adding unlimited variable pairs dynamically
NEW: Add “Load Defaults” button to auto-fetch placeholder values from agent config

4.2 Dynamic Variables Styling (`src/popup/popup.css`)

Style variable rows with inline layout (key and value inputs side-by-side)
Add remove button styling (small X icon or button)
Add “Add Variable” button styling consistent with “Start” button
Ensure proper spacing between variable rows
Make variable section visually distinct with border or background
NEW: Add “Load Defaults” button styling with purple theme matching primary button

4.3 Dynamic Variables Logic (`src/popup/popup.ts`)

Implement dynamic row addition when “Add Variable” button is clicked
Implement row removal when remove button (X) is clicked
Collect all variable key-value pairs when “Start” button is clicked
Skip empty variable rows (where both key and value are empty)
Build dynamic_variables object: { [key: string]: string }
Update runSimulation() function signature to accept dynamicVariables: Record<string, string>

Include dynamic_variables in API request body:

body: JSON.stringify({
  simulation_specification: {
    simulated_user_config: {
      prompt: {
        prompt: userPrompt,
        llm: 'gpt-4o',
        temperature: 0.7,
      },
    },
    dynamic_variables: dynamicVariables, // Add this field
  },
})

4.4 Auto-Load Default Variables Feature NEW ✨

API Integration: Created fetchAgentConfig() function that calls GET /v1/convai/agents/{agent_id} endpoint
Data Extraction: Navigates to conversation_config.agent.dynamic_variables.dynamic_variable_placeholders in response
Type Conversion: Converts all placeholder values (string, number, integer, boolean) to strings for UI consistency
Load Handler: Implemented handleLoadVariables() function with:
- Input validation (API key and Agent ID required)
- Loading state with disabled button
- Success/error status messages
- Automatic population of variable rows
UI Enhancement: Updated addVariableRow() to accept optional default key/value parameters
Event Wiring: Connected “Load Defaults” button to load handler

User Flow:

User opens extension on an ElevenLabs agent page (Agent ID auto-detected)
User enters their API key
User clicks “Load Defaults” button
Extension fetches agent configuration from ElevenLabs API
Default dynamic variable placeholders are automatically populated
User can modify, add, or remove variables as needed
User clicks “Start Simulation” with pre-filled variables

Benefits:

Eliminates manual entry of known placeholder values
Reduces user errors from typos or incorrect variable names
Improves testing workflow efficiency
Graceful degradation if API fails (user can still manually enter variables)

Phase 5: Persistent State Management ✅ COMPLETED

Problem: Currently, when the popup closes (user clicks outside or switches tabs), all input values are lost. When reopening the popup, users must re-enter their API key, simulated user prompt, and dynamic variables every time.

Goal: Preserve user inputs across popup sessions using Chrome’s storage API, so values persist even when the popup is closed and reopened.

5.1 Storage Strategy

API: Use chrome.storage.local API (persistent, local-only storage)
Storage Keys:
- apiKey: string (API key input)
- userPrompt: string (simulated user prompt textarea)
- dynamicVariables: Array<{ key: string, value: string }> (dynamic variables rows)
Note: Agent ID should NOT be persisted - it should always be auto-detected from the current tab URL
Privacy: All data stored locally on user’s device, never synced to cloud

5.2 Manifest Updates (`manifest.json`)

Add storage permission to manifest.json:

"permissions": ["activeTab", "storage"]

5.3 Save Logic (`src/popup/popup.ts`)

Create saveState() function that:
- Collects current values from all inputs
- Saves to chrome.storage.local
- Handles errors silently (don’t block user workflow)
Call saveState() when:
- User types in API key input (debounced, 500ms delay)
- User types in simulated user prompt textarea (debounced, 500ms delay)
- User adds/removes/modifies dynamic variable rows (debounced, 500ms delay)
Implement debounce utility function to avoid excessive storage writes

5.4 Load Logic (`src/popup/popup.ts`)

Create loadState() async function that:
- Retrieves stored values from chrome.storage.local
- Populates API key input field
- Populates simulated user prompt textarea
- Recreates dynamic variable rows from stored array
- Gracefully handles missing/corrupted storage data
Call loadState() on popup initialization (DOMContentLoaded)
Load order:
1. Auto-detect and populate Agent ID from current tab URL (always fresh)
2. Load persisted state (API key, user prompt, dynamic variables)
3. If no dynamic variables stored, show one empty row (current behavior)

5.5 Clear State Feature

Add “Clear All” button to reset all inputs and clear storage
Implement clearState() function that:
- Clears chrome.storage.local for the extension
- Resets all input fields to empty
- Shows one empty dynamic variable row
Add confirmation dialog before clearing to prevent accidental data loss
Style “Clear All” button in header with semi-transparent white background

5.6 Storage Schema

interface StoredState {
  apiKey?: string;
  userPrompt?: string;
  dynamicVariables?: Array<{ key: string; value: string }>;
}

5.7 Edge Cases Handled

Empty storage: First-time users see default UI (empty inputs, one variable row)
Corrupted storage: Gracefully fallback to defaults if errors occur
Agent ID changes: When switching between different agent pages, Agent ID auto-updates but other fields remain (expected behavior)
Storage quota: Errors handled gracefully with console logging

5.8 Security Considerations

API Key Storage: Storing API keys in chrome.storage.local is acceptable for local development tools
- Data is stored only on user’s device
- Not accessible to websites or other extensions
- Cleared when extension is uninstalled

User Flow After Phase 5:

User opens extension → Sees previously entered values auto-filled
User modifies inputs → Values auto-save in background (500ms debounce)
User closes popup (clicks outside) → State persisted
User reopens popup later → All values restored, ready to run simulation
User can click “Clear All” to reset everything

Benefits:

Drastically improved UX - no re-entering values every time
Faster testing iterations
Reduced friction for frequent users
Maintains fresh Agent ID detection (not persisted)

Phase 6: Side Panel Migration ✅ COMPLETED

Problem: Popup closes automatically when users click outside or interact with the webpage.

Solution: Migrate to Chrome’s Side Panel API for persistent UI that stays open during page interactions.

6.1 Changes Required

Manifest (public/manifest.json):

Add "sidePanel" to permissions array
Add side_panel configuration pointing to popup.html
Update action to remove default_popup (keep icon configuration)
Add minimum Chrome version: 114
Add background service worker configuration

Background Script (src/background/background.ts):

Create background service worker that calls chrome.sidePanel.setPanelBehavior({ openPanelOnActionClick: true })
This enables the side panel to open when clicking the extension icon

Webpack (webpack.config.js):

Add background script to entry points for bundling

6.2 Testing

Side panel opens when clicking extension icon
Side panel stays open when clicking on webpage
All existing functionality works

Phase 7: Conversation Display Enhancement ✅ COMPLETED

Problem: The current output displays the entire raw JSON response, which contains too much technical metadata that users don’t need for everyday testing. While the full JSON is useful for debugging, users primarily care about the actual conversation flow between the agent and the simulated user.

Goal: Add a user-friendly conversation display section that shows a clean, readable transcript of the dialogue, while keeping the full JSON output available below for those who need it.

7.1 UI Structure (`src/popup/popup.html`) ✅

Add new Conversation Transcript section between “Output Section” header and the JSON <pre> element
Create a dedicated container (<div id="conversationTranscript">) for displaying the conversation
Keep existing JSON output section (rename to “Full Response (JSON)”)
Layout order (top to bottom):
1. Input fields (API Key, Agent ID, User Prompt, Dynamic Variables)
2. Start button
3. NEW: Conversation Transcript section (when results available)
4. Full Response JSON section (existing, when results available)

7.2 Conversation Display Format ✅

Data Source: Extract from response.simulated_conversation array

Fields to Use:

role: “agent” or “user”
message: The actual message text (may be null for some turns, e.g., tool calls)

Display Format:

User messages: Blue background with blue left border
Agent messages: Purple background with purple left border
Each message includes a label (USER/AGENT) and the message text

Edge Cases Handled:

Skip conversation turns where message is null (e.g., tool calls without spoken messages)
Handle system messages like “==! END_CALL!==” gracefully (skip them)
Handle empty conversations (show friendly message)
Preserve newlines and formatting within messages

7.3 Styling (`src/popup/popup.css`) ✅

Style conversation transcript section with:
- Clear visual separation from JSON output (border-top separator)
- Readable typography (13px font for message text)
- Proper line spacing between conversation turns
- Visual distinction between “User” and “Agent” labels (uppercase, colored, bold)
Make transcript scrollable if conversation is long (max-height: 400px)
Use alternating background colors for user/agent messages for readability
- User: Light blue background (#e3f2fd) with blue border (#1976d2)
- Agent: Light purple background (#f3e5f5) with purple border (#7b1fa2)
Section is clearly labeled (“Conversation Transcript”)
Added custom scrollbar styling for consistency

7.4 Display Logic (`src/popup/popup.ts`) ✅

Create displayConversation() function that:
- Takes simulated_conversation array from API response
- Filters out turns where message is null or empty
- Filters out system messages (e.g., “==! END_CALL!==”)
- Builds formatted HTML with styled message elements
- Displays in conversation transcript container
- Shows fallback message if no displayable messages found
Call displayConversation() when simulation completes successfully (via displayResult())
Clear conversation transcript when starting new simulation
Clear conversation transcript when “Clear” button is clicked
Added TypeScript interfaces for type safety (ConversationTurn, SimulationResponse)

User Flow After Phase 7:

User runs a simulation with their agent
Results section appears with two parts:
- Conversation Transcript: Clean, readable dialogue between user and agent
- Full Response (JSON): Complete API response for debugging
User can quickly review the conversation flow without parsing JSON
User can still access full technical details if needed

Benefits:

Significantly improved UX for everyday testing workflows
Easy-to-read conversation format reduces cognitive load
Maintains full technical data access for debugging
Professional appearance with color-coded messages
Handles edge cases gracefully (null messages, system messages, empty conversations)

Technical Considerations

Security

HTTPS Only: All API calls over HTTPS (ElevenLabs SDK handles this)
Content Security Policy: Strict CSP in manifest
Input Validation: Basic validation for required fields

Performance

Simple Architecture: Everything runs in popup - no background processing
Direct API Calls: Single API call, no complex state management
Memory Efficient: Popup closes after use, no persistent processes

Browser Compatibility

Target Chrome 110+ (Manifest V3)
Compatible with Edge (Chromium-based)

Rate Limiting

Display error messages if API rate limits are hit
User can manually retry after waiting

Dependencies & APIs

ElevenLabs REST API

Endpoint: POST https://api.elevenlabs.io/v1/convai/agents/{agent_id}/simulate-conversation
Authentication: xi-api-key header with API key
Request Body:
- simulation_specification object containing:
  - simulated_user_config: Configuration for the simulated user
    - prompt: Object with prompt (string), llm (string), temperature (number)
  - Optional: dynamic_variables, extra_evaluation_criteria, new_turns_limit, tool_mock_config
Response: AgentSimulatedChatTestResponseModel containing:
- simulated_conversation: Array of conversation turns
- analysis: Evaluation results and metrics
Note: This is a REST API endpoint, not an SDK. No NPM packages needed for the API call itself.

Why Not Use the SDKs?

@elevenlabs/elevenlabs-js (Node.js SDK): Cannot run in browser environments (Node.js only)
@elevenlabs/client (Browser SDK): Designed for real-time voice conversations via WebSocket/WebRTC, does NOT support the simulate-conversation REST API endpoint
Solution: Direct REST API calls using native Fetch API - simple, lightweight, no dependencies

Chrome Extension APIs

chrome.tabs - Tab management and URL access for Agent ID auto-detection
chrome.storage.local - Persistent state management (API key, user prompt, dynamic variables)

elevenlabs-agent-simulator

ElevenLabs Agent Simulator - Chrome Extension Implementation Plan

Project Overview

Architecture Overview

Extension Structure

Implementation Phases

Phase 1: Project Setup & Foundation

1.1 Initialize Project Structure

1.2 Dependencies

1.3 Create manifest.json

Phase 2: Popup UI Implementation

2.1 URL Parser Utility (src/utils/url-parser.ts)

2.2 Popup Interface (src/popup/popup.html)

2.3 Popup Styling (src/popup/popup.css)

2.4 Popup Logic (src/popup/popup.ts)

Phase 3: Simulation Logic

3.1 Simulation Function (src/popup/popup.ts)

3.2 Response Display

Phase 4: Dynamic Variables Support ✅ COMPLETED

4.1 Dynamic Variables UI (src/popup/popup.html)

4.2 Dynamic Variables Styling (src/popup/popup.css)

4.3 Dynamic Variables Logic (src/popup/popup.ts)

4.4 Auto-Load Default Variables Feature NEW ✨

Phase 5: Persistent State Management ✅ COMPLETED

5.1 Storage Strategy

5.2 Manifest Updates (manifest.json)

5.3 Save Logic (src/popup/popup.ts)

5.4 Load Logic (src/popup/popup.ts)

5.5 Clear State Feature

5.6 Storage Schema

5.7 Edge Cases Handled

5.8 Security Considerations

Phase 6: Side Panel Migration ✅ COMPLETED

6.1 Changes Required

6.2 Testing

Phase 7: Conversation Display Enhancement ✅ COMPLETED

7.1 UI Structure (src/popup/popup.html) ✅

7.2 Conversation Display Format ✅

7.3 Styling (src/popup/popup.css) ✅

7.4 Display Logic (src/popup/popup.ts) ✅

Technical Considerations

Security

Performance

Browser Compatibility

Rate Limiting

Dependencies & APIs

ElevenLabs REST API

Why Not Use the SDKs?

Chrome Extension APIs

2.1 URL Parser Utility (`src/utils/url-parser.ts`)

2.2 Popup Interface (`src/popup/popup.html`)

2.3 Popup Styling (`src/popup/popup.css`)

2.4 Popup Logic (`src/popup/popup.ts`)

3.1 Simulation Function (`src/popup/popup.ts`)

4.1 Dynamic Variables UI (`src/popup/popup.html`)

4.2 Dynamic Variables Styling (`src/popup/popup.css`)

4.3 Dynamic Variables Logic (`src/popup/popup.ts`)

5.2 Manifest Updates (`manifest.json`)

5.3 Save Logic (`src/popup/popup.ts`)

5.4 Load Logic (`src/popup/popup.ts`)

7.1 UI Structure (`src/popup/popup.html`) ✅

7.3 Styling (`src/popup/popup.css`) ✅

7.4 Display Logic (`src/popup/popup.ts`) ✅