Skip to content

Provider Implementation Guide

Jump to bottom
github-actions[bot] edited this page Jan 25, 2026 · 4 revisions

This guide shows the correct patterns for implementing providers and models in dartantic 1.0.

Critical Patterns Summary

Every provider MUST:

  1. Have a private static final logger: static final Logger _logger = ...
  2. Define a public defaultBaseUrl: static final defaultBaseUrl = Uri.parse('...')
  3. Use super.baseUrl in constructor (not baseUrl: baseUrl)
  4. Pass baseUrl ?? defaultBaseUrl to models in create methods
  5. Models should accept required Uri baseUrl from provider
  6. Implement createMediaModel (or throw UnsupportedError) for providers supporting media generation

Provider Implementation Pattern

class ExampleProvider extends Provider<
  ExampleChatOptions,
  ExampleEmbeddingsOptions,
  ExampleMediaOptions
> {
  // IMPORTANT: Logger must be private (_logger not log) and static final
  static final Logger _logger = Logger('dartantic.chat.providers.example');

  /// Default base URL for the Example API.
  /// IMPORTANT: All providers must have a public defaultBaseUrl constant
  static final defaultBaseUrl = Uri.parse('https://api.example.com/v1');

  /// Environment variable name for API key
  static const defaultApiKeyName = 'EXAMPLE_API_KEY';

  /// Creates a provider instance with optional overrides.
  ///
  /// API key resolution:
  /// - Constructor: Uses tryGetEnv() to allow lazy initialization without throwing
  /// - Model creation: Validates API key and throws if required but not found
  ExampleProvider({
    String? apiKey,
    super.baseUrl,  // Use super.baseUrl, don't provide defaults here
  }) : super(
          apiKey: apiKey ?? tryGetEnv(defaultApiKeyName),
          name: 'example',
          displayName: 'Example AI',
          aliases: const ['ex', 'example-ai'],
          apiKeyName: defaultApiKeyName,  // null for local providers
          defaultModelNames: const {
            ModelKind.chat: 'example-chat-v1',
            ModelKind.embeddings: 'example-embed-v1',
          },
        );

  @override
  ChatModel createChatModel({
    String? name,  // Note: 'name' not 'modelName'
    List<Tool>? tools,
    double? temperature,
    ExampleChatOptions? options,
  }) {
    final modelName = name ?? defaultModelNames[ModelKind.chat]!;

    _logger.info(
      'Creating Example model: $modelName with '
      '${tools?.length ?? 0} tools, '
      'temperature: $temperature',
    );

    // Validate API key at model creation time
    if (apiKey == null) {
      throw ArgumentError('EXAMPLE_API_KEY is required for Example provider');
    }

    return ExampleChatModel(
      name: modelName,  // Pass as 'name'
      apiKey: apiKey,  // Now validated to be non-null
      baseUrl: baseUrl ?? defaultBaseUrl,  // IMPORTANT: Pass baseUrl with fallback
      tools: tools,
      temperature: temperature,
      defaultOptions: ExampleChatOptions(
        temperature: temperature ?? options?.temperature,
        topP: options?.topP,
        maxTokens: options?.maxTokens,
        // Add other options as needed
      ),
    );
  }

  @override
  EmbeddingsModel createEmbeddingsModel({
    String? name,
    ExampleEmbeddingsOptions? options,
  }) {
    final modelName = name ?? defaultModelNames[ModelKind.embeddings]!;

    _logger.info(
      'Creating Example embeddings model: $modelName with '
      'options: $options',
    );

    // Validate API key at model creation time
    if (apiKey == null) {
      throw ArgumentError('EXAMPLE_API_KEY is required for Example provider');
    }

    return ExampleEmbeddingsModel(
      name: modelName,
      apiKey: apiKey,  // Now validated to be non-null
      baseUrl: baseUrl ?? defaultBaseUrl,  // Use provider's default
      defaultOptions: options,  // Pass options directly
    );
  }

  @override
  Stream<ModelInfo> listModels() async* {
    // Use defaultBaseUrl when baseUrl is null
    final resolvedBaseUrl = baseUrl ?? defaultBaseUrl;
    final url = appendPath(resolvedBaseUrl, 'models');

    _logger.info('Fetching models from Example API: $url');

    // Implementation to list available models
    // Real implementations would make HTTP calls with the resolved URL
    
    yield ModelInfo(
      name: 'example-chat-v1',
      providerName: name,
      kinds: {ModelKind.chat},
      displayName: 'Example Chat Model v1',
      description: 'A chat model for text generation',
    );
    yield ModelInfo(
      name: 'example-embed-v1',
      providerName: name,
      kinds: {ModelKind.embeddings},
      displayName: 'Example Embeddings Model v1',
      description: 'A model for text embeddings',
    );
  }

  @override
  MediaGenerationModel<ExampleMediaOptions> createMediaModel({
    String? name,
    List<Tool>? tools,
    ExampleMediaOptions? options,
  }) {
    // Providers that don't support media generation should throw:
    // throw UnsupportedError('Media generation is not supported');

    final modelName = name ?? defaultModelNames[ModelKind.media]!;
    return ExampleMediaModel(
      name: modelName,
      tools: tools,
      defaultOptions: options ?? const ExampleMediaOptions(),
    );
  }
}

Chat Model Implementation Pattern

class ExampleChatModel extends ChatModel<ExampleChatOptions> {
  /// Creates a chat model instance.
  ExampleChatModel({
    required super.name,  // Always 'name', passed to super
    required this.apiKey,  // Non-null for cloud providers
    required this.baseUrl,  // Required from provider (already has fallback)
    super.tools,
    super.temperature,
    super.defaultOptions,
  }) : _client = ExampleClient(
          apiKey: apiKey,
          // How to pass baseUrl depends on client library:
          // If client accepts nullable String:
          baseUrl: baseUrl.toString(),
          // If client requires non-nullable String and you need different default:
          // baseUrl: baseUrl.toString() ?? 'https://api.example.com/v1',
        );

  /// The API key (required for cloud providers).
  final String apiKey;

  /// Base URL for API requests (provider supplies with fallback).
  final Uri baseUrl;

  final ExampleClient _client;

  @override
  Stream<ChatResult<ChatMessage>> sendStream(
    List<ChatMessage> messages, {
    ExampleChatOptions? options,
    Schema? outputSchema,
  }) async* {
    // Process messages
    final processedMessages = messages;
    
    // Stream implementation
    await for (final chunk in _client.stream(...)) {
      yield ChatResult<ChatMessage>(
        // ... result construction
      );
    }
  }

  @override
  void dispose() {
    _client.close();
  }
}

Embeddings Model Implementation Pattern

class ExampleEmbeddingsModel extends EmbeddingsModel<ExampleEmbeddingsOptions> {
  /// Creates an embeddings model instance.
  ExampleEmbeddingsModel({
    required super.name,  // Always 'name'
    required this.apiKey,
    required this.baseUrl,  // Required from provider
    super.defaultOptions,
    super.dimensions,
    super.batchSize,
  }) : _client = ExampleClient(
          apiKey: apiKey,
          baseUrl: baseUrl.toString(),
        );

  final String apiKey;
  final Uri baseUrl;
  final ExampleClient _client;

  @override
  Future<EmbeddingsResult> embedQuery(
    String query, {
    ExampleEmbeddingsOptions? options,
  }) async {
    final response = await _client.embed(
      texts: [query],
      model: name,
      dimensions: options?.dimensions ?? dimensions,
    );
    
    return EmbeddingsResult(
      embedding: response.embeddings.first,
      usage: LanguageModelUsage(
        inputTokens: response.usage?.inputTokens,
        outputTokens: response.usage?.outputTokens,
      ),
    );
  }

  @override
  Future<BatchEmbeddingsResult> embedDocuments(
    List<String> texts, {
    ExampleEmbeddingsOptions? options,
  }) async {
    final response = await _client.embed(
      texts: texts,
      model: name,
      dimensions: options?.dimensions ?? dimensions,
    );
    
    return BatchEmbeddingsResult(
      embeddings: response.embeddings,
      usage: LanguageModelUsage(
        inputTokens: response.usage?.inputTokens,
        outputTokens: response.usage?.outputTokens,
      ),
    );
  }

  @override
  void dispose() {
    _client.close();
  }
}

Local Provider Pattern (No API Key or Base URL)

class LocalProvider extends Provider<LocalChatOptions, EmbeddingsModelOptions> {
  // Logger must still be private and static final
  static final Logger _logger = Logger('dartantic.chat.providers.local');

  // Local providers typically connect to localhost, so may have a default
  // But some may not need any URL at all
  static final defaultBaseUrl = Uri.parse('http://localhost:11434/api');

  LocalProvider() : super(
    name: 'local',
    displayName: 'Local Model',
    aliases: const [],
    apiKeyName: null,  // No API key needed
    defaultModelNames: const {
      ModelKind.chat: 'llama3.2',
    },
    baseUrl: null,  // No base URL override in constructor
    apiKey: null,
  );

  @override
  ChatModel createChatModel({
    String? name,
    List<Tool>? tools,
    double? temperature,
    LocalChatOptions? options,
  }) {
    final modelName = name ?? defaultModelNames[ModelKind.chat]!;

    _logger.info(
      'Creating Local model: $modelName with '
      '${tools?.length ?? 0} tools, '
      'temp: $temperature',
    );

    return LocalChatModel(
      name: modelName,
      tools: tools,
      temperature: temperature,
      baseUrl: baseUrl ?? defaultBaseUrl,  // Even local providers should follow pattern
      defaultOptions: LocalChatOptions(
        temperature: temperature ?? options?.temperature,
        // Add other options as needed
      ),
    );
  }

  @override
  EmbeddingsModel<EmbeddingsModelOptions> createEmbeddingsModel({
    String? name,
    EmbeddingsModelOptions? options,
  }) => throw Exception('Local provider does not support embeddings models');
}

Provider Factory Registration

Add your provider factory to Agent.providerFactories:

// In your application code or during initialization:
Agent.providerFactories['example'] = ExampleProvider.new;

// For providers with aliases:
Agent.providerFactories['example'] = ExampleProvider.new;
Agent.providerFactories['ex'] = ExampleProvider.new;  // alias

// Now the provider is available:
final provider = Agent.getProvider('example');
final agent = Agent('example:example-chat-v1');

Message Mapping Rules

⚠️ CRITICAL: ThinkingPart MUST NEVER Be Sent to LLMs

Every message mapper that converts List<ChatMessage> to provider-specific format MUST include an assertion at the entry point that fails if any ThinkingPart appears in outbound messages:

// IMPORTANT: ThinkingPart MUST NEVER be sent to the LLM. Thinking content
// is generated by the model and should only flow FROM the model, never TO
// it. If ThinkingPart appears in outbound messages, it indicates a bug in
// the message handling pipeline.
assert(
  !any((m) => m.parts.any((p) => p is ThinkingPart)),
  'ThinkingPart must never be sent to the LLM. '
  'Thinking content is model-generated output only.',
);

This assertion exists in ALL provider message mappers (Anthropic, Google, OpenAI, OpenAI Responses, Mistral, Ollama). When implementing a new provider, include this assertion in your message mapper.

When implementing message mappers:

  1. ThinkingPart filtering: Assert that no ThinkingPart exists in outbound messages (see above)
  2. Part type handling: Handle TextPart, DataPart, LinkPart, ToolPart explicitly
  3. Unknown parts: Skip unknown part types gracefully or throw if strict validation is needed
  4. Role mapping: Map ChatMessageRole to provider-specific role strings

Key Implementation Rules

  1. Parameter Naming: Always use name for model names, not model, modelId, or modelName
  2. API Key Handling:
    • Cloud providers: use tryGetEnv() in constructor (allows lazy initialization)
    • Model creation: validate API key and throw if required but not found
    • Local providers: no API key parameter at all
  3. Base URL Management:
    • Provider: Define public static final defaultBaseUrl = Uri.parse('...')
    • Constructor: Use super.baseUrl parameter (no defaults)
    • Model creation: Pass baseUrl ?? defaultBaseUrl to models
    • Models: Accept required Uri baseUrl from provider
  4. Options Handling: Create new options objects with merged values from parameters and options
  5. Logger Convention:
    • MUST be private: static final Logger _logger = ... (not log or public)
    • Place immediately after class declaration
    • Use hierarchical naming: Logger('dartantic.chat.providers.example')
    • Log lifecycle milestones at INFO, detailed events at FINE
  6. Capabilities: Accurately declare what your provider supports
  7. Error Handling:
    • Follow exception transparency: no try-catch blocks that suppress errors
    • Let exceptions bubble up for diagnosis
    • Only wrap provider-specific exceptions at boundaries
  8. ModelInfo: Include displayName and description when available
  9. HTTP Client: Wrap HTTP clients with RetryHttpClient for automatic retry on transient failures
  10. Message History: Must pass validateMessageHistory() utility
    • System messages only at index 0
    • Strict user/model/user/model alternation thereafter
  11. Metadata: All metadata values must be JSON-serializable (String, num, bool, List, Map, null)
  12. Tool ID Coordination: Use tool_id_helpers.dart for providers that don't supply tool IDs

Special Cases

Different API Endpoints

If your provider needs different endpoints for different operations (e.g., OpenAI Responses):

class SpecialProvider extends Provider<...> {
  // Primary endpoint (e.g., for chat)
  static final defaultBaseUrl = Uri.parse('https://api.example.com/v1/special');

  // Secondary endpoint (e.g., for embeddings or model listing)
  static final _standardApiUrl = Uri.parse('https://api.example.com/v1');

  @override
  ChatModel createChatModel(...) {
    // Uses special endpoint
    return SpecialChatModel(
      baseUrl: baseUrl ?? defaultBaseUrl,
      ...
    );
  }

  @override
  EmbeddingsModel createEmbeddingsModel(...) {
    // Uses standard endpoint
    return SpecialEmbeddingsModel(
      baseUrl: baseUrl ?? _standardApiUrl,
      ...
    );
  }

  @override
  Stream<ModelInfo> listModels() async* {
    // Uses standard endpoint for listing
    final resolvedBaseUrl = baseUrl ?? _standardApiUrl;
    ...
  }
}

Client Library Requirements

Different HTTP client libraries have different requirements:

// If client accepts nullable String:
ExampleClient(
  baseUrl: baseUrl?.toString(),
)

// If client requires non-nullable String:
OpenAIClient(
  baseUrl: baseUrl.toString() ?? 'https://default.url',
)

Testing Your Provider

// Register the provider factory first
Agent.providerFactories['example'] = ExampleProvider.new;

// Test provider discovery
final provider = Agent.getProvider('example');
assert(provider.name == 'example');

// Test model creation
final chatModel = provider.createChatModel();
final embeddingsModel = provider.createEmbeddingsModel();

// Test model listing
await for (final model in provider.listModels()) {
  print('${model.id} supports ${model.kinds}');
}

// Test Agent integration
final agent = Agent('example');
final result = await agent.send('Hello');

// Test embeddings
final embed = await agent.embedQuery('test');

Custom Orchestrators for Provider Limitations

Some providers have API limitations that require custom orchestration logic. The ChatOrchestratorProvider interface allows providers to supply their own orchestrators based on the request context.

When to Use Custom Orchestrators

Use a custom orchestrator when:

  1. API Limitations: Provider doesn't support certain combinations of features (e.g., tools + typed output)
  2. Special Workflows: Provider requires multi-step workflows for certain capabilities
  3. Optimization: Provider-specific patterns can be optimized for better performance

Implementing ChatOrchestratorProvider

import 'package:json_schema/json_schema.dart';
import '../agent/orchestrators/streaming_orchestrator.dart';
import '../agent/orchestrators/default_streaming_orchestrator.dart';
import 'chat_orchestrator_provider.dart';

class GoogleProvider extends Provider<GoogleChatModelOptions, GoogleEmbeddingsModelOptions>
    implements ChatOrchestratorProvider {
  // ... existing provider code ...

  @override
  (StreamingOrchestrator, List<Tool>?) getChatOrchestratorAndTools({
    required Schema? outputSchema,
    required List<Tool>? tools,
  }) {
    final hasTools = tools != null && tools.isNotEmpty;

    if (outputSchema != null && hasTools) {
      // Use custom orchestrator for tools + typed output
      // Note: Not const because orchestrator has mutable state
      return (GoogleDoubleAgentOrchestrator(), tools);
    }

    // Standard cases use default orchestrator
    return (const DefaultStreamingOrchestrator(), tools);
  }
}

Example: Google Double Agent Pattern

Google's API doesn't support using tools and outputSchema simultaneously. The GoogleDoubleAgentOrchestrator works around this with a two-phase approach:

Phase 1 - Tool Execution:

  • Send request with tools (no outputSchema)
  • Suppress text output (only care about tool calls)
  • Execute all tool calls
  • Accumulate tool results

Phase 2 - Structured Output:

  • Send tool results with outputSchema (no tools)
  • Return structured JSON output
  • Attach metadata about suppressed content from Phase 1
class GoogleDoubleAgentOrchestrator extends DefaultStreamingOrchestrator {
  // Instance state is safe - each request gets new orchestrator instance
  bool _isPhase1 = true;

  @override
  String get providerHint => 'google-double-agent';

  @override
  void initialize(StreamingState state) {
    super.initialize(state);
    _isPhase1 = true;
  }

  @override
  Stream<StreamingIterationResult> processIteration(
    ChatModel<ChatModelOptions> model,
    StreamingState state, {
    Schema? outputSchema,
  }) async* {
    if (_isPhase1) {
      // Phase 1: Execute tools
      yield* _executePhase1(model, state);

      // Transition to Phase 2
      if (!_isPhase1) {
        yield* processIteration(model, state, outputSchema: outputSchema);
      }
    } else {
      // Phase 2: Get structured output
      yield* _executePhase2(model, state, outputSchema);
    }
  }

  @override
  bool allowTextStreaming(
    StreamingState state,
    ChatResult<ChatMessage> result,
  ) => !_isPhase1; // Only stream text in Phase 2
}

Model Behavior Changes

When implementing a double agent pattern, the model's sendStream() method needs to conditionally exclude tools when outputSchema is present:

@override
Stream<ChatResult<ChatMessage>> sendStream(
  List<ChatMessage> messages, {
  GoogleChatModelOptions? options,
  Schema? outputSchema,
}) {
  final request = _buildRequest(
    messages,
    options: options,
    outputSchema: outputSchema,
  );

  // ... streaming implementation
}

gl.GenerateContentRequest _buildRequest(
  List<ChatMessage> messages, {
  GoogleChatModelOptions? options,
  Schema? outputSchema,
}) {
  // Google doesn't support tools + outputSchema simultaneously.
  // When outputSchema is provided, exclude tools (double agent phase 2).
  final toolsToSend = outputSchema != null
      ? const <Tool>[]
      : (tools ?? const <Tool>[]);

  return gl.GenerateContentRequest(
    model: normalizedModel,
    systemInstruction: _extractSystemInstruction(messages),
    contents: contents,
    generationConfig: generationConfig,
    tools: toolsToSend.toToolList(
      enableCodeExecution: enableCodeExecution,
    ),
  );
}

Key Implementation Notes

  1. Instance State: Custom orchestrators can have mutable instance variables because each request gets a new orchestrator instance
  2. Metadata Preservation: Use StreamingState.addSuppressedTextParts() and addSuppressedMetadata() to track suppressed content
  3. Tool Result Consolidation: Phase 1 executes tools and adds results to history before Phase 2
  4. Text Suppression: Override allowTextStreaming() to control when text is streamed to the user
  5. Clean Separation: The orchestrator handles workflow logic; the model handles API communication

Test Capability Registration

When implementing a provider with advanced features like double agent pattern, ensure the provider is registered in the test capability mapping (test/test_helpers/run_provider_test.dart) with appropriate capabilities like ProviderTestCaps.typedOutputWithTools. This ensures the provider is included in tests that verify tools + typed output functionality.

Note: Provider capabilities (ProviderTestCaps) are test-only and describe what the default model supports for testing. They are NOT part of the public API.

Core Architecture

Features

Configuration

Implementation

Clone this wiki locally