Skip to main content

# Configuring LLMs

Large Language Models (LLMs) serve as the intelligence foundation for Agent Mesh, powering everything from natural language understanding to complex reasoning and decision-making. The system provides flexible configuration options that allow you to connect with various LLM providers through a unified interface, making it easy to switch between providers or use multiple models for different purposes.

You can configure LLM settings in two locations within your Agent Mesh deployment. The `apps.app_config.model` field allows you to specify model settings for individual agents or gateways, providing fine-grained control over which models specific components use. Alternatively, you can define models globally in the `shared_config.yaml` file under the `models` section, creating reusable configurations that multiple components can reference. For detailed information about the overall configuration structure and shared configuration management, see the Configuring Agent Mesh.

## Understanding LiteLLM Integration​

Agent Mesh leverages LiteLLM to provide seamless integration with numerous LLM providers. This integration layer abstracts the differences between various provider APIs, allowing you to use a consistent configuration format regardless of whether you're connecting to OpenAI, Anthropic, Google, Amazon, or other supported providers.

The configuration system passes all fields from the `models` section directly to LiteLLM, giving you access to the full range of provider-specific options and features. This approach ensures that you can take advantage of advanced capabilities offered by different providers while maintaining a consistent configuration experience across your deployment.

Environment variables provide a secure and flexible way to manage sensitive information such as API keys and endpoint URLs. The configuration system supports environment variable substitution using the format `${ENV_VAR_NAME, default_value}`, allowing you to keep secrets out of your configuration files while providing sensible defaults for development environments.

## Provider-Specific Configurations​

### OpenAI​

OpenAI provides some of the most widely-used language models, including the GPT series. The configuration requires minimal setup, needing only the model name and your API key. The system uses OpenAI's default endpoints automatically, simplifying the configuration process.

    model: gpt-5
    api_key: ${OPENAI_API_KEY}

If your organization belongs to multiple OpenAI organizations, you can specify which organization to use by adding the `organization` parameter. This parameter helps ensure billing and usage tracking align with your organizational structure.

For comprehensive details about OpenAI-specific configuration options and advanced features, see the OpenAI documentation.

### Azure OpenAI​

Azure OpenAI Service provides OpenAI models through Microsoft's cloud infrastructure, offering additional enterprise features such as private networking and enhanced security controls. The configuration requires specifying your custom Azure endpoint, API key, and API version.

    model: azure/gpt-5
    api_base: ${AZURE_API_BASE,"https://your-custom-endpoint.openai.azure.com/"}
    api_key: ${AZURE_API_KEY}
    api_version: ${AZURE_API_VERSION,"2024-12-01-preview"}

The model name must include the `azure/` prefix to indicate you're using Azure OpenAI rather than the standard OpenAI service. The API base URL points to your specific Azure OpenAI resource, which you configure during the Azure setup process.

For detailed information about Azure-specific configuration options, deployment models, and enterprise features, see the Azure OpenAI documentation.

### Google Vertex AI​

Google Vertex AI provides access to both Google's own models and third-party models through a unified platform. This service offers enterprise-grade features including fine-tuning capabilities, model versioning, and integration with other Google Cloud services.

    model: vertex_ai/claude-sonnet-4@20250514
    vertex_project: ${VERTEX_PROJECT}
    vertex_location: ${VERTEX_LOCATION,"us-east5"}
    vertex_credentials: ${VERTEX_CREDENTIALS}

The `vertex_credentials` parameter requires a JSON string containing your Google Cloud service account key. This credential provides the necessary authentication for accessing Vertex AI services. You can obtain this key from the Google Cloud Console by creating a service account with appropriate Vertex AI permissions.

An example of the credential structure follows this format:

    export VERTEX_CREDENTIALS='{"type": "", "project_id": "", "private_key_id": "", "private_key": "", "client_email": "", "client_id": "", "auth_uri": "", "token_uri": "", "auth_provider_x509_cert_url": "", "client_x509_cert_url": "", "universe_domain": ""}'

For comprehensive information about Vertex AI configuration, available models, and advanced features, see the Vertex AI documentation.

### Amazon Bedrock​

Amazon Bedrock provides access to foundation models from various providers through AWS infrastructure. This service offers enterprise features such as private VPC connectivity, AWS IAM integration, and comprehensive logging through AWS CloudTrail.

    model: bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
    aws_region_name: ${AWS_REGION_NAME,"us-east-1"}
    aws_access_key_id: ${AWS_ACCESS_KEY_ID}
    aws_secret_access_key: ${AWS_SECRET_ACCESS_KEY}

The model name includes the `bedrock/` prefix followed by the specific model identifier as defined in the Bedrock service. AWS credentials follow standard AWS authentication patterns, allowing you to use IAM roles, environment variables, or credential files depending on your deployment environment.

For detailed information about Bedrock-specific configuration options, available models, and AWS integration features, see the AWS Bedrock documentation.

### Anthropic​

Anthropic provides the Claude family of models, known for their strong reasoning capabilities and helpful, harmless, and honest behavior. The direct Anthropic API offers the most up-to-date model versions and features.

    model: claude-4
    api_key: ${ANTHROPIC_API_KEY}

The configuration requires only the model name and your Anthropic API key, making it straightforward to integrate Claude models into your agent workflows. Anthropic regularly updates their models with improved capabilities, and the direct API typically provides access to the latest versions first.

For comprehensive details about Anthropic-specific configuration options and model capabilities, see the Anthropic documentation.

### Additional Providers​

LiteLLM supports numerous other providers including Cohere, Hugging Face, Together AI, and many more. Each provider may have specific configuration requirements and capabilities, but the general pattern of specifying model names, endpoints, and authentication credentials remains consistent.

For a complete list of supported providers and their specific configuration requirements, see the LiteLLM providers documentation.

## Prompt Caching​

Agent Mesh supports prompt caching to significantly reduce costs and latency when using LLM providers that support this feature. Prompt caching allows frequently-used content such as system instructions and tool definitions to be cached by the LLM provider, reducing both processing time and token costs on subsequent requests.

### How Prompt Caching Works​

When you configure prompt caching, the system marks specific portions of each request for caching by the LLM provider. These cached portions persist for a provider-defined duration (typically 5 minutes to 1 hour) and can be reused across multiple requests without re-processing. This approach provides substantial cost savings for agents with large system instructions or extensive tool definitions.

### Supported Providers​

The caching mechanism operates transparently through LiteLLM's provider-agnostic interface. Prompt caching support varies by provider:

- Anthropic Claude: Full support with explicit cache control, 90% cost reduction on cache hits
- OpenAI: Automatic caching for content exceeding 1,024 tokens
- Azure OpenAI: Automatic caching following OpenAI behavior
- AWS Bedrock: Native caching support via LiteLLM translation
- Deepseek: Native caching support via LiteLLM translation

Providers without caching support safely ignore cache control markers, ensuring backward compatibility across all providers.

### Cache Strategy Configuration​

Agent Mesh provides three cache strategies that you can configure per model to optimize costs based on usage patterns:

Strategy | Description | Cache Duration | Best For
--- | --- | --- | ---
`"5m"` | 5-minute ephemeral cache | 5 minutes | High-frequency agents (10+ calls/hour)
`"1h"` | 1-hour extended cache | 1 hour | Burst patterns with gaps (3-10 calls/hour)
`"none"` | Disable caching | N/A | Rarely-used agents (less than 2 calls/hour)

The default strategy is `"5m"` when not explicitly specified, providing optimal performance for most use cases without requiring configuration changes.

### Configuration Examples​

Configure prompt caching in your model settings using the `cache_strategy` parameter:

    models:
      # High-frequency orchestrator with 5-minute cache
      planning:
        model: anthropic/claude-sonnet-4-5-20250929
        api_key: ${ANTHROPIC_API_KEY}
        cache_strategy: "5m"
        temperature: 0.1

      # Burst-pattern agent with 1-hour cache
      analysis:
        model: anthropic/claude-sonnet-4-5-20250929
        api_key: ${ANTHROPIC_API_KEY}
        cache_strategy: "1h"
        temperature: 0.7

      # Low-frequency agent with caching disabled
      maintenance:
        model: anthropic/claude-sonnet-4-5-20250929
        api_key: ${ANTHROPIC_API_KEY}
        cache_strategy: "none"

### Cache Strategy Selection Guidelines​

Choose your cache strategy based on agent usage patterns:

Use "5m" strategy when:

- Agent receives 10 or more requests per hour
- Requests arrive in steady streams rather than isolated bursts
- Cache remains warm through continuous use
- Example: Primary orchestrator agents handling user interactions

Use "1h" strategy when:

- Agent receives 3-10 requests per hour in burst patterns
- Gaps between request bursts exceed 5 minutes
- Extended cache duration bridges usage gaps
- Example: Development and testing scenarios, periodic analysis agents

Use "none" strategy when:

- Agent receives fewer than 2 requests per hour
- Cache write premium exceeds potential savings
- System instructions change frequently
- Example: Maintenance agents, backup handlers, rarely-used specialized agents

### What Gets Cached​

The caching system optimizes two primary components of LLM requests:

System Instructions: The complete agent system prompt, including capabilities, guidelines, and any static context. System instructions typically represent the largest cacheable content and provide the most significant cost savings.

Tool Definitions: All tool declarations available to the agent, including peer agent communication tools. Agent Mesh ensures tool order stability through alphabetical sorting, maintaining cache validity across requests.

Conversation history and user messages are never cached, as these components change with each request and represent the unique context for each interaction.

### Cache Invalidation​

The system automatically handles cache invalidation, requiring no manual intervention. When the cache expires or invalidates, the next request writes new cache content, and subsequent requests benefit from the refreshed cache.

## OAuth 2.0 Authentication​

Agent Mesh supports OAuth 2.0 Client Credentials authentication for LLM providers that require OAuth-based authentication instead of traditional API keys. This authentication method provides enhanced security through automatic token management, secure credential handling, and seamless integration with OAuth-enabled LLM endpoints.

### Overview​

The OAuth 2.0 Client Credentials flow is a machine-to-machine authentication method defined in RFC 6749. Agent Mesh handles the complete OAuth lifecycle automatically, including token acquisition, caching, refresh, and injection into LLM requests. This implementation ensures secure and efficient authentication without requiring manual token management.

### Configuration Parameters​

OAuth authentication requires several configuration parameters that you can specify through environment variables and YAML configuration:

Parameter | Required | Description | Default
--- | --- | --- | ---
`oauth_token_url` | Yes | OAuth token endpoint URL | -
`oauth_client_id` | Yes | OAuth client identifier | -
`oauth_client_secret` | Yes | OAuth client secret | -
`oauth_scope` | No | OAuth scope (space-separated) | None
`oauth_ca_cert` | No | Custom CA certificate path for OAuth endpoint | None
`oauth_token_refresh_buffer_seconds` | No | Seconds before expiration to refresh token | 300

### Environment Variables​

Configure OAuth credentials securely using environment variables in your `.env` file:

    # Required OAuth Configuration
    OAUTH_TOKEN_URL="https://auth.example.com/oauth/token"
    OAUTH_CLIENT_ID="your_client_id"
    OAUTH_CLIENT_SECRET="your_client_secret"

    # Optional OAuth Configuration
    OAUTH_SCOPE="llm.read llm.write"
    OAUTH_CA_CERT_PATH="/path/to/ca.crt"
    OAUTH_TOKEN_REFRESH_BUFFER_SECONDS="300"

    # LLM Endpoint Configuration
    OAUTH_LLM_API_BASE="https://api.example.com/v1"

### YAML Configuration​

Configure OAuth-authenticated models in your `shared_config.yaml` file:

    models:
      # OAuth-authenticated planning model
      planning:
        model: ${OAUTH_LLM_PLANNING_MODEL_NAME}
        api_base: ${OAUTH_LLM_API_BASE}

        # OAuth 2.0 Client Credentials configuration
        oauth_token_url: ${OAUTH_TOKEN_URL}
        oauth_client_id: ${OAUTH_CLIENT_ID}
        oauth_client_secret: ${OAUTH_CLIENT_SECRET}
        oauth_scope: ${OAUTH_SCOPE}
        oauth_ca_cert: ${OAUTH_CA_CERT_PATH}
        oauth_token_refresh_buffer_seconds: ${OAUTH_TOKEN_REFRESH_BUFFER_SECONDS, 300}

        parallel_tool_calls: true
        temperature: 0.1

      # OAuth-authenticated general model
      general:
        model: ${OAUTH_LLM_GENERAL_MODEL_NAME}
        api_base: ${OAUTH_LLM_API_BASE}

        # OAuth 2.0 Client Credentials configuration
        oauth_token_url: ${OAUTH_TOKEN_URL}
        oauth_client_id: ${OAUTH_CLIENT_ID}
        oauth_client_secret: ${OAUTH_CLIENT_SECRET}
        oauth_scope: ${OAUTH_SCOPE}
        oauth_ca_cert: ${OAUTH_CA_CERT_PATH}
        oauth_token_refresh_buffer_seconds: ${OAUTH_TOKEN_REFRESH_BUFFER_SECONDS, 300}

### Error Handling and Fallback​

The OAuth system implements robust error handling:

- 4xx Errors: Client configuration errors result in no retries, as these indicate credential or configuration issues
- 5xx Errors: Server errors trigger exponential backoff with jitter for up to 3 retry attempts
- Network Errors: Connection issues trigger exponential backoff with jitter for up to 3 retry attempts

If OAuth authentication fails and an `api_key` is configured in the model settings, the system automatically falls back to API key authentication and logs the OAuth failure. If no fallback is available, the request fails with the OAuth error.

### Security Considerations​

When implementing OAuth authentication, follow these security best practices:

1. Credential Storage: Always store OAuth credentials securely using environment variables, never hardcode them in configuration files
2. Token Caching: Tokens are cached in memory only and never persisted to disk
3. SSL/TLS: Always use HTTPS for OAuth endpoints to protect credentials in transit
4. Custom CA Certificates: Use the `oauth_ca_cert` parameter for private or internal OAuth servers with custom certificate authorities
5. Scope Limitation: Request only the minimal OAuth scopes required for your LLM operations

### Troubleshooting OAuth Issues​

Common OAuth authentication issues and their solutions:

Invalid Client Credentials

    ERROR: OAuth token request failed with status 401: Invalid client credentials

Verify that `OAUTH_CLIENT_ID` and `OAUTH_CLIENT_SECRET` are correct and properly URL-encoded if they contain special characters.

Invalid Scope

    ERROR: OAuth token request failed with status 400: Invalid scope

Verify that `OAUTH_SCOPE` matches your provider's requirements and that scope values are space-separated.

SSL Certificate Issues

    ERROR: OAuth token request failed: SSL certificate verification failed

Set `OAUTH_CA_CERT_PATH` to point to your custom CA certificate file and verify the certificate chain is complete.

Token Refresh Issues

    WARNING: OAuth token request failed (attempt 1/4): Connection timeout

Check network connectivity to the OAuth endpoint, verify the OAuth endpoint URL is correct, and consider increasing timeout values if needed.

### Supported Providers​

This OAuth implementation works with any LLM provider that supports OAuth 2.0 Client Credentials flow, accepts Bearer tokens in the `Authorization` header, and is compatible with LiteLLM's request format. Examples include Azure OpenAI with OAuth-enabled endpoints, custom enterprise LLM deployments, and third-party LLM services with OAuth support.

## Security and SSL/TLS Configuration​

Agent Mesh provides comprehensive security controls for connections to LLM endpoints, allowing you to fine-tune SSL/TLS behavior to meet your organization's security requirements. These settings help ensure secure communication with LLM providers while providing flexibility for various network environments and security policies.

The SSL verification setting controls whether the system validates SSL certificates when connecting to LLM endpoints. Although disabling verification can resolve connectivity issues in development environments, production deployments should always use proper SSL verification to maintain security.

SSL security levels determine the cryptographic standards required for connections. Higher security levels enforce stricter requirements but may cause compatibility issues with older endpoints. The default level provides a good balance between security and compatibility for most deployments.

Custom SSL certificates allow you to specify additional trusted certificate authorities or use self-signed certificates in controlled environments. You can provide certificates either as file paths or as direct certificate content in PEM format.

Parameter | Type | Description | Default
--- | --- | --- | ---
`SSL_VERIFY` | `boolean` | Controls SSL certificate verification for outbound connections. | `true`
`SSL_SECURITY_LEVEL` | `integer` | Sets the SSL security level (higher values enforce stricter checks). | `2`
`SSL_CERT_FILE` | `string` | Path to a custom SSL certificate file to use for verification. | (none)
`SSL_CERTIFICATE` | `string` | Direct content of the SSL certificate (PEM format). | (none)
`DISABLE_AIOHTTP_TRANSPORT` | `boolean` | Flag to disable the use of aiohttp transport for HTTP requests. | `false`
`AIOHTTP_TRUST_ENV` | `boolean` | Flag to enable aiohttp to trust environment proxy settings. | `false`

The HTTP transport settings control how the system makes network requests to LLM endpoints. The aiohttp transport provides efficient asynchronous HTTP handling, although some environments may require disabling it for compatibility reasons. The trust environment setting allows the HTTP client to use proxy settings from environment variables, which can be useful in corporate networks.

For detailed information about each security setting and specific use cases, see the LiteLLM security documentation.

### Example Environment Configuration​

    # SSL Configuration
    SSL_VERIFY=true
    SSL_SECURITY_LEVEL=2
    SSL_CERT_FILE=/path/to/your/certificate.pem
    SSL_CERTIFICATE="-----BEGIN CERTIFICATE-----
    MIIDXTCCAkWgAwIBAg...T2u3V4w5X6y7Z8
    -----END CERTIFICATE-----"

    # HTTP Transport Configuration
    DISABLE_AIOHTTP_TRANSPORT=false
    AIOHTTP_TRUST_ENV=false

This example demonstrates how to configure SSL settings through environment variables, providing a secure foundation for LLM communications while maintaining flexibility for different deployment scenarios. The certificate content should be replaced with your actual certificate data, and file paths should point to your specific certificate locations.