| | |
- builtins.object
-
- EvaluationClient
class EvaluationClient(builtins.object) |
| |
EvaluationClient(base_url: str, auth_url: str = None, client_id: str = None, client_secret: str = None, cert_str: str = None, key_str: str = None, cert_file_path: str = None, key_file_path: str = None, resource_group: str = None, aws_access_key_id: str = None, aws_secret_access_key: str = None, ai_core_client: ai_core_sdk.ai_core_v2_client.AICoreV2Client = None, orchestration_url: str = None, input_object_store_secret_name: str = None, provider_name: str = 'aws')
Base Client for the Evaluations service
|
| |
Methods defined here:
- __init__(self, base_url: str, auth_url: str = None, client_id: str = None, client_secret: str = None, cert_str: str = None, key_str: str = None, cert_file_path: str = None, key_file_path: str = None, resource_group: str = None, aws_access_key_id: str = None, aws_secret_access_key: str = None, ai_core_client: ai_core_sdk.ai_core_v2_client.AICoreV2Client = None, orchestration_url: str = None, input_object_store_secret_name: str = None, provider_name: str = 'aws')
- EvaluationsClient root object to be used for Evaluations.
:param base_url: Base URL of the AI Core instance (must include `/v2` suffix).
:type base_url: str
:param auth_url: Authentication URL used to retrieve access tokens.
:type auth_url: str, optional
:param client_id: OAuth client ID.
:type client_id: str, optional
:param client_secret: OAuth client secret.
:type client_secret: str, optional
:param cert_str: X.509 certificate content as a string.
:type cert_str: str, optional
:param key_str: X.509 private key content as a string.
:type key_str: str, optional
:param cert_file_path: File path to X.509 certificate.
:type cert_file_path: str, optional
:param key_file_path: File path to X.509 private key.
:type key_file_path: str, optional
:param resource_group: Resource group name within the AI Core instance.
:type resource_group: str, optional
:param aws_access_key_id: AWS access key ID.
:type aws_access_key_id: str, optional
:param aws_secret_access_key: AWS secret access key.
:type aws_secret_access_key: str, optional
:param ai_core_client: Pre-configured AI Core client instance.
:type ai_core_client: AICoreV2Client, optional
:param orchestration_url: Pre-existing orchestration deployment URL.
:type orchestration_url: str, optional
:param input_object_store_secret_name: Name of input object store secret.
:type input_object_store_secret_name: str, optional
:param provider_name: Hyperscaler provider name (e.g., "aws").
:type provider_name: str, optional
:raises ValueError: If required hyperscaler provider parameters are missing.
- __repr__(self)
- Return repr(self).
- create_or_update_object_store_secret(self, *, context, secret_body: dict, is_default: bool, result_key: str, attr_name: str, creator_mapping: dict, replace_existing: bool, result: dict)
- evaluate(self, evaluation_configs: List[gen_ai_hub.evaluations.models.evaluation_config.EvaluationConfig]) -> List[gen_ai_hub.evaluations.models.evaluation_run.EvaluationRun]
- Main evaluate function to create the Evaluation job
Parameters:
evaluation_configs(List[EvaluationConfig]): A list of one or more of the EvaluationConfig objects
Returns:
List[EvaluationRun]: A list of EvaluationRun objects, one for each EvaluationConfig provided.
- get_system_supported_metrics(self) -> List[str]
- helper method to get the list of all supported metric ids
- list_available_models(self)
- Method to list all the available llm models
- resolve_orchestration_deployment_url(self) -> str
- Resolves the orchestration deployment URL.
For non-default resource groups, creates a new deployment.
For default resource group, attempts to discover existing deployment
with the default config name using the orchestration service,
or creates one if not found.
:return: The orchestration deployment URL.
:rtype: str
- setup(self, input_secret_body: dict | None = None, default_secret_body: dict | None = None, replace_existing: bool = False)
- One time setup function which does object store secrets creation
and orchestration deployment url creation if not provided.
- validate_secret_type(self, secret_type: str, creator_mapping: dict)
Static methods defined here:
- from_env(profile_name: str = None, **kwargs)
- Alternative way to create an EvaluationClient object.
Parameter resolution precedence:
1. Explicit keyword arguments
2. Environment variables
3. Configuration file
4. VCAP_SERVICES environment variable
:param profile_name: Profile name defined in configuration.
:type profile_name: str, optional
:param kwargs: Additional parameters passed to constructor.
:return: Configured EvaluationClient instance.
:rtype: EvaluationClient
Data descriptors defined here:
- __dict__
- dictionary for instance variables (if defined)
- __weakref__
- list of weak references to the object (if defined)
| |