| | |
- build_accumulated_config(evaluation_configs_data: List[gen_ai_hub.evaluations._internal._models._EvaluationConfigData], has_mixed_config_types: bool = False) -> Tuple[Union[List[gen_ai_hub.evaluations._internal._models._EvaluationConfigData], gen_ai_hub.evaluations._internal._models._EvaluationConfigData], bool, bool]
- Build accumulated configuration data and determine execution flow strategy.
Analyzes evaluation configurations to determine whether they can be executed
as a single job (if datasets and metrics match) or require multiple executions.
Also determines if artifacts can be reused across executions.
:param evaluation_configs_data: List of extracted evaluation configuration data objects
:type evaluation_configs_data: List[_EvaluationConfigData]
:param has_mixed_config_types: Whether evaluation configs have mixed types (llm+template and orchestration_registry)
:type has_mixed_config_types: bool
:return: Tuple containing:
- accumulated_config_data: Either a single accumulated config or list of configs
- single_execution_flow: True if all configs can be executed as one job
- reusable_artifact: True if dataset artifact can be reused across executions
:rtype: Tuple[Union[List[_EvaluationConfigData], _EvaluationConfigData], bool, bool]
- extract_config_data(evaluation_configs: List[gen_ai_hub.evaluations.models.evaluation_config.EvaluationConfig], ai_core_client: ai_core_sdk.ai_core_v2_client.AICoreV2Client, object_store_credentials: gen_ai_hub.evaluations._internal._models._AWSObjectStoreData, resource_group: str, gen_ai_hub_proxy_client: gen_ai_hub.proxy.gen_ai_hub_proxy.client.GenAIHubProxyClient, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> List[gen_ai_hub.evaluations._internal._models._EvaluationConfigData]
- Extract configuration data from user-provided evaluation configs.
This function processes evaluation configurations to extract orchestration config,
dataset data, metric templates, and variable mappings for each configuration.
:param evaluation_configs: List of evaluation configuration objects to process
:type evaluation_configs: List[EvaluationConfig]
:param ai_core_client: AI Core V2 client for API interactions
:type ai_core_client: AICoreV2Client
:param object_store_credentials: Credentials for accessing object storage (AWS S3)
:type object_store_credentials: _AWSObjectStoreData
:param resource_group: AI Core resource group name
:type resource_group: str
:param gen_ai_hub_proxy_client: GenAI Hub proxy client for orchestration operations
:type gen_ai_hub_proxy_client: GenAIHubProxyClient
:param error_collector: Collector for validation errors
:type error_collector: ValidationCollector
:return: List of extracted evaluation configuration data objects
:rtype: List[_EvaluationConfigData]
|