| | |
- extract_deployment_id(orch_url) -> str
- fetch_and_validate_orchestration_config(ai_core_client: ai_core_sdk.ai_core_v2_client.AICoreV2Client, configuration_id: str, orchestration_config_data: List[dict], resource_group: str, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector)
- validate_config_data_collection(accumulated_config_data: Union[List[gen_ai_hub.evaluations._internal._models._EvaluationConfigData], gen_ai_hub.evaluations._internal._models._EvaluationConfigData], error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector)
- wrapper function to perform validation of fetched config data in case of single vs multiple executions flow
- validate_filtered_models(configuration_param_bindings, orchestration_config_data: List[dict], error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector)
- validate_input_config(orchestration_config_data: List[dict], metrics: List[str], metric_templates: List[dict], error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector)
- Validates the input parameters of run data and metrics
- validate_merged_config_data(evaluation_config_data: gen_ai_hub.evaluations._internal._models._EvaluationConfigData, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector)
- handles the validation of config provided from the user
- validate_orchestration_configuration(orchestration_config_data: List[dict], error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector)
- Validates the Orchestration configuration provided by user
- validate_orchestration_url(evaluation_config_data: gen_ai_hub.evaluations._internal._models._EvaluationConfigData, orchestration_url: str, ai_core_client: ai_core_sdk.ai_core_v2_client.AICoreV2Client, resource_group: str, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector, proxy_client=None)
- Validates if the orchestration deployment url provided via config resides in same
resourceGroup as workload or not. Also validates if url is valid
and orchestration deployment is not in terminal state
- validate_orchestration_url_across_configs(accumulated_config_data: Union[List[gen_ai_hub.evaluations._internal._models._EvaluationConfigData], gen_ai_hub.evaluations._internal._models._EvaluationConfigData], orchestration_url: str, ai_core_client: ai_core_sdk.ai_core_v2_client.AICoreV2Client, resource_group: str, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector, proxy_client=None)
- wrapper function to perform validation of fetched config data in case of single vs multiple executions flow
- validate_variable_mapping_with_input_config(orchestration_config_data: List[dict], dataset_data: dict, variable_mapping: dict, metrics: List[str], metric_templates: List[dict], error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector)
- Validates all the required variable mappings provided in input config with a zero-tolerance failure threshold.
Args:
orchestration_config_data(list): Orchestration run configuration
dataset_data (dict): Dataset rows to validate
variable_mapping (dict): The variable mapping provided in the input configuration.
metrics (list[str]): List of metrics provided in the input configuration.
metric_templates (list[dict]): Metric templates information resolved from Metric Management Service
error_collector (ValidationCollector): To accumulate the errors occurred during the process
Raises:
ValidationError: If any required variable mapping is invalid or the default column does not exist in the dataset.
|