| | |
- build_model_versions_map(model_list) -> Dict[str, List[str]]
- Builds a map of model names to their versions.
- check_if_content_filter_provider_supported(orch_config: dict, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> bool
- Validates if all filters in the filtering module configuration are of supported types.
:param orch_config: Orchestration configuration dictionary.
:type orch_config: dict
:param error_collector: ValidationCollector instance for collecting validation errors.
:type error_collector: ValidationCollector
:return: True if all filters are supported or no filtering is configured, False otherwise.
:rtype: bool
- check_if_metric_is_defined(metrics: List[str], metric_templates: List[dict], error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> None
- count_user_prompts_from_template_list(template_list) -> int
- create_custom_metric_name(custom_metric_config: dict, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> str
- Creates a custom metric name based on the provided custom metric configuration.
:param custom_metric_config: Dictionary containing metric configuration.
:param error_collector: ValidationCollector instance for collecting validation errors.
:return: A string representing the custom metric name.
:raises ValidationError: If required fields are missing or invalid.
- create_model_versions_map_from_configuration_param_bindings(param_bindings, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> Tuple[Dict[str, List[str]], Optional[str]]
- create_model_versions_map_from_custom_metric_config(custom_metric_config_data) -> Dict[str, List[str]]
- create_model_versions_map_from_orch_configs(orchestration_configs_data: List[dict], error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> Optional[Dict[str, List[str]]]
- extract_dataset_columns(template_variables) -> List[str]
- Extracts column names from the template variables provided. If the value is a list, extracts column names from the first rows; else, extracts from the template_variables directly.
- extract_metrics_variables(metric_templates, metric_name: str = None) -> Set
- Extracts unique set of variables from the 'variables' key.
- flatten_prompt_configuration(prompt_config: dict) -> str
- Flatten a nested prompt configuration dictionary into a readable string.
- get_accumulated_config_data(evaluation_configs_data: List[gen_ai_hub.evaluations._internal._models._EvaluationConfigData]) -> gen_ai_hub.evaluations._internal._models._EvaluationConfigData
- get_custom_metric_ids_from_input(custom_metric_config_data: List[dict], error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> list[str]
- Retrieve custom metric ids from the file data provided by the user.
:param custom_metric_config_data: List of dictionaries containing custom metric definitions.
:type custom_metric_config_data: List[dict]
:param error_collector: ValidationCollector instance for collecting validation errors.
:type error_collector: ValidationCollector
:return: List of custom metric ids.
:rtype: list[str]
- get_defaults(orchestration_configuration: dict) -> dict
- Returns the default field from the orchestration configuration.
- get_filter_config(orch_config: dict) -> dict
- Extract the filtering configuration from the orchestration config.
:param orch_config: Orchestration configuration dictionary.
:type orch_config: dict
:return: Filtering configuration dictionary, or empty dict if not present.
:rtype: dict
- get_grounding_config_from_orch_config(orch_config) -> dict
- Extracts the grounding configuration from the orchestration configuration.
Args:
orch_config (dict): Orchestration configuration.
Returns:
dict: Grounding configuration if present, otherwise an empty dictionary.
- get_grounding_output_param_key(orch_config: dict) -> str
- Determines the correct key to extract the grounding output parameter based on the API version.
Args:
orch_config (dict): Orchestration configuration.
Returns:
str: The key to extract the grounding output parameter.
- get_mapped_value_if_exists(key, mapping_keys, variable_mapping, dataset_columns) -> str
- Gets the first valid mapped value from the list of keys if it exists in variable mapping, else returns the first key
- get_prompt_variables_from_orch_config(orch_config: dict) -> Set[str]
- get_template_list_from_orch_config(orch_config) -> List
- handle_json_schema_match(metrics: list[str], dataset_data: list, variable_mapping: dict, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> None
- handle_language_match(metrics: list, dataset_data: list, variable_mapping: dict, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> None
- handle_missing_dependent_variables_in_dataset(dataset_data: List[dict], metrics: list[str], metric_templates: list[dict], variable_mapping: dict, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> None
- validates whether all the dependent variables for the metrics list are either directly present as columns in dataset or a variable mapping is provided
Args:
dataset_data (List[dict]): Dataset rows to validate (list of row dictionaries)
metrics (list[str]): List of metrics provided in the input configuration.
metric_templates (list[dict]): Metric templates information resolved from Metric Management Service
variable_mapping (dict): The variable mapping provided in the input configuration.
Raises:
ValidationError: If any dependent variable is missing in the dataset and the variable mapping is invalid for that metric.
- handle_reference_missing_rows(dataset_data: List[dict], variable_mapping: dict, metrics: list, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> None
- validates whether the reference columns in the rows are missing in the dataset for all metrics and for each individual metrics
Args:
dataset_data (List[dict]): Dataset rows to validate (list of row dictionaries)
variable_mapping (dict): The variable mapping provided in the input configuration.
metrics (list): List of metrics provided in the input configuration.
Raises:
ValidationError: If any required variable mapping is invalid or the default column does not exist in the dataset.
- has_filter_key(orch_config: dict) -> bool
- Check if the 'filtering' key is present in the orchestration config.
:param orch_config: Orchestration configuration dictionary.
:type orch_config: dict
:return: True if filtering key is present, False otherwise.
:rtype: bool
- is_value_in_json(value, name, data: dict[str, str]) -> bool
- Check if a value matches a name or exists in a mapping dictionary.
:param value: The value to search for.
:type value: Any
:param name: The name to compare against.
:type name: str
:param data: Dictionary to search in (keys or values).
:type data: dict[str, str]
:return: True if value matches name or is found in data, False otherwise.
:rtype: bool
- list_prompt_variables(format_string: str) -> list[str]
- Get all fields (parameters) of the form {{ ?param_name }} from the template.
Optionally return the raw field names without stripping spaces and '?'.
- parse_model_filter_list(param, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> List
- Parse and return model filter list from a param.
- populate_dataset_data_if_data_missing(dataset_data: list, variable_mapped_key, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> None
- Validates and Populates the dataset_data with missing data fields if golden truth is present and throws error if dataset is partially filled
- populate_dataset_data_if_individual_metric_reference_provided(dataset_data: List[dict], variable_mapping: dict, metrics: list, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> None
- populates reference value across all rows of dataset_data if individual metric reference is provided
and is different than all metrics reference provided. This population happens if the provided reference is a golden instance
Args:
dataset_data (List[dict]): Dataset rows to validate (list of row dictionaries)
variable_mapping (dict): The variable mapping provided in the input configuration.
metrics (list): List of metrics provided in the input configuration.
- populate_dataset_data_if_single_reference_provided(dataset_data: List[dict], variable_mapping: dict, collector) -> None
- Populates the dataset_data with missing reference column entries across rows of
dataset_data for only all metrics case where a golden reference is present
Args:
dataset_data (List[dict]): Dataset rows to validate (list of row dictionaries)
variable_mapping (dict): The variable mapping provided in the input configuration.
- populate_dataset_data_if_single_schema_provided(dataset_data, variable_mapping, collector) -> None
- Populates the dataset_data with missing json schema column entries across rows of dataset_data
- remove_filter_metrics_if_provider_not_supported(orchestration_config_data: List[dict], metrics: List[str], error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> None
- Removes content filter-related metric IDs from the metrics list if the content filter
provider is not supported for any of the runs in orchestration_config_data.
- resolve_orchestration_config_v2(template_data: List[gen_ai_hub.prompt_registry.models.prompt_template.PromptTemplate], llm: gen_ai_hub.orchestration_v2.models.llm_model_details.LLMModelDetails) -> dict
- select_model_details_randomly(orchestration_config_data: List[dict], error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> Optional[Tuple[str, str]]
- Selects at random, model name and version from the list of model names and versions provided by the users run data.
- set_model_details_from_run_configs(orch_config) -> Optional[Tuple[str, str]]
- Sets the model name and version from the run data if available.
If not available, it returns None.
- update_artifact_dict(artifact_reference: gen_ai_hub.evaluations.models.artifact_source.ArtifactSource, artifact_dict_count: dict) -> None
- update_test_orch_config(model_name, model_version, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> Optional[dict]
- update_variable_mapping(variable_mapping: dict, prefix_key: str, variable_mapping_dict: dict) -> dict
- validate_all_metrics_mapping(variable_mapping: dict, dataset_columns: list, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> None
- Validates the variable mapping for 'all_metrics' with a zero-tolerance failure threshold.
Args:
variable_mapping (dict): The variable mapping provided in the input configuration.
dataset_columns (list): List of column names in the dataset.
Raises:
ValidationError: If any 'all_metrics' mapping is invalid or the direct column does not exist in the dataset.
- validate_individual_custom_metrics(variable_mapping: dict, dataset_columns: list, custom_metric_ids: list, custom_metric_variables: set, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> None
- Validates the variable mapping for any metric mapping with a zero-tolerance failure threshold.
- validate_individual_metrics(metrics: list[str], variable_mapping: dict, dataset_columns: list, metric_dependent_variables: set, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> None
- Validates the variable mapping for any metric mapping with a zero-tolerance failure threshold.
Args:
metrics (list): List of metrics provided in the input configuration.
variable_mapping (dict): The variable mapping provided in the input configuration.
dataset_columns (list): List of column names in the dataset.
metric_dependent_variables (set): Set of dependent variables for all metrics
Raises:
ValidationError: If any metric mapping is invalid or the direct column does not exist in the dataset.
- validate_language_code_and_data_population(dataset_data: list, variable_mapping: dict, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> None
- Populates the dataset_data with missing language column entries across rows of dataset_data
- validate_metric_name(metric: str, all_supported_metrics: List, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> None
- Validates if metrics name is not empty and the value actually exists in the list of supported metrics
- validate_metrics(metrics: List[str], metric_templates: List[dict], orchestration_config_data: List[dict], error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> None
- Validates if metrics list is empty or metric name is invalid
- validate_prompts_in_templating_module(orchestration_config_data: List[dict], metric: str, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> None
- Checks whether the templating config provided in the Orchestration Config has exactly one user prompt
- validate_variable_mapping_of_metrics(metrics: list[str], metric_templates: list[dict], dataset_data: List[dict], variable_mapping: dict, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> None
- Validates variable mapping of metrics with tolerance to zero failure threshold
Args:
metrics: List of metrics provided in the input config
metric_templates (list[dict]): Metric templates information resolved from Metric Management Service
dataset_data: Dataset rows to validate
variable_mapping: variable mapping provided in input config
Returns:
Validates and throws validation error even if one variable mapping related to metrics is invalid.
- validate_variable_mapping_of_prompts(orchestration_config_data: list, dataset_data: List[dict], variable_mapping: dict, error_collector: gen_ai_hub.evaluations.helpers.collector.ValidationCollector) -> None
- Validates the variable mapping for prompts with a zero-tolerance failure threshold.
Args:
orchestration_config_data (list): Orchestration run configuration
dataset_data (dict): Dataset rows to validate
variable_mapping (dict): The variable mapping provided in the input configuration.
Raises:
ValidationError: If any prompts variable mapping is invalid or does not exist in the dataset.
|