Model Evaluation¶

Model evaluation preview components.

Components:

`DetectDataBiasOp`(gcp_resources, ...[, ...])	Detects data bias metrics in a dataset.
`DetectModelBiasOp`(gcp_resources, ...[, ...])	Detects bias metrics from a model's predictions.
`ModelEvaluationFeatureAttributionOp`(...[, ...])	Compute feature attribution on a trained model's batch explanation results.

preview.model_evaluation.DetectDataBiasOp(gcp_resources: dsl.OutputPath(str), data_bias_metrics: dsl.Output[system.Artifact], project: str, target_field_name: str, bias_configs: list, location: str = 'us-central1', dataset_format: str = 'jsonl', dataset_storage_source_uris: list = [], dataset: dsl.Input[google.VertexDataset] = None, columns: list = [], encryption_spec_key_name: str = '')¶

Detects data bias metrics in a dataset.

Creates a Dataflow job with Apache Beam to category each data point in the dataset to the corresponding bucket based on bias configs, then compute data bias metrics for the dataset.

Parameters¶

project: str¶

Project to run data bias detection.

location: str = 'us-central1'¶

Location for running data bias detection.

target_field_name: str¶

The full name path of the features target field in the predictions file. Formatted to be able to find nested columns, delimited by .. Alternatively referred to as the ground truth (or ground_truth_column) field.

bias_configs: list¶

A list of google.cloud.aiplatform_v1beta1.types.ModelEvaluation.BiasConfig. When provided, compute data bias metrics for each defined slice. Below is an example of how to format this input. 1: First, create a BiasConfig.

from google.cloud.aiplatform_v1beta1.types.ModelEvaluation import BiasConfig

from google.cloud.aiplatform_v1.types.ModelEvaluationSlice.Slice import SliceSpec

from google.cloud.aiplatform_v1.types.ModelEvaluationSlice.Slice.SliceSpec import SliceConfig

``bias_config = BiasConfig(bias_slices=SliceSpec(configs={
’feature_a’: SliceConfig(SliceSpec.Value(string_value= ‘label_a’) ) }))``

2: Create a list to store the bias configs into.: bias_configs = []
3: Format each BiasConfig into a JSON or Dict.: bias_config_json = json_format.MessageToJson(bias_config or bias_config_dict = json_format.MessageToDict(bias_config).
4: Combine each bias_config JSON into a list.: bias_configs.append(bias_config_json)
5: Finally, pass bias_configs as an parameter for this component.: DetectDataBiasOp(bias_configs=bias_configs)

dataset_format: str = 'jsonl'¶

The file format for the dataset. jsonl and csv are the currently allowed formats.

dataset_storage_source_uris: list = []¶

Google Cloud Storage URI(-s) to unmanaged test datasets.``jsonl`` and csv is currently allowed format. If dataset is also provided, this field will be overriden by the provided Vertex Dataset.

dataset: dsl.Input[google.VertexDataset] = None¶

A google.VertexDataset artifact of the dataset. If dataset_gcs_source is also provided, this Vertex Dataset argument will override the GCS source.

encryption_spec_key_name: str = ''¶

Customer-managed encryption key options for the Dataflow. If this is set, then all resources created by the Dataflow will be encrypted with the provided encryption key. Has the form: projects/my-project/locations/my-location/keyRings/my-kr/cryptoKeys/my-key. The key needs to be in the same region as where the compute resource is created.

Returns¶

data_bias_metrics: dsl.Output[system.Artifact]: Artifact tracking the data bias detection output.
gcp_resources: dsl.OutputPath(str): Serialized gcp_resources proto tracking the Dataflow job. For more details, see https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/proto/README.md.

preview.model_evaluation.DetectModelBiasOp(gcp_resources: dsl.OutputPath(str), bias_model_metrics: dsl.Output[system.Artifact], project: str, target_field_name: str, bias_configs: list, location: str = 'us-central1', predictions_format: str = 'jsonl', predictions_gcs_source: dsl.Input[system.Artifact] = None, predictions_bigquery_source: dsl.Input[google.BQTable] = None, thresholds: list = [0.5], encryption_spec_key_name: str = '')¶

Detects bias metrics from a model’s predictions.

Creates a Dataflow job with Apache Beam to category each data point to the corresponding bucket based on bias configs and predictions, then compute model bias metrics for classification problems.

Parameters¶

project: str¶

Project to run data bias detection.

location: str = 'us-central1'¶

Location for running data bias detection.

target_field_name: str¶

The full name path of the features target field in the predictions file. Formatted to be able to find nested columns, delimited by .. Alternatively referred to as the ground truth (or ground_truth_column) field.

predictions_format: str = 'jsonl'¶

The file format for the batch prediction results. jsonl is the only currently allow format.

predictions_gcs_source: dsl.Input[system.Artifact] = None¶

An artifact with its URI pointing toward a GCS directory with prediction or explanation files to be used for this evaluation. For prediction results, the files should be named “prediction.results-”. For explanation results, the files should be named “explanation.results-“.

predictions_bigquery_source: dsl.Input[google.BQTable] = None¶

BigQuery table with prediction or explanation data to be used for this evaluation. For prediction results, the table column should be named “predicted_*”.

bias_configs: list¶

A list of google.cloud.aiplatform_v1beta1.types.ModelEvaluation.BiasConfig. When provided, compute model bias metrics for each defined slice. Below is an example of how to format this input. 1: First, create a BiasConfig.

from google.cloud.aiplatform_v1beta1.types.ModelEvaluation import BiasConfig

from google.cloud.aiplatform_v1.types.ModelEvaluationSlice.Slice import SliceSpec

from google.cloud.aiplatform_v1.types.ModelEvaluationSlice.Slice.SliceSpec import SliceConfig

``bias_config = BiasConfig(bias_slices=SliceSpec(configs={
’feature_a’: SliceConfig(SliceSpec.Value(string_value= ‘label_a’) ) }))``

2: Create a list to store the bias configs into.: bias_configs = []
3: Format each BiasConfig into a JSON or Dict.: bias_config_json = json_format.MessageToJson(bias_config or bias_config_dict = json_format.MessageToDict(bias_config)
4: Combine each bias_config JSON into a list.: bias_configs.append(bias_config_json)
5: Finally, pass bias_configs as an parameter for this component.: DetectModelBiasOp(bias_configs=bias_configs)

thresholds: list = [0.5]¶

A list of float values to be used as prediction decision thresholds.

encryption_spec_key_name: str = ''¶

Customer-managed encryption key options for the Dataflow. If this is set, then all resources created by the Dataflow will be encrypted with the provided encryption key. Has the form: projects/my-project/locations/my-location/keyRings/my-kr/cryptoKeys/my-key. The key needs to be in the same region as where the compute resource is created.

Returns¶

bias_model_metrics: dsl.Output[system.Artifact]: Artifact tracking the model bias detection output.
gcp_resources: dsl.OutputPath(str): Serialized gcp_resources proto tracking the Dataflow job. For more details, see https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/proto/README.md.

preview.model_evaluation.ModelEvaluationFeatureAttributionOp(gcp_resources: dsl.OutputPath(str), feature_attributions: dsl.Output[system.Metrics], project: str, problem_type: str, location: str = 'us-central1', predictions_format: str = 'jsonl', predictions_gcs_source: dsl.Input[system.Artifact] = None, predictions_bigquery_source: dsl.Input[google.BQTable] = None, dataflow_service_account: str = '', dataflow_disk_size_gb: int = 50, dataflow_machine_type: str = 'n1-standard-4', dataflow_workers_num: int = 1, dataflow_max_workers_num: int = 5, dataflow_subnetwork: str = '', dataflow_use_public_ips: bool = True, encryption_spec_key_name: str = '', force_runner_mode: str = '')¶

Compute feature attribution on a trained model’s batch explanation results.

Creates a dataflow job with Apache Beam and TFMA to compute feature attributions. Will compute feature attribution for every target label if possible, typically possible for AutoML Classification models.

Parameters¶

project: str¶: Project to run feature attribution container.
location: str = 'us-central1'¶: Location running feature attribution. If not set, defaulted to us-central1.
problem_type: str¶: Problem type of the pipeline: one of classification,
forecasting. : regression and
predictions_format: str = 'jsonl'¶: The file format for the batch prediction results. jsonl, csv, and bigquery are the allowed formats, from Vertex Batch Prediction. If not set, defaulted to jsonl.
predictions_gcs_source: dsl.Input[system.Artifact] = None¶: An artifact with its URI pointing toward a GCS directory with prediction or explanation files to be used for this evaluation. For prediction results, the files should be named “prediction.results-” or “predictions_”. For explanation results, the files should be named “explanation.results-“.
predictions_bigquery_source: dsl.Input[google.BQTable] = None¶: BigQuery table with prediction or explanation data to be used for this evaluation. For prediction results, the table column should be named “predicted_*”.
dataflow_service_account: str = ''¶: Service account to run the dataflow job. If not set, dataflow will use the default worker service account. For more details, see https://cloud.google.com/dataflow/docs/concepts/security-and-permissions#default_worker_service_account
dataflow_disk_size_gb: int = 50¶: The disk size (in GB) of the machine executing the evaluation run. If not set, defaulted to 50.
dataflow_machine_type: str = 'n1-standard-4'¶: The machine type executing the evaluation run. If not set, defaulted to n1-standard-4.
dataflow_workers_num: int = 1¶: The number of workers executing the evaluation run. If not set, defaulted to 10.
dataflow_max_workers_num: int = 5¶: The max number of workers executing the evaluation run. If not set, defaulted to 25.
dataflow_subnetwork: str = ''¶: Dataflow’s fully qualified subnetwork name, when empty the default subnetwork will be used. More details: https://cloud.google.com/dataflow/docs/guides/specifying-networks#example_network_and_subnetwork_specifications
dataflow_use_public_ips: bool = True¶: Specifies whether Dataflow workers use public IP addresses.
encryption_spec_key_name: str = ''¶: Customer-managed encryption key for the Dataflow job. If this is set, then all resources created by the Dataflow job will be encrypted with the provided encryption key.
force_runner_mode: str = ''¶: Flag to choose Beam runner. Valid options are DirectRunner and Dataflow.

Returns¶

gcs_output_directory: Unknown: JsonArray of the downsampled dataset GCS output.
bigquery_output_table: Unknown: String of the downsampled dataset BigQuery output.
gcp_resources: dsl.OutputPath(str): Serialized gcp_resources proto tracking the dataflow job. For more details, see https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/proto/README.md.