google_cloud_pipeline_components.experimental.evaluation module

Google Cloud Pipeline Model Evaluation components.

google_cloud_pipeline_components.experimental.evaluation.EvaluationDataSamplerOp(gcp_resources: <kfp.components.types.type_annotations.OutputPath object at 0x7f9f6d847b20>, bigquery_output_table: <kfp.components.types.type_annotations.OutputPath object at 0x7f9f6d847490>, gcs_output_directory: <kfp.components.types.type_annotations.OutputPath object at 0x7f9f6d847a90>, project: str, root_dir: str, location: str = 'us-central1', gcs_source_uris: list = [], bigquery_source_uri: str = '', instances_format: str = 'jsonl', sample_size: int = 10000, dataflow_service_account: str = '', dataflow_subnetwork: str = '', dataflow_use_public_ips: bool = True, encryption_spec_key_name: str = '')

Randomly downsamples an input dataset to a specified size.

Used for computing Vertex XAI feature attributions for AutoML Tables and custom models. Creates a Dataflow job with Apache Beam to downsample the dataset.

Args:

project (str): Project to retrieve dataset from. location (Optional[str]): Location to retrieve dataset from. If not set,

defaulted to us-central1.

root_dir (str): The GCS directory for keeping staging files. A random

subdirectory will be created under the directory to keep job info for resuming the job in case of failure.

gcs_source_uris (Sequence[str]): Google Cloud Storage URI(-s) to your

instances to run data sampler on. They must match instances_format. May contain wildcards. For more information on wildcards, see

https://cloud.google.com/storage/docs/gsutil/addlhelp/WildcardNames.

bigquery_source_uri (Optional[str]): Google BigQuery Table URI to your

instances to run data sampler on.

instances_format (Optional[str]): The format in which instances are given,

must be one of the model’s supported input storage formats. If not set, default to “jsonl”.

sample_size (Optional[int]): Sample size of the randomly sampled dataset.

10k by default.

dataflow_service_account (Optional[str]): Service account to run the

dataflow job. If not set, dataflow will use the default worker service account. For more details, see https://cloud.google.com/dataflow/docs/concepts/security-and-permissions#default_worker_service_account

dataflow_subnetwork (Optional[str]): Dataflow’s fully qualified subnetwork

name, when empty the default subnetwork will be used. More details:: https://cloud.google.com/dataflow/docs/guides/specifying-networks#example_network_and_subnetwork_specifications

dataflow_use_public_ips (Optional[bool]): Specifies whether Dataflow workers

use public IP addresses.

encryption_spec_key_name (Optional[str]): Customer-managed encryption key

for the Dataflow job. If this is set, then all resources created by the Dataflow job will be encrypted with the provided encryption key.

Returns:

gcs_output_directory (JsonArray): JsonArray of the downsampled dataset GCS

output.

bigquery_output_table (str): String of the downsampled dataset BigQuery

output.

gcp_resources (str): Serialized gcp_resources proto tracking the dataflow

job.

For more details, see https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/proto/README.md.

google_cloud_pipeline_components.experimental.evaluation.ModelEvaluationClassificationOp(gcp_resources: <kfp.components.types.type_annotations.OutputPath object at 0x7f9f6d847700>, evaluation_metrics: ~google_cloud_pipeline_components.types.artifact_types._ClassificationMetrics[~google_cloud_pipeline_components.types.artifact_types._ClassificationMetrics], project: str, root_dir: str, target_field_name: str, model: ~google_cloud_pipeline_components.types.artifact_types.VertexModel[~google_cloud_pipeline_components.types.artifact_types.VertexModel] | None = None, location: str = 'us-central1', predictions_format: str = 'jsonl', predictions_gcs_source: ~kfp.components.types.artifact_types.Artifact[~kfp.components.types.artifact_types.Artifact] | None = None, predictions_bigquery_source: ~google_cloud_pipeline_components.types.artifact_types.BQTable[~google_cloud_pipeline_components.types.artifact_types.BQTable] | None = None, ground_truth_format: str = 'jsonl', ground_truth_gcs_source: list = [], ground_truth_bigquery_source: str = '', classification_type: str = 'multiclass', class_labels: list = [], prediction_score_column: str = '', prediction_label_column: str = '', slicing_specs: list = [], positive_classes: list = [], dataflow_service_account: str = '', dataflow_disk_size: int = 50, dataflow_machine_type: str = 'n1-standard-4', dataflow_workers_num: int = 1, dataflow_max_workers_num: int = 5, dataflow_subnetwork: str = '', dataflow_use_public_ips: bool = True, encryption_spec_key_name: str = '')

Computes a google.ClassificationMetrics Artifact, containing evaluation metrics given a model’s prediction results.

Creates a dataflow job with Apache Beam and TFMA to compute evaluation metrics. Supports mutliclass classification evaluation for tabular, image, video, and text data.

Args:

project (str): Project to run evaluation container. location (Optional[str]): Location for running the evaluation. If not set,

defaulted to us-central1.

root_dir (str): The GCS directory for keeping staging files. A random

subdirectory will be created under the directory to keep job info for resuming the job in case of failure.

predictions_format (Optional[str]): The file format for the batch prediction

results. jsonl, csv, and bigquery are the allowed formats, from Vertex Batch Prediction. If not set, defaulted to jsonl.

predictions_gcs_source (Optional[system.Artifact]): An artifact with its URI

pointing toward a GCS directory with prediction or explanation files to be used for this evaluation. For prediction results, the files should be named “prediction.results-” or “predictions_”. For explanation results, the files should be named “explanation.results-“.

predictions_bigquery_source (Optional[google.BQTable]): BigQuery table with

prediction or explanation data to be used for this evaluation. For prediction results, the table column should be named “predicted_*”.

ground_truth_format (Optional[str]): Required for custom tabular and non

tabular data. The file format for the ground truth files. jsonl, csv, and bigquery are the allowed formats. If not set, defaulted to jsonl.

ground_truth_gcs_source (Optional[Sequence[str]]): Required for custom

tabular and non tabular data. The GCS uris representing where the ground truth is located. Used to provide ground truth for each prediction instance when they are not part of the batch prediction jobs prediction instance.

ground_truth_bigquery_source (Optional[str]): Required for custom tabular.

The BigQuery table uri representing where the ground truth is located. Used to provide ground truth for each prediction instance when they are not part of the batch prediction jobs prediction instance.

classification_type (Optional[str]): The type of classification problem,

either multiclass or multilabel. If not set, defaulted to multiclass.

class_labels (Optional[Sequence[str]]): The list of class names for the

target_field_name, in the same order they appear in the batch predictions jobs predictions output file. For instance, if the values of target_field_name could be either 1 or 0, and the predictions output contains [“1”, “0”] for the prediction_label_column, then the class_labels input will be [“1”, “0”]. If not set, defaulted to the classes found in the prediction_label_column in the batch prediction jobs predictions file.

target_field_name (str): The full name path of the features target field in

the predictions file. Formatted to be able to find nested columns, delimited by .. Alternatively referred to as the ground truth (or ground_truth_column) field.

model (Optional[google.VertexModel]): The Model used for predictions job.

Must share the same ancestor Location.

prediction_score_column (Optional[str]): The column name of the field

containing batch prediction scores. Formatted to be able to find nested columns, delimited by .. If not set, defaulted to prediction.scores for classification.

prediction_label_column (Optional[str]): The column name of the field

containing classes the model is scoring. Formatted to be able to find nested columns, delimited by .. If not set, defaulted to prediction.classes for classification.

slicing_specs (Optional[Sequence[SlicingSpec]]): Optional. List of

google.cloud.aiplatform_v1.types.ModelEvaluationSlice.SlicingSpec. When provided, compute metrics for each defined slice. Below is an example of how to format this input. 1: First, create a SlicingSpec. ```from

google.cloud.aiplatform_v1.types.ModelEvaluationSlice.Slice import SliceSpec from google.cloud.aiplatform_v1.types.ModelEvaluationSlice.Slice.SliceSpec import SliceConfig slicing_spec = SliceSpec(configs={ ‘feature_a’: SliceConfig(SliceSpec.Value(string_value=’label_a’) ) })```

2: Create a list to store the slicing specs into. slicing_specs = []. 3: Format each SlicingSpec into a JSON or Dict. `slicing_spec_json =

json_format.MessageToJson(slicing_spec)` or `slicing_spec_dict = json_format.MessageToDict(slicing_spec).

4: Combine each slicing_spec JSON into a list.: slicing_specs.append(slicing_spec_json).
5: Finally, pass slicing_specs as an parameter for this component.: ModelEvaluationClassificationOp(slicing_specs=slicing_specs) For more details on configuring slices, see

https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform_v1.types.ModelEvaluationSlice

positive_classes (Optional[Sequence[str]]): Optional. The list of class

names to create binary classification metrics based on one-vs-rest for each value of positive_classes provided.

dataflow_service_account (Optional[str]): Optional. Service account to run

the dataflow job. If not set, dataflow will use the default worker service account. For more details, see https://cloud.google.com/dataflow/docs/concepts/security-and-permissions#default_worker_service_account

dataflow_disk_size (Optional[int]): The disk size (in GB) of the machine

executing the evaluation run. If not set, defaulted to 50.

dataflow_machine_type (Optional[str]): The machine type executing the

evaluation run. If not set, defaulted to n1-standard-4.

dataflow_workers_num (Optional[int]): The number of workers executing the

evaluation run. If not set, defaulted to 10.

dataflow_max_workers_num (Optional[int]): The max number of workers

executing the evaluation run. If not set, defaulted to 25.

dataflow_subnetwork (Optional[str]): Dataflow’s fully qualified subnetwork

name, when empty the default subnetwork will be used. More details:

https://cloud.google.com/dataflow/docs/guides/specifying-networks#example_network_and_subnetwork_specifications

dataflow_use_public_ips (Optional[bool]): Specifies whether Dataflow workers

use public IP addresses.

encryption_spec_key_name (Optional[str]): Customer-managed encryption key.

Returns:

evaluation_metrics (google.ClassificationMetrics): Artifact

google.ClassificationMetrics representing the classification evaluation metrics in GCS.

gcp_resources (str): Serialized gcp_resources proto tracking the dataflow

job.

For more details, see https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_components/proto/README.md.

google_cloud_pipeline_components.experimental.evaluation.ModelEvaluationFeatureAttributionOp(gcp_resources: <kfp.components.types.type_annotations.OutputPath object at 0x7f9f6d839a00>, feature_attributions: ~kfp.components.types.artifact_types.Metrics[~kfp.components.types.artifact_types.Metrics], project: str, root_dir: str, location: str = 'us-central1', predictions_format: str = 'jsonl', predictions_gcs_source: ~kfp.components.types.artifact_types.Artifact[~kfp.components.types.artifact_types.Artifact] | None = None, predictions_bigquery_source: ~google_cloud_pipeline_components.types.artifact_types.BQTable[~google_cloud_pipeline_components.types.artifact_types.BQTable] | None = None, dataflow_service_account: str = '', dataflow_disk_size: int = 50, dataflow_machine_type: str = 'n1-standard-4', dataflow_workers_num: int = 1, dataflow_max_workers_num: int = 5, dataflow_subnetwork: str = '', dataflow_use_public_ips: bool = True, encryption_spec_key_name: str = '')

Compute feature attribution on a trained model’s batch explanation results.

Creates a dataflow job with Apache Beam and TFMA to compute feature attributions. Will compute feature attribution for every target label if possible, typically possible for AutoML Classification models.

Args:

project (str): Project to run feature attribution container. location (Optional[str]): Location running feature attribution. If not set,