Release Notes

Upcoming release

Release 2.0.0

Google Cloud Pipeline Components v2 is generally available!

Structure

Major changes

  • Migrate many components to the ``v1` GA namespace <https://google-cloud-pipeline-components.readthedocs.io/en/google-cloud-pipeline-components-2.0.0/>`_

  • Migrate components to the ``preview` namespace <>`_

    • preview.model_evaluation.ModelEvaluationFeatureAttributionOp

    • preview.model_evaluation.DetectModelBiasOp

    • preview.model_evaluation.DetectDataBiasOp

    • preview.dataflow.DataflowFlexTemplateJobOp

  • Add many new components:

    • v1.dataflow.DataflowFlexTemplateJobOp

    • v1.model.evaluation.vision_model_error_analysis_pipeline

    • v1.model.evaluation.evaluated_annotation_pipeline

    • v1.model.evaluation.evaluation_automl_tabular_feature_attribution_pipeline

    • v1.model.evaluation.evaluation_automl_tabular_pipeline

    • v1.model.evaluation.evaluation_automl_unstructure_data_pipeline

    • v1.model.evaluation.evaluation_feature_attribution_pipeline

  • Make GCPC artifacts usable in user-defined KFP SDK Python components (Containerized Python Components recommended)

Runtime

  • Change runtime base image to marketplace.gcr.io/google/ubuntu2004

  • Apply latest GCPC image vulnerability resolutions (base OS and software updates)

Dependencies

  • Depend on KFP SDK v2 (GCPC v2 is not compatible with KFP v1)

  • Set google-api-core<1.34.0 to avoid 900s timeout

  • Remove google-cloud-notebooks and google-cloud-storage dependencies

Documentation

  • Refresh GCPC v2 reference documentation

Other

  • Assorted minor component interface changes

  • Assorted bug fixes

  • Change force_direct_runner flag to force_direct_runner_mode in experimental evaluation components to allow users to choose the runner of the evaluation pipeline

  • Support upload model with pipeline job id in UploadModel GCPC component

  • Change default value of prediction_score_column for AutoML Forecasting & Regression components to prediction.value

  • Change dataflow_disk_size parameter to dataflow_disk_size_gb in all model evaluation components

  • Remove aiplatform.CustomContainerTrainingJobRunOp and aiplatform.CustomPythonPackageTrainingJobRunOp components

Upcoming changes

  • Additional migrations from the 1.x.x’s experimental namespace to the v1 and preview namespaces

Release 2.0.0b5

  • Fix experimental evaluation component runtime bugs

  • Add model evaluation pipelines:

    • v1.model.evaluation.vision_model_error_analysis_pipeline

    • v1.model.evaluation.evaluated_annotation_pipeline

    • v1.model.evaluation.evaluation_automl_tabular_feature_attribution_pipeline

    • v1.model.evaluation.evaluation_automl_tabular_pipeline

    • v1.model.evaluation.evaluation_automl_unstructure_data_pipeline

    • v1.model.evaluation.evaluation_feature_attribution_pipeline

  • Make GCPC artifacts usable in user-defined KFP SDK Python Components and add documentation

  • Change force_direct_runner flag to force_direct_runner_mode in experimental evaluation components to allow users to choose the runner of the evaluation pipeline

  • Add experimental AutoML Forecasting Seq2Seq and Temporal Fusion Transformer pipelines

  • Apply latest GCPC image vulnerability resolutions (base OS and software updates)

Release 2.0.0b4

  • GCPC v2 reference documentation improvements

  • Change GCPC base image to marketplace.gcr.io/google/ubuntu2004

  • Apply latest GCPC image vulnerability resolutions (base OS and software updates)

  • Fix dataset components

  • Fix payload sanitation bug in google_cloud_pipeline_components.v1.batch_predict_job.ModelBatchPredictOp

  • Assorted experimental component bug fixes (note: experimental namespace will be removed in a future pre-release)

Release 2.0.0b3

  • Support sparse layer masking feature selection for experimental.automl.tabular classification/regression components

  • Fixes for GCPC v2 reference documentation

  • Fix experimental.dataflow.DataflowFlexTemplateJobOp component

  • Remove unused SDK dependency on google-cloud-notebooks and google-cloud-storage

Release 2.0.0b2

  • Add experimental.dataflow.DataflowFlexTemplateJobOp component

  • Remove aiplatform.CustomContainerTrainingJobRunOp and aiplatform.CustomPythonPackageTrainingJobRunOp components

  • Migrate other aiplatform.automl_training_job, aiplatform.ModelUndeployOp, aiplatform.EndpointDeleteOp, and aiplatform.ModelDeleteOp components to the v1 namespace

  • Deduplicate component definitions between experimental and v1 namespaces

Release 2.0.0b1

  • Change base image to ubuntu OS

  • Set google-api-core<1.34.0 to avoid 900s timeout

Release 2.0.0b0

  • Release of GCPC v2 beta

  • Supports KFP v2 beta

  • Experimental components that already in v1 folder are removed

  • Experimental components that are not fully tested (e.g. AutoML, Model Evaluation) are excluded for now, will be added in future releases

  • Even though the GCPC package’s version is v2, the components under v1 folder have no interface change, so the those components’ version remain as v1, decoupled from package version.

Release 1.0.44

  • Apply latest GCPC image vulnerability resolutions (base OS and software updates)

Release 1.0.43

  • Patch 5de4d78: unpin google-api-core version

Release 1.0.42

  • Patch cb7d9a8: Update import_model_evaluation so models with 100+ labels will not import confusion matrices at every threshold

Release 1.0.41

  • Add data-filter-split feature back to the ImageTrainingJob component

Release 1.0.40

  • Change base image to ubuntu OS

  • Set google-api-core<1.34.0 to avoid 900s timeout

Release 1.0.39

  • Fix AutoML Table pipeline failing on importing model evaluation metrics

Release 1.0.38

  • Fix default value issue in bigquery query API

Release 1.0.36

  • Cherrypick e358dee2f8d5c01580438ee54988f01fc3f16a7c and snap a new release

Release 1.0.35

  • Fix images for BQML components

Release 1.0.34

  • Cherrypick d1f1ee9f2bbd09df7ea6ab51b21f07ba5f86c871 and snap a new release

Release 1.0.33

  • Fix aiplatform & v1 batch predict job to work with KFP v2

  • Release Structured Data team’s updated components and pipelines

Release 1.0.32

  • Support a HyperparameterTuningJobWithMetrics type to take execution_metrics path

Release 1.0.31

  • Fix aiplatform serialization

  • Release Structured Data team’s updated components and pipelines

  • Add components for natural language: training TFHub model and preprocessing component for batch prediction

Release 1.0.30

  • Fix aiplatform & v1 batch predict job to work with KFP v2

  • Fix serialization for aiplatform components

  • Update Dataproc doc links

  • Update tags in Structured Data team’s forecasting pipelines

Release 1.0.29

  • Propagate vertex system labels to the downstream resources

  • Release Structured Data team’s updated components and pipelines

  • Fix Dataproc component doc to indicate that batch_id is optional

  • Simplify create_custom_training_job_op_from_component

  • Fix list and dict types for converted aiplatform components

Release 1.0.28

  • Support uploading for model versions for ModelUploadOp

  • Add text classification data processing component and training component

  • Propagates vertex system labels to the downstream resources for batch prediction job

Release 1.0.27

  • Add DataprocBatch resource to gcp_resources output parameter

  • Support serving default in bq export model job op

Release 1.0.26

  • Temporary fix for artifact types

  • Sync GCPC staging to prod to include AutoML model comparison and prophet pipelines

  • Update documentation for Eval components

  • Update HP tuning sample notebook

  • Improve folder structure for evaluation components

  • Model Evaluation, rename EvaluationDataSplitterOp to TargetFieldDataRemoverOp, rename ground_truth_column to target_field, rename class_names to class_labels, and remove key_columns input

  • Add model input to vertex ai model evaluation component

Release 1.0.25

  • Bigquery: Update public doc for evaluate model per customer feedback

  • Add Infra Validation remote runner

  • Add notification v1 doc to the v1 page

  • AutoML: Sync GCPC staging to prod to include bug fix for built-in algorithms

Release 1.0.24

  • Add notification v1 doc

  • Convert all v1 components into individual launchers and remote runners

  • Update AutoML Tables components to have latest SDK features

  • Add support for staging Dataflow options (sdk_location and extra_package)

Release 1.0.23

  • AutoML: Sync GCPC staging to prod to include recent API changes

  • TensorBoard: Make some input parameters optional to provide better user experience

Release 1.0.22

  • TensorBoard: Make some input parameters optional to provide better user experience

Release 1.0.21

  • Fix input parameter in tensorboard experiment creator component

  • Convert bigquery components into individual launchers and remote runners

  • Model Evaluation: Add metadata field for pipeline resource name

Release 1.0.20

  • Add special case in json_util.py where explanation_spec metadata outputs can have empty values

  • Update the docstring for missing arguments on feature_importance component

  • Create new tensorboard experiment creator component

  • Remove unused input in evaluation classification yaml

  • Update the docstring for exported_model_path in export_model

Release 1.0.19

  • Propagating labels for explain_forecast_model component

  • Model Evaluation - Add evaluation forecasting default of 0.5 for quantiles

  • Dataproc - Fix missing error payload from logging

  • Added BigQuery input support to evaluation components

  • Model Evaluation - Allow dataset paths list

  • Fix the docstring for ml_advanced_weights component

  • Fix the duplicated arguments in bigquery_ml_global_explain_job

  • Import importer from dsl namespace instead

  • Convert batch_prediction_job_remote_runner into individual launcher

Release 1.0.18

  • Model Evaluation - Give evaluation preprocessing components unique dataflow job names

  • Add vertex_notification_email component on v1 folder

Release 1.0.17

  • Model Evaluation - Rearrange json and yaml files in e2e test to eliminate duplicate defining and reading

  • Model Evaluation - Update JSON templates for evaluation

  • Model Evaluation - Split evaluation component into classification, forecasting, and regression evaluation & create artifact types for google.__Metrics

  • Model Evaluation - Match predictions input argument name to other Evaluation components

  • Model Evaluation - Update import_model_evaluation component to accept new google.___Metrics artifact types

  • Model Evaluation - Update regression and forecasting to contain ground truth input fields

  • Reverse re.findall order of arguments to (pattern, string) in job_remote_runner

  • Model Evaluation - Update evaluation container to v0.5 for data sampler and splitter preprocessing components

Release 1.0.16

  • Evaluation - Separate feature attribution from evaluation component to its own component

  • AutoML Tables - Include fix AMI issues for criteo dataset

  • AutoML Tables - Change Vertex evaluation pipeline templates

  • Model Evaluation - Import model evaluation slices when available in the metrics

  • Model Evaluation - Add nargs to allow for empty string input by component

Release 1.0.15

  • Sync AutomL components’ code to GCPC codebase to reflect bug fix in FTE component spec

  • Auto-generate batch id if none is specified in Dataproc components

  • Add ground_truth_column input argument to data splitter component

Release 1.0.14

  • Temporarily pin apache_beam version to <2.34.0 due to https://github.com/apache/beam/issues/22208.

  • Remove kms key name from the drop model interface.

  • Move new BQ components from experimental to v1

  • Fix the problem that AutoML Tabular pipeline could fail when using large number of features

Release 1.0.13

  • AutoML Tables - Fix AutoML Tabular pipeline always running evaluation.

  • AutoML Tables - Fix AutoML Tabular pipeline when there are a large set of input features.

  • Model Evaluation - Evaluation preprocessing component change output GCS artifact to JsonArray.

Release 1.0.12

  • Move generating feature ranking to utils to be available in SDK

  • Change JSON to primitive types for Tables v1, built-in algorithm and internal pipelines

  • AutoML Tables - update Tabular workflow to reference 1.0.10 launcher image

  • AutoML Tables - Add dataflow_service_account to specify custom service account to run dataflow jobs for stats_and_example_gen and transform components.

  • AutoML Tables - Update skip_architecture_search pipeline

  • AutoML Tables - Add algorithm to pipeline, also switch the default algorithm to be AMI

  • AutoML Tables - Use feature transform engine docker image for related components

  • AutoML Tables - Make calculation logic in SDK helper function run inside a component for Tables v1 and skip_architecture_search pipelines

  • AutoML Tables - weight_column_name -> weight_column and target_column_name -> target_column for Tables v1 and skip_architecture_search pipelines

  • AutoML Tables - For built-in algorithms, the transform_config input is expected to be a GCS file path.

  • AutoML Tables - Make generate analyze/transform data and split materialized data as components

  • AutoML Tables - Add automl_tabular_pipeline pipeline for Tabular Workflow.

  • AutoML Tables - Use FTE image directly to launch FTE component

  • Model Evaluation - Add display name to import model evaluation component

  • Model Evaluation - Update default number of workers.

Release 1.0.11

  • Add custom component to automl_tabular default pipeline

  • Add transformations_path to stats_and_example_gen and enable for v1 default pipeline and testing pipeline

  • Use ‘unmanaged_container_model’ instead of ‘model’ in infra validator component for automl tabular

  • Update evaluation component to v0.3

Release 1.0.10

  • Add new Evaluation components ‘evaluation_data_sampler’ and ‘evaluation_data_splitter’

  • Make AutoML Tables ensemble also output explanation_metadata artifact

  • AutoML Tables - decouple transform config planner from metadata

  • AutoML Tables - Feature transform engine config planner to generate training schema & instance baseline

Release 1.0.9

  • FTE transform config passed as path to config file instead of directly as string to FTE

  • Support BigQuery ML weights job component

  • FTE now outputs training schema.

  • Support BigQuery ML reconstruction loss and trial info job components

  • Adding ML.TRAINING_INFO KFP and ML.EXPLAIN_PREDICT BQ Component.

  • Add additional experiments in distillation pipeline.

  • Support BigQuery ML advanced weights job component.

  • Support BigQuery drop model job components.

  • Support BigQuery ML centroids job components.

  • Wide and Deep and Tabnet models both now use the Feature Transform Engine pipeline instead of the Transform component.

  • Adding ML.CONFUSION_MATRIX KFP BQ Component.

  • Adding ML.FEATURE_INFO KFP BQ Component.

  • Merge distill_skip_evaluation and skip_evaluation pipelines with default pipeline using dsl.Condition

  • Adding ML.ROC_CURVE KFP BQ Component.

  • Adding ML.PRINCIPAL_COMPONENTS and ML.PRINCIPAL_COMPONENT_INFO KFP BQ component.

  • Adding ML.FEATURE_IMPORTANCE KFP BQ Component.

  • Add ML.ARIMA_COEFFICIENTS in component.yaml

  • Adding ML.Recommend KFP BQ component.

  • Add ML.ARIMA_EVALUATE in component.yaml

  • KFP component for ml.explain_forecast

  • KFP component for ml.forecast

  • Add distill + evaluation pipeline for Tables

  • Adding ML.GLOBAL_EXPLAIN KFP BQ Component.

  • KFP component for ml.detect_anomalies

  • Make stats-gen component to support running with example-gen only mode

  • Fix AutoML Tables pipeline and builtin pipelines on VPC-SC environment.

  • Preserve empty features in explanation_spec

Release 1.0.8

  • Use BigQuery batch queries in ARIMA pipeline after first 50 queries

  • Stats Gen and Feature Transform Engine pipeline integration.

  • Add window config to ARIMA pipeline

  • Removed default location setting from AutoML components and documentation.

  • Update default machine type to c2-standard-16 for built-in algorithms Custom and HyperparameterTuning Jobs

  • Use float instead of int max windows, which caused ARIMA pipeline failure

  • Renamed “Feature Transform Engine Transform Configuration” component to “Transform Configuration Planner” for clarity.

  • Preserve empty features in explanation_spec

  • Change json util to not remove empty primitives in a list.

  • Add model eval component to built-in algorithm default pipelines

  • Quick fix to Batch Prediction component input “bigquery_source_input_uri”

Release 1.0.7

  • Allow metrics and evaluated examples tables to be overwritten.

  • Replace custom copy_table component with BQ first-party query component.

  • Support vpc in feature selection.

  • Add import eval metrics to model to AutoML Tables default pipeline.

  • Add default Wide & Deep study_spec_parameters configs and add helper function to utils.py to get parameters.

Release 1.0.6

  • Update import evaluation metrics component.

  • Support parameterized input for reserved_ip_range and other Vertex Training parameters in custom job utility.

  • Generate feature selection tuning pipeline and test utils.

  • Add retries to queries hitting BQ write quota on BQML Arima pipeline.

  • Minor changes to the feature transform engine and transform configuration component specs to support their integration.

  • Update Executor component for Pipeline to support kernel_spec.

  • Add default TabNet study_spec_parameters_override configs for different dataset sizes and search space modes and helper function to get the parameters.

Release 1.0.5

  • Add VPC-SC and CMEK support for the experimental evaluation component

  • Add an import evaluation metrics component

  • Modify AutoML Tables template JSON pipeline specs

  • Add feature transform engine AutoML Table component.

Release 1.0.4

  • Create alias for create_custom_training_job_op_from_component as create_custom_training_job_from_component

  • Add support for env variables in Custom_Job component.

Release 1.0.3

  • Add API docs for Vertex Notification Email

  • Add template JSON pipeline spec for running evaluation on a managed GCP Vertex model.

  • Update documentation for Dataproc Serverless components v1.0.

  • Use if:cond:then when specifying image name in built-in algorithm hyperparameter tuning job component and add separate hyperparameter tuning job default pipelines for TabNet and Wide & Deep

  • Add gcp_resources in the eval component output

  • Add downsampled_test_split_json to example_and_stats_gen component.

Release 1.0.2

  • Dataproc Serverless components v1.0 launch.

  • Bump google-cloud-aiplatform version

  • Fix HP Tuning documentation, fixes #7460

  • Use feature ranking and selected features in AutoML Tables stage 1 tuning component.

  • Update distill_skip_evaluation_pipeline for performance improvement.

Release 1.0.1

  • Add experimental email notification component

  • add docs for create_custom_training_job_op_from_component

  • Remove ForecastingTrainingWithExperimentsOp component.

  • Use unmanaged_container_model for model_upload for AutoML Tables pipelines

  • add nfs mount support for create_custom_training_job_op_from_component

  • Implement cancellation for dataproc components

  • bump google-api-core version to 2.0+

  • Add retry for batch prediction component

Release 1.0.0

  • add enable_web_access for create_custom_training_job_op_from_component

  • remove remove training_filter_split, validation_filter_split, test_filter_split from automl components

  • Update the dataproc component docs

Release 0.3.1

  • Implement cancellation propagation

  • Remove encryption key in input for BQ create model

  • Add Dataproc Batch components

  • Add AutoML Tables Wide & Deep trainer component and pipeline

  • Create GCPC v1 and readthedocs for v1

  • Fix bug when ExplanationMetadata.InputMetadata field is provided the batch prediction job component

Release 0.3.0

  • Update BQML export model input from string to artifact

  • Move model/endpoint/job/bqml compoennts to 1.0 namespace

  • Expose enable_web_access and reserved_ip_ranges for custom job component

  • Add delete model and undeploy model components

  • Add utility library for google artifacts

Release 0.2.2

  • Fixes for BQML components

  • Add util functions for HP tuning components and update samples

Release 0.2.1

  • Add BigqueryQueryJobOp, BigqueryCreateModelJobOp, BigqueryExportModelJobOp and BigqueryPredictModelJobOp components

  • Add ModelEvaluationOp component

  • Accept UnmanagedContainerModel artifact in Batch Prediction component

  • Add util components and fix YAML for HP Tuning Job component; delete lightweight python version

  • Add generic custom training job component

  • Fix Dataflow error log reporting and component sample

Release 0.2.0

  • Update custom job name to create_custom_training_job_op_from_component

  • Remove special handling for “=” in remote runner.

  • Bug fixes and documentation updates.

Release 0.1.9

  • Dataflow and wait components

  • Bug fixes

Release 0.1.8

  • Update the CustomJob component interface, and rename to custom_training_job_op

  • Define new artifact types for Google Cloud resources.

  • Update the AI Platform components. Added the component YAML and uses the new Google artifact types

  • Add Vertex notebook component

  • Various doc updates

Release 0.1.7

  • Add support for labels in custom_job wrapper.

  • Add a component that connects the forecasting preprocessing and training components.

  • Write GCP_RESOURCE proto for the custom_job output.

  • Expose Custom Job parameters Service Account, Network and CMEK via Custom Job wrapper.

  • Increase KFP min version dependency.

  • AUpdate documentations for GCPC components.

  • Update typing checks to include Python3.6 deprecated types.

Release 0.1.6

  • Experimental component for Model Forecast.

  • Fixed issue with parameter passing for Vertex AI components

  • Simplify auto generated API docs

  • Fix parameter passing for explainability on ModelUploadOp

  • Update naming of project and location parameters for all for GCPC components

Release 0.1.5

  • Experimental component for vertex forecasting preprocessing and validation

Release 0.1.4

  • Experimental component for tfp_anomaly_detection.

  • Experimental module for Custom Job Wrapper.

  • Fix to include YAML files in PyPI package.

  • Restructure the google_cloud_pipeline_components.

Release 0.1.3

  • Use correct dataset type when passing dataset to CustomTraining.

  • Bump google-cloud-aiplatform to 1.1.1.

Release 0.1.2

  • Add components for AutoMLForecasting.

  • Update API documentation.

Release 0.1.1

  • Fix issue with latest version of KFP not accepting pipeline_root in kfp.compile.

  • Fix Compatibility with latest AI Platform name change to replace resource name class with Vertex AI

Release 0.1.0

First release

  • Initial release of the Python SDK with data and model managemnet operations for Image, Text, Tabular, and Video Data.