google_cloud_pipeline_components.experimental.custom_job.custom_job module

Module for supporting Google Vertex AI Custom Training Job Op.

google_cloud_pipeline_components.experimental.custom_job.custom_job.custom_training_job_op(component_spec: Callable, display_name: Optional[str] = '', replica_count: Optional[int] = 1, machine_type: Optional[str] = 'n1-standard-4', accelerator_type: Optional[str] = '', accelerator_count: Optional[int] = 1, boot_disk_type: Optional[str] = 'pd-ssd', boot_disk_size_gb: Optional[int] = 100, timeout: Optional[str] = '', restart_job_on_worker_restart: Optional[bool] = False, service_account: Optional[str] = '', network: Optional[str] = '', worker_pool_specs: Optional[List[Mapping[str, Any]]] = None, encryption_spec_key_name: Optional[str] = '', tensorboard: Optional[str] = '', base_output_directory: Optional[str] = '', labels: Optional[Dict[str, str]] = None) → Callable

Run a pipeline task using Vertex AI custom training job.

For detailed doc of the service, please refer to https://cloud.google.com/vertex-ai/docs/training/create-custom-job

Args:

component_spec:

The task (ContainerOp) object to run as Vertex AI custom job.

display_name:

Optional. The name of the custom job. If not provided the component_spec.name will be used instead.

replica_count:

Optional. The number of replicas to be split between master workerPoolSpec and worker workerPoolSpec. (master always has 1 replica).

machine_type:

Optional. The type of the machine to run the custom job. The default value is “n1-standard-4”.

For more details about this input config, see https://cloud.google.com/vertex-ai/docs/training/configure-compute#machine-types

accelerator_type:

Optional. The type of accelerator(s) that may be attached to the machine as per accelerator_count.

For more details about this input config, see https://cloud.google.com/vertex-ai/docs/reference/rest/v1/MachineSpec#acceleratortype

accelerator_count:

Optional. The number of accelerators to attach to the machine. Defaults to 1 if accelerator_type is set.

boot_disk_type:

Optional. Type of the boot disk (default is “pd-ssd”). Valid values: “pd-ssd” (Persistent Disk Solid State Drive) or “pd-standard”

(Persistent Disk Hard Disk Drive).

boot_disk_size_gb:

Optional. Size in GB of the boot disk (default is 100GB).

timeout:

Optional. The maximum job running time. The default is 7 days. A duration in seconds with up to nine fractional digits, terminated by ‘s’. Example: “3.5s”.

restart_job_on_worker_restart:

Optional. Restarts the entire CustomJob if a worker gets restarted. This feature can be used by distributed training jobs that are not resilient to workers leaving and joining a job.

service_account:

Optional. Sets the default service account for workload run-as account. Users submitting jobs must have act-as permission on this run-as account. If unspecified, the Vertex AI Custom Code Service Agent(https://cloud.google.com/vertex-ai/docs/general/access-control#service-agents) for the CustomJob’s project is used.

network:

Optional. The full name of the Compute Engine network to which the job should be peered. For example, projects/12345/global/networks/myVPC. Format is of the form projects/{project}/global/networks/{network}. Where {project} is a project number, as in 12345, and {network} is a network name. Private services access must already be configured for the network. If left unspecified, the job is not peered with any network.

worker_pool_specs:

Optional, worker_pool_specs for distributed training. this will overwite all other cluster configurations. For details, please see: https://cloud.google.com/ai-platform-unified/docs/training/distributed-training

encryption_spec_key_name:

Optional, customer-managed encryption key options for the CustomJob. If this is set, then all resources created by the CustomJob will be encrypted with the provided encryption key.

tensorboard:

The name of a Vertex AI Tensorboard resource to which this CustomJob will upload Tensorboard logs.

base_output_directory:

The Cloud Storage location to store the output of this CustomJob or HyperparameterTuningJob. see below for more details: https://cloud.google.com/vertex-ai/docs/reference/rest/v1/GcsDestination

labels:

Optional. The labels with user-defined metadata to organize CustomJobs. See https://goo.gl/xmQnxf for more information.

Returns:

A Custom Job component OP correspoinding to the input component OP.