tfx.v1.dsl.Pipeline

Logical TFX pipeline object.

Inherits From: BaseNode

tfx.v1.dsl.Pipeline(
    pipeline_name: str,
    pipeline_root: Optional[Union[str, ph.Placeholder]] = '',
    metadata_connection_config: Optional[metadata.ConnectionConfigType] = None,
    components: Iterable[tfx.v1.types.BaseNode] = (),
    enable_cache: bool = False,
    beam_pipeline_args: Optional[List[Union[str, ph.Placeholder]]] = None,
    platform_config: Optional[message.Message] = None,
    execution_mode: tfx.v1.dsl.ExecutionMode = tfx.v1.dsl.ExecutionMode.SYNC,
    inputs: Optional[PipelineInputs] = None,
    outputs: Optional[Dict[str, channel.OutputChannel]] = None,
    dsl_context_registry: Optional[dsl_context_registry_lib.DslContextRegistry] = None
)

Used in the notebooks

Used in the tutorials

Pipeline object represents the DAG of TFX components, which can be run using one of the pipeline orchestration systems that TFX supports. For details, please refer to the guide.

pipeline_name Name of the pipeline;
pipeline_root Path to root directory of the pipeline. This will most often be just a string. Some orchestrators may have limited support for constructing this from a Placeholder, e.g. a RuntimeInfoPlaceholder that refers to fields from the platform config. pipeline_root is optional only if the pipeline is composed within another parent pipeline, in which case it will inherit its parent pipeline's root.
metadata_connection_config The config to connect to ML metadata.
components Optional list of components to construct the pipeline.
enable_cache Whether or not cache is enabled for this run.
beam_pipeline_args Pipeline arguments for Beam powered Components.
platform_config Pipeline level platform config, in proto form.
execution_mode The execution mode of the pipeline, can be SYNC or ASYNC.
inputs Optional inputs of a pipeline.
outputs Optional outputs of a pipeline.
dsl_context_registry DslContextRegistry to use for this pipeline, if not provided then the current context (potentially a new DslContext) will be used.

components A deterministic list of logical components of this pipeline, which are deduped and topologically sorted.
enable_cache Whether or not cache is enabled for this run.
metadata_connection_config The config to connect to ML metadata.
execution_mode Execution mode of the pipeline. Currently only support synchronous execution mode.
beam_pipeline_args Pipeline arguments for Beam powered Components. Use with_beam_pipeline_args to set component level Beam args.
platform_config Pipeline level platform config, in proto form.
exec_properties

inputs

outputs

Methods

with_node_execution_options

with_node_execution_options(
    node_execution_options: utils.NodeExecutionOptions
) -> typing_extensions.Self