Skip to content

[Feature Request] Partially Sequential PipelineRuns/Temporary Task Halting Guards #8341

@RobertDiebels

Description

@RobertDiebels

Feature request

For some Pipelines it may be useful to temporarily halt a PipelineRun when it reaches a Task T that:

  • Performs a transformation on a data-source.
  • Uses an earlier output of that transformation (from a previous PipelineRun) as an input to the current execution of the transformation.
  • May behave incorrectly when multiple TaskRuns of Task T are running in Parallel.

In this case a PipelineRun should halt execution until all PipelineRuns that were initiated earlier have completed Task T (either successfully or unsuccessfully).

This would create a situation where PipelineRuns halt execution (behave sequentially) only for certain tasks, and execute the rest in parallel (default behavior). In turn creating Partially Sequential PipelineRuns or Partially Parallel PipelineRuns (whichever you prefer).

Use case

Assume we have a Task V in Pipeline P, that reads the current version from a VCS and based on other data determines a new version.
Furthermore, we have 2 PipelineRuns X and Y for Pipeline P that are running in parallel, where PipelineRun X is initiated earlier than PipelineRun Y. Hence, we expect PipelineRun to write a (new) version to VCS and PipelineRun Y to use X's version as its current version.

If PipelineRun X is stalled, for instance due to waiting on scheduling, then PipelineRun Y can potentially catch up to it, in terms of tasks executed.
As a result it may occur that, for both PipelineRun X and Y, Task V is initiated at the same moment in time. Since we're dependent on our CVS and data existing within it, this may cause the output of incorrect versions. As both TaskRuns of Task T will use the same "current version", which is highly undesirable.

Additional Information

In essence, this feature is similar to an execution guard, however for this feature execution would be temporarily halted/postponed (Temporary Task Halting Guard), whereas execution guards prevent execution altogether.

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/featureCategorizes issue or PR as related to a new feature.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions