Work Avoidance¶

v2.9 and after

You can make workflows faster and more robust by employing work avoidance. A workflow that utilizes this is simply a workflow containing steps that do not run if the work has already been done.

This is a technique is similar to memoization. Work avoidance is totally in your control and you make the decisions as to have to skip the work. Memoization is a feature of Argo Workflows to automatically skip steps which generate outputs. Prior to version 3.5 this required outputs to be specified, but you can use memoization for all steps and tasks in version 3.5 or later.

This simplest way to do this is to use marker files.

Use cases:

An expensive step appears across multiple workflows - you want to avoid repeating them.
A workflow has unreliable tasks - you want to be able to resubmit the workflow.

A marker file is a file that indicates the work has already been done. Before doing the work you check to see if the marker has already been done:

if [ -e /work/markers/name-of-task ]; then
    echo "work already done"
    exit 0
fi
echo "working very hard"
touch /work/markers/name-of-task

Choose a name for the file that is unique for the task, e.g. the template name and all the parameters:

touch /work/markers/$(date +%Y-%m-%d)-echo-{{inputs.parameters.num}}

You need to store the marker files between workflows and this can be achieved using a PVC and optional input artifact.

This complete work avoidance example has the following:

A PVC to store the markers on.
A load-markers step that loads the marker files from artifact storage.
Multiple echo tasks that avoid work using marker files.
A save-markers exit handler to save the marker files, even if they are not needed.

Work Avoidance¶

Comments