Synchronization¶
v2.10 and after
You can limit the parallel execution of workflows or templates:
- You can use mutexes to restrict workflows or templates to only having a single concurrent execution.
- You can use semaphores to restrict workflows or templates to a configured number of parallel executions.
- You can use parallelism to restrict concurrent tasks or steps within a single workflow.
The term "locks" on this page means mutexes and semaphores.
You can create multiple semaphore configurations in a ConfigMap that can be referred to from a workflow or template.
For example:
apiVersion: v1
kind: ConfigMap
metadata:
name: my-config
data:
workflow: "1" # Only one workflow can run at given time in particular namespace
template: "2" # Two instances of template can run at a given time in particular namespace
Workflow-level Synchronization¶
You can limit parallel execution of workflows by using the same synchronization reference.
In this example the synchronization key workflow is configured as limit "1", so only one workflow instance will execute at a time even if multiple workflows are created.
Using a semaphore configured by a ConfigMap:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: synchronization-wf-level-
spec:
entrypoint: hello-world
synchronization:
semaphores: # v3.6 and after
- configMapKeyRef:
name: my-config
key: workflow
# semaphore: # deprecated: v3.5 and before
# configMapKeyRef:
# name: my-config
# key: workflow
templates:
- name: hello-world
container:
image: busybox
command: [echo]
args: ["hello world"]
Using a mutex is equivalent to a limit "1" semaphore:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: synchronization-wf-level-
spec:
entrypoint: hello-world
synchronization:
mutexes: # v3.6 and after
- name: test
# mutex: # deprecated: v3.5 and before
# name: test
templates:
- name: hello-world
container:
image: busybox
command: [echo]
args: ["hello world"]
Template-level Synchronization¶
You can limit parallel execution of templates by using the same synchronization reference.
In this example the synchronization key template is configured as limit "2", so a maximum of two instances of the acquire-lock template will execute at a time.
This applies even when multiple steps or tasks within a workflow or different workflows refer to the same template.
Using a semaphore configured by a ConfigMap:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: synchronization-tmpl-level-
spec:
entrypoint: synchronization-tmpl-level-example
templates:
- name: synchronization-tmpl-level-example
steps:
- - name: synchronization-acquire-lock
template: acquire-lock
arguments:
parameters:
- name: seconds
value: "{{item}}"
withParam: '["1","2","3","4","5"]'
- name: acquire-lock
synchronization:
semaphores: # v3.6 and after
- configMapKeyRef:
name: my-config
key: template
# semaphore: # deprecated: v3.5 and before
# configMapKeyRef:
# name: my-config
# key: template
container:
image: alpine:latest
command: [sh, -c]
args: ["sleep 10; echo acquired lock"]
Using a mutex will limit to a single concurrent execution of the template:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: synchronization-tmpl-level-mutex-
spec:
entrypoint: synchronization-tmpl-level-mutex-example
templates:
- name: synchronization-tmpl-level-mutex-example
steps:
- - name: synchronization-acquire-lock
template: acquire-lock
arguments:
parameters:
- name: seconds
value: "{{item}}"
withParam: '["1","2","3","4","5"]'
- name: synchronization-acquire-lock1
template: acquire-lock-1
arguments:
parameters:
- name: seconds
value: "{{item}}"
withParam: '["1","2","3","4","5"]'
- name: acquire-lock
synchronization:
mutexes: # v3.6 and after
- name: welcome
# mutex: # deprecated: v3.5 and before
# name: welcome
container:
image: alpine:latest
command: [sh, -c]
args: ["sleep 20; echo acquired lock"]
- name: acquire-lock-1
synchronization:
mutexes: # v3.6 and after
- name: test
# mutex: # deprecated: v3.5 and before
# name: test
container:
image: alpine:latest
command: [sh, -c]
args: ["sleep 50; echo acquired lock"]
Queuing¶
When a workflow cannot acquire a lock it will be placed into a ordered queue.
You can set a priority on workflows.
The queue is first ordered by priority: a higher priority number is placed before a lower priority number.
The queue is then ordered by creationTimestamp: older workflows are placed before newer workflows.
Workflows can only acquire a lock if they are at the front of the queue for that lock.
Multiple locks¶
v3.6 and after
You can specify multiple locks in a single workflow or template.
synchronization:
mutexes:
- name: alpha
- name: beta
semaphores:
- configMapKeyRef:
key: foo
name: my-config
- configMapKeyRef:
key: bar
name: my-config
The workflow will block until all of these locks are available.
Workflow-level parallelism¶
You can use parallelism within a workflow or template to restrict the total concurrent executions of steps or tasks.
(Note that this only restricts concurrent executions within the same workflow.)
Examples:
parallelism-limit.yamlrestricts the parallelism of a loopparallelism-nested.yamlrestricts the parallelism of a nested loopparallelism-nested-dag.yamlrestricts the number of dag tasks that can be run at any one timeparallelism-nested-workflow.yamlshows how parallelism is inherited by childrenparallelism-template-limit.yamlshows how parallelism of looped templates is also restricted
Warning
If a Workflow is at the front of the queue and it needs to acquire multiple locks, all other Workflows that also need those same locks will wait. This applies even if the other Workflows only wish to acquire a subset of those locks.
Other Parallelism support¶
You can also restrict parallelism at the Controller-level.