Creating a queue system with Argo Workflows

Argo Workflows has no concept of a queue, so it has no way of knowing when the queue is full. If you need queue control, that should happen before submitting workflows.

Once the workflows are submitted, there are a number of ways to limit resource usage.

1. Pod resources - each Workflow step is represented by a Kubernetes Pod. You can set resource requests and limits just like you would with a Pod in a Deployment.
2. Step parallelism limit - within a Workflow, you can limit the number of steps running concurrently. This can help when a step is particularly resource-intensive.
3. Workflow parallelism limit - you can limit the number of workflows running concurrently by configuring them to us a semaphore.

There are a number of other performance optimizations like setting Workflow and Pod TTLs and offloading YAML for large Workflows to a DB instead of keeping them on the cluster.

As far as I know, there is no way to set a Workflow limit so that Argo will reject additional Workflow submissions until more resources are available. This is a problem if you're worried about Kubernetes etcd filling up with too many Workflow definitions.

To keep from blowing up etcd, you'll need another app of some kind sitting in from of Argo to queue Workflows submissions until more resources become available.