k8s cluster hangs when running 22 airflow worker pods in parallel
It looks like like a resource issue.
You can try to increase CPU and memory in Docker settings and connect to docker VM to check resources usage.
Alternatively, you can create one master cluster with one or two nodes in Virtualbox and try to run tasks there. In this case master node will not suffer from lack of resources and cluster should still be available to connect.
Here is how to connect to docker-desktop for mac
Please let me know if that helped.
Airflow defaults task concurrency is set to 32, which then tries to run all 22 pods at same time, demanding more memory than my k8s cluster could bare. Reducing down AIRFLOW__CORE__PARALLELISM in airflow config from 32 (default) to 8 did the work for me. And of-course, for those whose business case demands running more tasks in parallel, increasing memory and multi-clustering is the option.