Limited pods in Kubernetes (EKS) and Airflow

Ok, I've already seen where the problem is. Kubernetes does not manage pods well without requests or limits. I have added requests and limits and now the nodes are filled completely with 20 pods each.

Now I have another problem. The pods don't seem to disappear when they finish. The pods only print "Hello world", despite this, in dag_run there are dags that take from 49 seconds to 22 minutes. With the fact that although there are more pods in each node, the whole system still takes more than 20 minutes to complete, as before.

Something is wrong. If I have two nodes that can host 100 pods. And every pod takes a minute to finish, if I run five hundred pods simultaneously, all the work should end in five minutes. But it always takes between 16-20 minutes. The nodes are never full of pods at full capacity and the pods finish their work but take some time to be deleted. What makes it so slow?

Use Airflow 1.10.9 with this configuration:

ENV AIRFLOW__CORE__PARALLELISM=100ENV AIRFLOW__CORE__NON_POOLED_TASK_SLOT_COUNT=100ENV AIRFLOW__CORE__DAG_CONCURRENCY=100ENV AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG=100ENV AIRFLOW__CORE__SQL_ALCHEMY_POOL_SIZE=0ENV AIRFLOW__CORE__SQL_ALCHEMY_MAX_OVERFLOW=-1ENV AIRFLOW__KUBERNETES_WORKER_PODS_CREATION_BATCH_SIZE=10ENV AIRFLOW__SCHEDULER__MAX_THREADS=6