Parallel queries in python, with a file and several pods in kubernetes Parallel queries in python, with a file and several pods in kubernetes kubernetes kubernetes

Parallel queries in python, with a file and several pods in kubernetes


I think that you should work it with asynchronous tasks worker such as Celery or Dramatiq.

The conceptual is send the tasks (your query line) into the Message Broker (Redis or RabbitMQ) then let the worker consumed the tasks (receive the query line and make a request to API)

However, The Dramatiq or Celery have a feature like retry the task if it failed.

Not sure is that you want, But you can explore with this repo,https://github.com/matiaslindgren/celery-kubernetes-example