Kubernetes Python Client error create_namespaced_binding: (409) Reason: Conflict

When a pod is created, the scheduler gets three "Pending" events:

1. Pod is not scheduled yet ('node_name': None, 'status': {'conditions': None,...})
2. Pod is scheduled ('node_name': 'some_node_name','status': {'conditions': [...,'status': True, 'type':'PodScheduled'],...} )
3. Pod is initialized but not ready yet ('node_name': 'minikube','status': {'conditions': [...,'status': True, 'type':'Initialized'], ... ,'status': False, 'type':'Ready']} )

Therefore, your custom scheduler should bind pod to the node on the first event, check the status of the pod and make sure it is scheduled when the second event appears, and then check if the pod is initialized when the third event appears.

If something goes wrong, the scheduler may need to take into account the previous errors and probably try to schedule the pod to different nodes.

In your case, your scheduler threats all three events like the first one and tries to schedule the pod again and again. That's why you see that "pod xxx is already assigned to node yyy" error.

Here is the updated main method (according to @VAS's answer, thanks) to find the right PENDING pod that has not been scheduled yet.

def main():    w = watch.Watch()    for event in w.stream(v1.list_namespaced_pod, 'default'): # default == namespace name        # All pending pods have 3 states (not scheduled, scheduled, initialized but not ready yet)        # We look for NOT SCHEDULED pod and conditions==None        if event['object'].status.phase == 'Pending' and event['object'].status.conditions == None and event['object'].spec.scheduler_name == CUSTOM_SCHEDULER_NAME:            print "Pending and Not Scheduled POD Found "+event['object'].metadata.name            try:                res = scheduler(event['object'].metadata.name,random.choice(nodes_available())) # nodes_available() returns all available nodes                print "success"            except Exception as a:                print ("Exception when calling CoreV1Api->create_namespaced_binding: %s\n" % a)