Cron Jobs in Kubernetes - connect to existing Pod, execute script

As far as I'm aware there is no "official" way to do this the way you want, and that is I believe by design. Pods are supposed to be ephemeral and horizontally scalable, and Jobs are designed to exit. Having a cron job "attach" to an existing pod doesn't fit that module. The Scheduler would have no idea if the job completed.

Instead, a Job can to bring up an instance of your application specifically for running the Job and then take it down once the Job is complete. To do this you can use the same Image for the Job as for your Deployment but use a different "Entrypoint" by setting command:.

If they job needs access to data created by your application then that data will need to be persisted outside the application/Pod, you could so this a few ways but the obvious ways would be a database or a persistent volume.For example useing a database would look something like this:

apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: APPspec:  template:    metadata:      labels:        name: THIS        app: THAT    spec:      containers:        - image: APP:IMAGE          name: APP          command:          - app-start          env:            - name: DB_HOST              value: "127.0.0.1"            - name: DB_DATABASE              value: "app_db"

And a job that connects to the same database, but with a different "Entrypoint" :

apiVersion: batch/v1kind: Jobmetadata:  name: APP-JOBspec:  template:    metadata:      name: APP-JOB      labels:        app: THAT    spec:      containers:      - image: APP:IMAGE        name: APP-JOB        command:        - app-job        env:          - name: DB_HOST            value: "127.0.0.1"          - name: DB_DATABASE            value: "app_db"

Or the persistent volume approach would look something like this:

apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: APPspec:  template:    metadata:      labels:        name: THIS        app: THAT    spec:      containers:        - image: APP:IMAGE          name: APP          command:          - app-start          volumeMounts:          - mountPath: "/var/www/html"            name: APP-VOLUME      volumes:        - name:  APP-VOLUME          persistentVolumeClaim:            claimName: APP-CLAIM---apiVersion: v1kind: PersistentVolumemetadata:  name: APP-VOLUMEspec:  capacity:    storage: 10Gi  accessModes:    - ReadWriteMany  persistentVolumeReclaimPolicy: Retain  nfs:    path: /app---apiVersion: v1kind: PersistentVolumeClaimmetadata:  name: APP-CLAIMspec:  accessModes:    - ReadWriteMany  resources:    requests:      storage: 10Gi  selector:    matchLabels:      service: app

With a job like this, attaching to the same volume:

apiVersion: batch/v1kind: Jobmetadata:  name: APP-JOBspec:  template:    metadata:      name: APP-JOB      labels:        app: THAT    spec:      containers:      - image: APP:IMAGE        name: APP-JOB        command:        - app-job        volumeMounts:        - mountPath: "/var/www/html"          name: APP-VOLUME    volumes:      - name:  APP-VOLUME        persistentVolumeClaim:          claimName: APP-CLAIM

Create a scheduled pod that uses the Kubernetes API to run the command you want on the target pods, via the exec function. The pod image should contain the client libraries to access the API -- many of these are available or you can build your own.

For example, here is a solution using the Python client that execs to each ZooKeeper pod and runs a database maintenance command:

import timefrom kubernetes import configfrom kubernetes.client import Configurationfrom kubernetes.client.apis import core_v1_apifrom kubernetes.client.rest import ApiExceptionfrom kubernetes.stream import streamimport urllib3config.load_incluster_config()configuration = Configuration()configuration.verify_ssl = Falseconfiguration.assert_hostname = Falseurllib3.disable_warnings()Configuration.set_default(configuration)api = core_v1_api.CoreV1Api()label_selector = 'app=zk,tier=backend'namespace = 'default'resp = api.list_namespaced_pod(namespace=namespace,                               label_selector=label_selector)for x in resp.items:  name = x.spec.hostname  resp = api.read_namespaced_pod(name=name,                                 namespace=namespace)  exec_command = [  '/bin/sh',  '-c',  'opt/zookeeper/bin/zkCleanup.sh -n 10'  ]  resp = stream(api.connect_get_namespaced_pod_exec, name, namespace,              command=exec_command,              stderr=True, stdin=False,              stdout=True, tty=False)  print("============================ Cleanup %s: ============================\n%s\n" % (name, resp if resp else "<no output>"))

and the associated Dockerfile:

FROM ubuntu:18.04ADD ./cleanupZk.py /RUN apt-get update \  && apt-get install -y python-pip \  && pip install kubernetes \  && chmod +x /cleanupZk.pyCMD /cleanupZk.py

Note that if you have an RBAC-enabled cluster, you may need to create a service account and appropriate roles to make this API call possible. A role such as the following is sufficient to list pods and to run exec, such as the example script above requires:

apiVersion: rbac.authorization.k8s.io/v1kind: Rolemetadata:  name: pod-list-exec  namespace: defaultrules:  - apiGroups: [""] # "" indicates the core API group    resources: ["pods"]    verbs: ["get", "list"]  - apiGroups: [""] # "" indicates the core API group    resources: ["pods/exec"]    verbs: ["create", "get"]

An example of the associated cron job:

apiVersion: v1kind: ServiceAccountmetadata:  name: zk-maint  namespace: default---apiVersion: rbac.authorization.k8s.io/v1kind: RoleBindingmetadata:  name: zk-maint-pod-list-exec  namespace: defaultsubjects:- kind: ServiceAccount  name: zk-maint  namespace: defaultroleRef:  kind: Role  name: pod-list-exec  apiGroup: rbac.authorization.k8s.io---apiVersion: batch/v1beta1kind: CronJobmetadata:  name: zk-maint  namespace: default  labels:    app: zk-maint    tier: jobsspec:  schedule: "45 3 * * *"  successfulJobsHistoryLimit: 3  failedJobsHistoryLimit: 1  concurrencyPolicy: Forbid  jobTemplate:    spec:      template:        spec:          containers:          - name: zk-maint            image: myorg/zkmaint:latest          serviceAccountName: zk-maint          restartPolicy: OnFailure          imagePullSecrets:          - name: azure-container-registry

This seems like an anti-pattern. Why can't you just run your worker pod as a job pod?

Regardless you seem pretty convinced you need to do this. Here is what I would do.

Take your worker pod and wrap your shell execution in a simple webservice, it's 10 minutes of work with just about any language. Expose the port and put a service in front of that worker/workers. Then your job pods can simply curl ..svc.cluster.local:/ (unless you've futzed with dns).