How to create custom istio ingress gateway controller?

Okay, I found the answer after looking at the code of Istio installation via helm. So, basically the istio have an official way (but not really documented in their readme.md file) to add additional gateway (ingress and egress gateway). I know that because I found this yaml file in their github repo and read the comment (also looking at the gateway chart template code for the spec and its logic).

So, I solved this by, for example, defining this values-custom-gateway.yaml file:

# Gateways Configuration# By default (if enabled) a pair of Ingress and Egress Gateways will be created for the mesh.# You can add more gateways in addition to the defaults but make sure those are uniquely named# and that NodePorts are not conflicting.# Disable specifc gateway by setting the `enabled` to false.#gateways:  enabled: true  agung-ingressgateway:    namespace: agung-ns    enabled: true    labels:      app: agung-istio-ingressgateway      istio: agung-ingressgateway    replicaCount: 1    autoscaleMin: 1    autoscaleMax: 2    resources: {}      # limits:      #  cpu: 100m      #  memory: 128Mi      #requests:      #  cpu: 1800m      #  memory: 256Mi    loadBalancerIP: ""    serviceAnnotations: {}    type: LoadBalancer #change to NodePort, ClusterIP or LoadBalancer if need be    ports:      ## You can add custom gateway ports    - port: 80      targetPort: 80      name: http2      # nodePort: 31380    - port: 443      name: https      # nodePort: 31390    - port: 31400      name: tcp    secretVolumes:    - name: ingressgateway-certs      secretName: istio-ingressgateway-certs      mountPath: /etc/istio/ingressgateway-certs    - name: ingressgateway-ca-certs      secretName: istio-ingressgateway-ca-certs      mountPath: /etc/istio/ingressgateway-ca-certs

If you take a look at yaml file above, I specified the namespace other than istio-system ns. In this case, we can have a way to customize the TLS and ca cert being used by our custom gateway.Also the agung-ingressgateway as the holder of the custom gateway controller spec is used as the gateway controller's name.

Then, i just install the istio via helm upgrade --install so that helm can intelligently upgrade the istio with additional gateway.

helm upgrade my-istio-release-name <istio-chart-folder> --install

Once it upgrades successfully, I can specify custom selector to my Gateway:

---apiVersion: networking.istio.io/v1alpha3kind: Gatewaymetadata:  name: agung-gateway  namespace: agung-nsspec:  selector:    app: agung-istio-ingressgateway # use custom gateway    # istio: ingressgateway # use Istio default gateway implementation  servers:  - port:      number: 80      name: http      protocol: HTTP    hosts:    - "*"  - port:      number: 443      name: https      protocol: HTTPS    tls:      mode: SIMPLE      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt      privateKey: /etc/istio/ingressgateway-certs/tls.key    hosts:    - "*"

This is what I use in Istio 1.4.

To generate the new istio-ingressgateway Deployment, Service and ServiceAccount in your own namespace (bookinfo in this example)

helm template install/kubernetes/helm/istio/ \  --namespace bookinfo \  --set global.istioNamespace=istio-system \  -x charts/gateways/templates/deployment.yaml \  -x charts/gateways/templates/service.yaml  \  -x charts/gateways/templates/serviceaccount.yaml \  --set gateways.istio-ingressgateway.enabled=true \  --set gateways.istio-egressgateway.enabled=false \  --set gateways.istio-ingressgateway.labels.app=custom-istio-ingressgateway \  --set gateways.istio-ingressgateway.labels.istio=custom-ingressgateway \  > customingress.yaml

then, apply the generated file:

kubectl apply -f customingress.yaml

Now, you can refer to this from your Gateway resource like:

apiVersion: networking.istio.io/v1alpha3kind: Gatewaymetadata:  name: bookinfo-gatewayspec:  selector:    istio: custom-ingressgateway # use the CUSTOM istio controller  servers:  - port:      number: 80      name: http      protocol: HTTP    hosts:    - "*"

You can set custom Service annotations by adding it to the helm template command like this:

 --set gateways.istio-ingressgateway.serviceAnnotations.'service\.kubernetes\.io/ibm-load-balancer-cloud-provider-ip-type'=private \

I tried this and worked:

---# Source: istio/charts/gateways/templates/serviceaccount.yamlapiVersion: v1kind: ServiceAccountmetadata:  name: beta-ingressgateway-service-account  namespace: beta  labels:    app: ingressgateway-beta------# Source: istio/charts/gateways/templates/clusterrole.yamlapiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRolemetadata:  labels:    app: gateways  name: ingressgateway-betarules:- apiGroups: ["extensions"]  resources: ["thirdpartyresources", "virtualservices", "destinationrules", "gateways"]  verbs: ["get", "watch", "list", "update"]------# Source: istio/charts/gateways/templates/clusterrolebindings.yamlapiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRoleBindingmetadata:  name: ingressgateway-betaroleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: ingressgateway-betasubjects:  - kind: ServiceAccount    name: beta-ingressgateway-service-account    namespace: beta------# Source: istio/charts/gateways/templates/service.yamlapiVersion: v1kind: Servicemetadata:  name: ingressgateway-beta  namespace: beta  annotations:  labels:    istio: ingressgateway-betaspec:  type: LoadBalancer  selector:    istio: ingressgateway-beta  ports:    -      name: http      port: 80      targetPort: 80    -      name: https      port: 443      targetPort: 443------# Source: istio/charts/gateways/templates/deployment.yamlapiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: ingressgateway-beta  namespace: beta  labels:    istio: ingressgateway-betaspec:  replicas: 1  template:    metadata:      labels:        istio: ingressgateway-beta      annotations:        sidecar.istio.io/inject: "false"        scheduler.alpha.kubernetes.io/critical-pod: ""    spec:      serviceAccountName: beta-ingressgateway-service-account      tolerations:      - key: "env"        operator: "Equal"        value: "beta"        effect: "NoSchedule"      nodeSelector:        env: beta      containers:        - name: istio-proxy          image: "ISTIO_PROXY_IMAGE"          imagePullPolicy: IfNotPresent          ports:            - containerPort: 80            - containerPort: 443          args:          - proxy          - router          - -v          - "2"          - --discoveryRefreshDelay          - '1s' #discoveryRefreshDelay          - --drainDuration          - '45s' #drainDuration          - --parentShutdownDuration          - '1m0s' #parentShutdownDuration          - --connectTimeout          - '10s' #connectTimeout          - --serviceCluster          - ingressgateway-beta          - --zipkinAddress          - zipkin.istio-system:9411          - --proxyAdminPort          - "15000"          - --controlPlaneAuthPolicy          - NONE          - --discoveryAddress          - istio-pilot.istio-system:8080          resources:            requests:              cpu: 10m          env:          - name: POD_NAME            valueFrom:              fieldRef:                apiVersion: v1                fieldPath: metadata.name          - name: POD_NAMESPACE            valueFrom:              fieldRef:                apiVersion: v1                fieldPath: metadata.namespace          - name: INSTANCE_IP            valueFrom:              fieldRef:                apiVersion: v1                fieldPath: status.podIP          - name: ISTIO_META_POD_NAME            valueFrom:              fieldRef:                fieldPath: metadata.name          volumeMounts:          - name: istio-certs            mountPath: /etc/certs            readOnly: true          - name: ingressgateway-beta-certs            mountPath: "/etc/istio/ingressgateway-beta-certs"            readOnly: true          - name: ingressgateway-beta-ca-certs            mountPath: "/etc/istio/ingressgateway-beta-ca-certs"            readOnly: true      volumes:      - name: istio-certs        secret:          secretName: istio.beta-ingressgateway-service-account          optional: true      - name: ingressgateway-beta-certs        secret:          secretName: "istio-ingressgateway-beta-certs"          optional: true      - name: ingressgateway-beta-ca-certs        secret:          secretName: "istio-ingressgateway-beta-ca-certs"          optional: true      affinity:        nodeAffinity:          requiredDuringSchedulingIgnoredDuringExecution:            nodeSelectorTerms:            - matchExpressions:              - key: beta.kubernetes.io/arch                operator: In                values:                - amd64                - ppc64le                - s390x          preferredDuringSchedulingIgnoredDuringExecution:          - weight: 2            preference:              matchExpressions:              - key: beta.kubernetes.io/arch                operator: In                values:                - amd64          - weight: 2            preference:              matchExpressions:              - key: beta.kubernetes.io/arch                operator: In                values:                - ppc64le          - weight: 2            preference:              matchExpressions:              - key: beta.kubernetes.io/arch                operator: In                values:                - s390x------# Source: istio/charts/gateways/templates/autoscale.yaml# Source: istio/charts/gateways/templates/autoscale.yamlapiVersion: autoscaling/v2beta1kind: HorizontalPodAutoscalermetadata:    name: ingressgateway-beta    namespace: betaspec:    maxReplicas: 5    minReplicas: 1    scaleTargetRef:      apiVersion: apps/v1beta1      kind: Deployment      name: ingressgateway-beta    metrics:    - type: Resource      resource:        name: cpu        targetAverageUtilization: 80---

remember to replace ISTIO_PROXY_IMAGE, nodeSelector and tolerations