.gitignore

@@ -1,7 +1,6 @@
 # Vim
 *.swp
 
-output
 .vscode
 .DS_Store
 .idea

CHANGELOG.md

@@ -0,0 +1,46 @@
+# Changelog
+
+## KubeZero - 2.18 ( Argoless )
+
+### High level / Admin changes
+- ArgoCD is now optional and NOT required nor used during initial cluster bootstrap
+- the bootstrap process now uses the same config and templates as the optional ArgoCD applications later on
+- the bootstrap is can now be restarted at any time and considerably faster
+- the top level KubeZero config for the ArgoCD app-of-apps is now also maintained via the gitops workflow. Changes can be applied by a simple git push rather than manual scripts
+
+### Calico
+- version bump
+
+### Cert-manager
+- local issuers are now cluster issuer to allow them being used across namespaces
+- all cert-manager resources moved into the cert-manager namespace
+- version bump to 1.10
+
+### Kiam
+- set priorty class to cluster essential
+- certificates are now issued by the cluster issuer
+
+### EBS / EFS
+- version bump
+
+### Istio
+- istio operator removed, deployment migrated to helm, various cleanups
+- version bump to 1.8
+- all ingress resources are now in the dedicated new namespace istio-ingress ( deployed via separate kubezero chart istio-ingress)
+- set priorty class of ingress components to cluster essential
+
+### Logging
+- ES/Kibana version bump to 7.10  
+- ECK operator is now installed on demand in logging ns
+- Custom event fields configurable via new fluent-bit chart    
+  e.g. clustername could be added to each event allowing easy filtering in case multiple clusters stream events into a single central ES cluster
+
+### ArgoCD
+- version bump, new app of app architecure
+
+### Metrics
+- version bump
+- all servicemonitor resources are now in the same namespaces as the respective apps to avoid deployments across multiple namespaces
+
+### upstream Kubernetes 1.18
+https://sysdig.com/blog/whats-new-kubernetes-1-18/

CHANGES.md

@@ -1,15 +0,0 @@
-# CFN / Platform
-- Kube to 1.17
-- Kube-proxy uses ipvs
-- metrics support for kube-proxy
-- no reliance on custom resource for S3 buckets anymore
-
-
-# Kubezero
-- fully automated one command bootstrap incl. all kubezero components 
-- migrated from kube-prometheuss to prometheus-operator helm charts for metrics
-- latest Grafana incl. peristence
-- kube-prometheus adapter improvements / customizations
-- integrated EFS CSI driver into Kubezero
-- prometheus itself can be exposed via istio ingress on demand to ease development of custom metrics
-- backup script to export all cert-manager items between clusters

Makefile

@@ -2,16 +2,12 @@ BUCKET ?= zero-downtime
 BUCKET_PREFIX ?= /cloudbender/distfiles
 FILES ?= distfiles.txt
 
-.PHONY: clean update docs
+.PHONY: update docs
 
 all: update
 
-
-clean:
-	rm -f kube*.tgz
-
 update:
-	./script/update_helm.sh
+	./scripts/update_helm.sh
 
 docs:
 	for c in charts/*; do \
@@ -19,3 +15,6 @@ docs:
 		[[ $$c =~ "kubeadm" ]] && continue ; \
 		helm-docs -c $$c ; \
 	done
+
+publish:
+	./scripts/publish.sh

README.md

@@ -1,43 +1,47 @@
 KubeZero - Zero Down Time Kubernetes platform
 ========================
-KubeZero is a pre-configured collection of components deployed onto a bare Kubernetes cluster.
+KubeZero is a Kubernetes distribution providing an integrated container platform so you can focus on your applications.
-All chosen components are 100% organic OpenSource.
 
-# Design goals
+# Design philosophy
 
-- Cloud provider agnostic, bare-metal / self-hosted possible
+- Cloud provider agnostic, bare-metal/self-hosted
-- No vendor lock in
+- No vendor lock in, most components are optional and could be exchanged
-- No closed source solutions
+- Organic Open Source / open and permissive licenses over closed-source solutions
 - No premium services / subscriptions required
-- Staying to upstream projects as close as possible
+- Staying and contributing back to upstream projects as much as possible
-- Minimal custom code
+
-- Work within each community / give back
 
 # Version / Support Matrix
 
-| KubeZero \ Kubernetes Version          | v1.17 | v1.18 | v1.19 | v1.20 | EOL         |
+| KubeZero \ Kubernetes Version          | v1.18 | v1.19 | v1.20 | EOL         |
-|----------------------------------------|-------|-------|-------|-------|-------------|
+|----------------------------------------|-------|-------|-------|-------------|
-| master branch                          | no    | yes   | beta  | no    |             |
+| master branch                          | yes   | yes   | beta  |             |
-| stable branch                          | no    | yes   | no    | no    |             |
+| stable branch                          | yes   | yes   | no    |             |
-| v2.18.0                                | no    | yes   | no    | no    | 30 Apr 2021 |
+| v2.19.0                                | yes   | yes   | no    | 30 Jun 2021 |
-| v1                                     | yes   | no    | no    | no    | 30 Jan 2021 |
+| v2.18.0                                | yes   | no    | no    | 30 Apr 2021 |
+
+# Architecure
+![aws_architecture](docs/aws_architecture.png)
 
 
-## General
+# Components
-- Container runtime cri-o rather than Docker for improved security and performance
 
+## Container runtime
+- cri-o rather than Docker for improved security and performance
 
 ## Control plane
 - support for single node control plane for small clusters / test environments to reduce costs
 - access to control plane from within the VPC only by default ( VPN access required for Admin tasks )
 - controller nodes are used for various platform admin controllers / operators to reduce costs and noise on worker nodes
-- integrated ArgoCD Gitops controller
+
+## GitOps
+- full ArgoCD support and integration (optional)
 
 ## AWS IAM access control
 - Kiam allowing IAM roles per pod
 - IAM roles are assumed / requested and cached on controller nodes for improved security
-- blocking access to meta-data service on all nodes
+- access to meta-data services is blocked / controlled on all nodes
-- IAM roles are maintained/ automated and tracked via CFN templates
+- core IAM roles are maintained via CFN templates
 
 ## Network
 - Calico using VxLAN incl. increased MTU
@@ -66,7 +70,7 @@ All chosen components are 100% organic OpenSource.
 
 ## Logging
 - all container logs are enhanced with Kubernetes metadata to provide context for each message
-- flexible ElasticSearch setup via ECK operator to ease maintenance and reduce required admin knowledge, incl automated backups to S3
+- flexible ElasticSearch setup, leveraging the ECK operator, for easy maintenance & minimal admin knowledge required, incl. automated backups to S3
-- Kibana allowing easy search and dashboards for all logs, incl. pre configured index templates and index management to reduce costs
+- Kibana allowing easy search and dashboards for all logs, incl. pre configured index templates and index management
-- fluentd central log ingress service allowing additional parsing and queuing to improved reliability
+- central fluentd service providing queuing during highload as well as additional parsing options
 - lightweight fluent-bit agents on each node requiring minimal resources forwarding logs secure via SSL to fluentd

charts/kubeadm/Chart.yaml

@@ -2,7 +2,7 @@ apiVersion: v2
 name: kubeadm
 description: KubeZero Kubeadm golden config
 type: application
-version: 1.18.14
+version: 1.19.8
 home: https://kubezero.com
 icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png
 keywords:
@@ -10,4 +10,4 @@ keywords:
   - kubeadm
 maintainers:
   - name: Quarky9
-kubeVersion: ">= 1.16.0"
+kubeVersion: ">= 1.18.0"

charts/kubeadm/README.md.gotmpl

@@ -17,6 +17,14 @@ Installs the Istio control plane
 
 {{ template "chart.valuesSection" . }}
 
+## Changes for 1.19
+
+### Logging to json of control plane components
+- https://github.com/kubernetes/website/blob/dev-1.19/content/en/docs/concepts/cluster-administration/system-logs.md
+
+### PodTopologySpread
+- https://kubernetes.io/blog/2020/05/introducing-podtopologyspread/#podtopologyspread-defaults
+
 ## Resources
 
 - https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/

charts/kubeadm/templates/ClusterConfiguration.yaml

@@ -10,30 +10,31 @@ networking:
 etcd:
   local:
     extraArgs:
-      listen-metrics-urls: "http://0.0.0.0:2381"
+      listen-metrics-urls: "http://{{ .Values.listenAddress }}:2381"
+      logger: "zap"
+      # log-level: "warn"
       {{- with .Values.etcdExtraArgs }}
       {{- toYaml . | nindent 6 }}
       {{- end }}
 controllerManager:
   extraArgs:
     profiling: "false"
-    bind-address: 0.0.0.0
+    bind-address: {{ .Values.listenAddress }}
     terminated-pod-gc-threshold: "300"
     leader-elect: {{ .Values.clusterHighAvailable | quote }}
-    # Default anyways but make kube-bench happy
+    logging-format: json
-    feature-gates: "RotateKubeletServerCertificate=true"
 scheduler:
   extraArgs:
     profiling: "false"
-    bind-address: 0.0.0.0
+    bind-address: {{ .Values.listenAddress }}
     leader-elect: {{ .Values.clusterHighAvailable | quote }}
+    logging-format: json
 apiServer:
   certSANs:
   - {{ regexSplit ":" .Values.apiEndpoint -1 | first }}
   extraArgs:
     etcd-servers: {{ .Values.allEtcdEndpoints }}
     profiling: "false"
-    feature-gates: "CSIMigration=true,CSIMigrationAWS=true,CSIMigrationAWSComplete=true"
     audit-log-path: "/var/log/kubernetes/audit.log"
     audit-policy-file: /etc/kubernetes/apiserver/audit-policy.yaml
     audit-log-maxage: "7"
@@ -41,11 +42,17 @@ apiServer:
     audit-log-maxbackup: "3"
     tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
     admission-control-config-file: /etc/kubernetes/apiserver/admission-configuration.yaml
+    {{- if eq .Values.platform "aws" }}
     authentication-token-webhook-config-file: /etc/kubernetes/apiserver/aws-iam-authenticator.yaml
+    feature-gates: "CSIMigrationAWS=true,CSIMigrationAWSComplete=true,DefaultPodTopologySpread=true"
+    {{- else }}
+    feature-gates: "DefaultPodTopologySpread=true"
+    {{- end }}
     enable-admission-plugins: NodeRestriction,EventRateLimit
     {{- if .Values.clusterHighAvailable }}
-    goaway-chance: ".001"
+    # goaway-chance: ".001"
     {{- end }}
+    logging-format: json
     {{- with .Values.apiExtraArgs }}
     {{- toYaml . | nindent 4 }}
     {{- end }}

charts/kubeadm/templates/JoinConfiguration.yaml

@@ -1,18 +1,16 @@
 apiVersion: kubeadm.k8s.io/v1beta2
 kind: JoinConfiguration
-metadata:
-  name: kubezero-joinconfiguration
 discovery:
-  bootstrapToken:
+  file:
-    apiServerEndpoint: {{ .Values.apiEndpoint }}
+    kubeConfigPath: /root/.kube/config
-    token: {{ .Values.joinToken }}
+controlPlane:
-    caCertHashes:
+  localAPIEndpoint:
-    - "{{ .Values.caCertHash }}"
+    advertiseAddress: IP_ADDRESS
+    bindPort: {{ regexSplit ":" .Values.apiEndpoint -1 | last }}
 nodeRegistration:
   ignorePreflightErrors:
   - DirAvailable--var-lib-etcd
+  - FileAvailable--etc-kubernetes-pki-ca.crt
   - Swap
-controlPlane:
+  kubeletExtraArgs:
-  localAPIEndpoint:
+    node-labels: {{ .Values.nodeLabels | quote }}
-    advertiseAddress: {{ .Values.ipAddress }}
-    bindPort: {{ regexSplit ":" .Values.apiEndpoint -1 | last }}

charts/kubeadm/templates/KubeProxyConfiguration.yaml

@@ -2,5 +2,5 @@ apiVersion: kubeproxy.config.k8s.io/v1alpha1
 kind: KubeProxyConfiguration
 metadata:
   name: kubezero-kubeproxyconfiguration
-metricsBindAddress: "0.0.0.0:10249"
+metricsBindAddress: "{{ .Values.listenAddress }}:10249"
 mode: ""

charts/kubeadm/templates/KubeletConfiguration.yaml

@@ -3,17 +3,35 @@ kind: KubeletConfiguration
 metadata:
   name: kubezero-kubeletconfiguration
 failSwapOn: false
+cgroupDriver: cgroupfs
+logging:
+  # We already have syslog-ng logging json
+  # format: json
 hairpinMode: hairpin-veth
+{{- if .Values.systemd }}
 resolvConf: /run/systemd/resolve/resolv.conf
-protectKernelDefaults: true
+{{- end }}
+protectKernelDefaults: {{ .Values.protectKernelDefaults }}
 eventRecordQPS: 0
 # Breaks kubelet at boot time
 # tlsCertFile: /var/lib/kubelet/pki/kubelet.crt
 # tlsPrivateKeyFile: /var/lib/kubelet/pki/kubelet.key
 tlsCipherSuites: [TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256]
 featureGates:
-  # Default anyways but make kube-bench happy
+  CustomCPUCFSQuotaPeriod: true
-  RotateKubeletServerCertificate: true
+{{- if eq .Values.platform "aws" }}
-  CSIMigration: true
   CSIMigrationAWS: true
   CSIMigrationAWSComplete: true
+{{- end }}
+# Minimal unit is 50m per pod
+podsPerCore: 20
+# Basic OS on Ubuntu 20.04 incl. crio
+systemReserved:
+  memory: 256Mi
+# This should be dynamic based on number of maxpods and available cores
+# https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#memory_cpu
+# Below are default minimal for 2 cores and minimum kubelet 
+kubeReserved:
+  cpu: 70m
+  memory: 128Mi
+# cpuCFSQuotaPeriod: 10ms

charts/kubeadm/templates/admin-kubectl.yaml

@@ -0,0 +1,27 @@
+{{- if eq .Values.platform "aws" }}
+apiVersion: v1
+kind: Config
+clusters:
+- cluster:
+    server: https://{{ .Values.apiEndpoint }}
+  name: {{ .Values.clusterName }}
+contexts:
+- context:
+    cluster: {{ .Values.clusterName }}
+    user: kubernetes-admin
+  name: kubernetes-admin@{{ .Values.clusterName }}
+current-context: kubernetes-admin@{{ .Values.clusterName }}
+preferences: {}
+users:
+- name: kubernetes-admin
+  user:
+    exec:
+      apiVersion: client.authentication.k8s.io/v1alpha1
+      command: aws-iam-authenticator
+      args:
+        - "token"
+        - "-i"
+        - "{{ .Values.clusterName }}"
+        - "-r"
+        - "{{ .Values.kubeAdminRole }}"
+{{- end }}

charts/kubeadm/templates/aws-iam-authenticator/crds.yaml

@@ -1,3 +1,4 @@
+{{- if eq .Values.platform "aws" }}
 apiVersion: apiextensions.k8s.io/v1beta1
 kind: CustomResourceDefinition
 metadata:
@@ -30,3 +31,4 @@ spec:
               type: array
               items:
                 type: string
+{{- end }}

charts/kubeadm/templates/aws-iam-authenticator/deployment.yaml

@@ -1,3 +1,4 @@
+{{- if eq .Values.platform "aws" }}
 kind: ClusterRole
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
@@ -51,7 +52,7 @@ metadata:
 
 ---
 kind: ClusterRoleBinding
-apiVersion: rbac.authorization.k8s.io/v1beta1
+apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: aws-iam-authenticator
   namespace: kube-system
@@ -116,7 +117,7 @@ spec:
 
       containers:
       - name: aws-iam-authenticator
-        image: public.ecr.aws/x8h8t2o1/aws-iam-authenticator:v0.5.2
+        image: public.ecr.aws/zero-downtime/aws-iam-authenticator:v0.5.2
         args:
         - server
         - --backend-mode=CRD,MountedFile
@@ -136,7 +137,7 @@ spec:
             cpu: 10m
           limits:
             memory: 20Mi
-            cpu: 100m
+            #cpu: 100m
 
         volumeMounts:
         - name: config
@@ -151,14 +152,4 @@ spec:
       - name: state
         hostPath:
           path: /var/aws-iam-authenticator/
----
+{{- end }}
-apiVersion: iamauthenticator.k8s.aws/v1alpha1
-kind: IAMIdentityMapping
-metadata:
-  name: kubernetes-admin
-spec:
-  # Arn of the User or Role to be allowed to authenticate
-  arn: {{ .Values.kubeAdminRole }}
-  username: kubernetes-admin
-  groups:
-  - system:masters

charts/kubeadm/templates/aws-iam-authenticator/mappings.yaml

@@ -0,0 +1,26 @@
+{{- if eq .Values.platform "aws" }}
+# Controller role for consistency, similar to kubeadm admin.conf
+apiVersion: iamauthenticator.k8s.aws/v1alpha1
+kind: IAMIdentityMapping
+metadata:
+  name: kubezero-worker-nodes
+spec:
+  arn: {{ .Values.WorkerNodeRole }}
+  username: system:node:{{ "{{" }}EC2PrivateDNSName{{ "}}" }}
+  groups:
+  # For now use masters, define properly with 1.20
+  - system:masters
+  - system:nodes
+  - system:bootstrappers
+---
+# Admin Role for remote access
+apiVersion: iamauthenticator.k8s.aws/v1alpha1
+kind: IAMIdentityMapping
+metadata:
+  name: kubernetes-admin
+spec:
+  arn: {{ .Values.kubeAdminRole }}
+  username: kubernetes-admin
+  groups:
+  - system:masters
+{{- end }}

charts/kubeadm/templates/device-plugins/fuse-device-plugin.yaml

@@ -0,0 +1,30 @@
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: fuse-device-plugin
+  namespace: kube-system
+spec:
+  selector:
+    matchLabels:
+      name: fuse-device-plugin
+  template:
+    metadata:
+      labels:
+        name: fuse-device-plugin
+    spec:
+      hostNetwork: true
+      containers:
+      - image: public.ecr.aws/zero-downtime/fuse-device-plugin:v1.0
+        # imagePullPolicy: Always
+        name: fuse-device-plugin
+        securityContext:
+          allowPrivilegeEscalation: false
+          capabilities:
+            drop: ["ALL"]
+        volumeMounts:
+          - name: device-plugin
+            mountPath: /var/lib/kubelet/device-plugins
+      volumes:
+        - name: device-plugin
+          hostPath:
+            path: /var/lib/kubelet/device-plugins

charts/kubeadm/templates/patches/coredns0.yaml

@@ -0,0 +1,24 @@
+spec:
+  replicas: {{ ternary 3 1 .Values.clusterHighAvailable }}
+  template:
+    spec:
+      containers:
+      - name: coredns
+        resources:
+          requests:
+            cpu: 100m
+            memory: 32Mi
+          limits:
+            memory: 128Mi
+      nodeSelector:
+        node-role.kubernetes.io/master: ""
+      affinity:
+        podAntiAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+          - labelSelector:
+              matchExpressions:
+              - key: k8s-app
+                operator: In
+                values:
+                - kube-dns
+            topologyKey: "kubernetes.io/hostname"

charts/kubeadm/templates/patches/etcd0.yaml

@@ -0,0 +1,7 @@
+spec:
+  containers:
+  - name: etcd
+    resources:
+      requests:
+        cpu: 250m
+        memory: 192Mi

charts/kubeadm/templates/patches/kube-apiserver0.yaml

@@ -0,0 +1,7 @@
+spec:
+  containers:
+  - name: kube-apiserver
+    resources:
+      requests:
+        cpu: 250m
+        memory: 1Gi

charts/kubeadm/templates/patches/kube-controller-manager0.yaml

@@ -0,0 +1,7 @@
+spec:
+  containers:
+  - name: kube-controller-manager
+    resources:
+      requests:
+        cpu: 200m
+        memory: 128Mi

charts/kubeadm/templates/patches/kube-scheduler0.yaml

@@ -0,0 +1,7 @@
+spec:
+  containers:
+  - name: kube-scheduler
+    resources:
+      requests:
+        cpu: 100m
+        memory: 64Mi

charts/kubeadm/values.yaml

@@ -1,11 +1,17 @@
-clusterVersion: 1.18.0
+clusterVersion: 1.19.0
+listenAddress: 0.0.0.0
 clusterName: pleasechangeme
 apiEndpoint: kube-api.changeme.org:6443
 etcdExtraArgs: {}
-  # Enable for > 1.18
-  # unsafe-no-fsync: "true"
 apiExtraArgs: {}
 clusterHighAvailable: false
 allEtcdEndpoints: ""
-joinToken: ""
+# supported values aws,bare-metal
-caCertHash: ""
+platform: "aws"
+# Set to false for openrc, eg. on Gentoo or Alpine
+systemd: true
+protectKernelDefaults: true
+
+WorkerNodeRole: "arn:aws:iam::000000000000:role/KubernetesNode"
+WorkerIamRole: "arn:aws:iam::000000000000:role/KubernetesNode"
+KubeAdminRole: "arn:aws:iam::000000000000:role/KubernetesNode"

charts/kubezero-argocd/Chart.yaml

@@ -1,7 +1,7 @@
 apiVersion: v2
 description: KubeZero ArgoCD Helm chart to install ArgoCD itself and the KubeZero ArgoCD Application
 name: kubezero-argocd
-version: 0.6.1
+version: 0.7.1
 home: https://kubezero.com
 icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png
 keywords:
@@ -15,6 +15,6 @@ dependencies:
     version: ">= 0.1.3"
     repository: https://zero-down-time.github.io/kubezero/
   - name: argo-cd
-    version: 2.9.5
+    version: 2.17.4
     repository: https://argoproj.github.io/argo-helm
-kubeVersion: ">= 1.16.0"
+kubeVersion: ">= 1.18.0"

charts/kubezero-argocd/README.md

@@ -1,6 +1,6 @@
 # kubezero-argocd
 
-![Version: 0.6.1](https://img.shields.io/badge/Version-0.6.1-informational?style=flat-square)
+![Version: 0.7.1](https://img.shields.io/badge/Version-0.7.1-informational?style=flat-square)
 
 KubeZero ArgoCD Helm chart to install ArgoCD itself and the KubeZero ArgoCD Application
 
@@ -14,11 +14,11 @@ KubeZero ArgoCD Helm chart to install ArgoCD itself and the KubeZero ArgoCD Appl
 
 ## Requirements
 
-Kubernetes: `>= 1.16.0`
+Kubernetes: `>= 1.18.0`
 
 | Repository | Name | Version |
 |------------|------|---------|
-| https://argoproj.github.io/argo-helm | argo-cd | 2.9.5 |
+| https://argoproj.github.io/argo-helm | argo-cd | 2.17.4 |
 | https://zero-down-time.github.io/kubezero/ | kubezero-lib | >= 0.1.3 |
 
 ## Values
@@ -35,7 +35,7 @@ Kubernetes: `>= 1.16.0`
 | argo-cd.controller.resources.requests.cpu | string | `"100m"` |  |
 | argo-cd.controller.resources.requests.memory | string | `"256Mi"` |  |
 | argo-cd.dex.enabled | bool | `false` |  |
-| argo-cd.global.image.tag | string | `"v1.7.10"` |  |
+| argo-cd.global.image.tag | string | `"v1.8.7"` |  |
 | argo-cd.installCRDs | bool | `false` |  |
 | argo-cd.repoServer.logFormat | string | `"json"` |  |
 | argo-cd.repoServer.metrics.enabled | bool | `false` |  |

charts/kubezero-argocd/values.yaml

@@ -42,7 +42,7 @@ argo-cd:
 
   global:
     image:
-      tag: v1.7.10
+      tag: v1.8.7
 
   controller:
     args:

charts/kubezero-aws-ebs-csi-driver/Chart.yaml

@@ -2,8 +2,8 @@ apiVersion: v2
 name: kubezero-aws-ebs-csi-driver
 description: KubeZero Umbrella Chart for aws-ebs-csi-driver
 type: application
-version: 0.3.4
+version: 0.4.4
-appVersion: 0.8.0
+appVersion: 0.9.0
 home: https://kubezero.com
 icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png
 sources:
@@ -18,9 +18,9 @@ maintainers:
   - name: Quarky9
 dependencies:
   - name: aws-ebs-csi-driver
-    version: 0.7.1
+    version: 0.9.14
     repository: https://kubernetes-sigs.github.io/aws-ebs-csi-driver
   - name: kubezero-lib
     version: ">= 0.1.3"
     repository: https://zero-down-time.github.io/kubezero/
-kubeVersion: ">= 1.16.0"
+kubeVersion: ">= 1.18.0"

charts/kubezero-aws-ebs-csi-driver/README.md

@@ -1,6 +1,6 @@
 # kubezero-aws-ebs-csi-driver
 
-![Version: 0.3.4](https://img.shields.io/badge/Version-0.3.4-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 0.8.0](https://img.shields.io/badge/AppVersion-0.8.0-informational?style=flat-square)
+![Version: 0.4.4](https://img.shields.io/badge/Version-0.4.4-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 0.9.0](https://img.shields.io/badge/AppVersion-0.9.0-informational?style=flat-square)
 
 KubeZero Umbrella Chart for aws-ebs-csi-driver
 
@@ -19,11 +19,11 @@ KubeZero Umbrella Chart for aws-ebs-csi-driver
 
 ## Requirements
 
-Kubernetes: `>= 1.16.0`
+Kubernetes: `>= 1.18.0`
 
 | Repository | Name | Version |
 |------------|------|---------|
-| https://kubernetes-sigs.github.io/aws-ebs-csi-driver | aws-ebs-csi-driver | 0.7.1 |
+| https://kubernetes-sigs.github.io/aws-ebs-csi-driver | aws-ebs-csi-driver | 0.9.14 |
 | https://zero-down-time.github.io/kubezero/ | kubezero-lib | >= 0.1.3 |
 
 ## IAM Role
@@ -41,14 +41,29 @@ This class is by default also set as default storage class.
 
 | Key | Type | Default | Description |
 |-----|------|---------|-------------|
-| StorageClass.create | bool | `true` |  |
+| aws-ebs-csi-driver.enableVolumeResizing | bool | `true` |  |
-| StorageClass.default | bool | `true` |  |
-| aws-ebs-csi-driver.enableVolumeResizing | bool | `false` |  |
 | aws-ebs-csi-driver.enableVolumeScheduling | bool | `true` |  |
-| aws-ebs-csi-driver.enableVolumeSnapshot | bool | `false` |  |
+| aws-ebs-csi-driver.enableVolumeSnapshot | bool | `true` |  |
 | aws-ebs-csi-driver.extraVolumeTags | object | `{}` | Optional tags to be added to each EBS volume |
+| aws-ebs-csi-driver.logLevel | int | `1` |  |
 | aws-ebs-csi-driver.nodeSelector."node-role.kubernetes.io/master" | string | `""` |  |
 | aws-ebs-csi-driver.podAnnotations | object | `{}` | iam.amazonaws.com/role: <IAM role ARN> to assume |
 | aws-ebs-csi-driver.replicaCount | int | `1` |  |
+| aws-ebs-csi-driver.resources.limits.memory | string | `"40Mi"` |  |
+| aws-ebs-csi-driver.resources.requests.cpu | string | `"10m"` |  |
+| aws-ebs-csi-driver.resources.requests.memory | string | `"24Mi"` |  |
+| aws-ebs-csi-driver.storageClasses[0].allowVolumeExpansion | bool | `true` |  |
+| aws-ebs-csi-driver.storageClasses[0].name | string | `"ebs-sc-gp2-xfs"` |  |
+| aws-ebs-csi-driver.storageClasses[0].parameters."csi.storage.k8s.io/fstype" | string | `"xfs"` |  |
+| aws-ebs-csi-driver.storageClasses[0].parameters.encrypted | string | `"true"` |  |
+| aws-ebs-csi-driver.storageClasses[0].parameters.type | string | `"gp2"` |  |
+| aws-ebs-csi-driver.storageClasses[0].volumeBindingMode | string | `"WaitForFirstConsumer"` |  |
+| aws-ebs-csi-driver.storageClasses[1].allowVolumeExpansion | bool | `true` |  |
+| aws-ebs-csi-driver.storageClasses[1].annotations."storageclass.kubernetes.io/is-default-class" | string | `"true"` |  |
+| aws-ebs-csi-driver.storageClasses[1].name | string | `"ebs-sc-gp3-xfs"` |  |
+| aws-ebs-csi-driver.storageClasses[1].parameters."csi.storage.k8s.io/fstype" | string | `"xfs"` |  |
+| aws-ebs-csi-driver.storageClasses[1].parameters.encrypted | string | `"true"` |  |
+| aws-ebs-csi-driver.storageClasses[1].parameters.type | string | `"gp3"` |  |
+| aws-ebs-csi-driver.storageClasses[1].volumeBindingMode | string | `"WaitForFirstConsumer"` |  |
 | aws-ebs-csi-driver.tolerations[0].effect | string | `"NoSchedule"` |  |
 | aws-ebs-csi-driver.tolerations[0].key | string | `"node-role.kubernetes.io/master"` |  |

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/.helmignore

@@ -0,0 +1,22 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/Chart.yaml

@@ -0,0 +1,18 @@
+apiVersion: v1
+appVersion: 0.9.0
+description: A Helm chart for AWS EBS CSI Driver
+home: https://github.com/kubernetes-sigs/aws-ebs-csi-driver
+keywords:
+- aws
+- ebs
+- csi
+kubeVersion: '>=1.17.0-0'
+maintainers:
+- email: chengpan@amazon.com
+  name: leakingtapan
+- name: krmichel
+  url: https://github.com/krmichel
+name: aws-ebs-csi-driver
+sources:
+- https://github.com/kubernetes-sigs/aws-ebs-csi-driver
+version: 0.9.14

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/NOTES.txt

@@ -0,0 +1,3 @@
+To verify that aws-ebs-csi-driver has started, run:
+
+    kubectl get pod -n {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "aws-ebs-csi-driver.name" . }},app.kubernetes.io/instance={{ .Release.Name }}"

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/_helpers.tpl

@@ -0,0 +1,69 @@
+{{/* vim: set filetype=mustache: */}}
+{{/*
+Expand the name of the chart.
+*/}}
+{{- define "aws-ebs-csi-driver.name" -}}
+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+
+{{/*
+Create a default fully qualified app name.
+We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
+If release name contains chart name it will be used as a full name.
+*/}}
+{{- define "aws-ebs-csi-driver.fullname" -}}
+{{- if .Values.fullnameOverride -}}
+{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
+{{- else -}}
+{{- $name := default .Chart.Name .Values.nameOverride -}}
+{{- if contains $name .Release.Name -}}
+{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
+{{- else -}}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+{{- end -}}
+{{- end -}}
+
+{{/*
+Create chart name and version as used by the chart label.
+*/}}
+{{- define "aws-ebs-csi-driver.chart" -}}
+{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+
+{{/*
+Common labels
+*/}}
+{{- define "aws-ebs-csi-driver.labels" -}}
+{{ include "aws-ebs-csi-driver.selectorLabels" . }}
+{{- if ne .Release.Name "kustomize" }}
+helm.sh/chart: {{ include "aws-ebs-csi-driver.chart" . }}
+{{- if .Chart.AppVersion }}
+app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
+{{- end }}
+app.kubernetes.io/managed-by: {{ .Release.Service }}
+{{- end }}
+{{- end -}}
+
+{{/*
+Common selector labels
+*/}}
+{{- define "aws-ebs-csi-driver.selectorLabels" -}}
+app.kubernetes.io/name: {{ include "aws-ebs-csi-driver.name" . }}
+{{- if ne .Release.Name "kustomize" }}
+app.kubernetes.io/instance: {{ .Release.Name }}
+{{- end }}
+{{- end -}}
+
+{{/*
+Convert the `--extra-volume-tags` command line arg from a map.
+*/}}
+{{- define "aws-ebs-csi-driver.extra-volume-tags" -}}
+{{- $result := dict "pairs" (list) -}}
+{{- range $key, $value := .Values.extraVolumeTags -}}
+{{- $noop := printf "%s=%s" $key $value | append $result.pairs | set $result "pairs" -}}
+{{- end -}}
+{{- if gt (len $result.pairs) 0 -}}
+{{- printf "%s=%s" "- --extra-volume-tags" (join "," $result.pairs) -}}
+{{- end -}}
+{{- end -}}

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-attacher.yaml

@@ -0,0 +1,23 @@
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: ebs-external-attacher-role
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+rules:
+  - apiGroups: [""]
+    resources: ["persistentvolumes"]
+    verbs: ["get", "list", "watch", "update", "patch"]
+  - apiGroups: [""]
+    resources: ["nodes"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["csi.storage.k8s.io"]
+    resources: ["csinodeinfos"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["storage.k8s.io"]
+    resources: ["volumeattachments"]
+    verbs: ["get", "list", "watch", "update", "patch"]
+  - apiGroups: [ "storage.k8s.io" ]
+    resources: [ "volumeattachments/status" ]
+    verbs: [ "patch" ]

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-provisioner.yaml

@@ -0,0 +1,38 @@
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: ebs-external-provisioner-role
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+rules:
+  - apiGroups: [ "" ]
+    resources: [ "persistentvolumes" ]
+    verbs: [ "get", "list", "watch", "create", "delete" ]
+  - apiGroups: [ "" ]
+    resources: [ "persistentvolumeclaims" ]
+    verbs: [ "get", "list", "watch", "update" ]
+  - apiGroups: [ "storage.k8s.io" ]
+    resources: [ "storageclasses" ]
+    verbs: [ "get", "list", "watch" ]
+  - apiGroups: [ "" ]
+    resources: [ "events" ]
+    verbs: [ "list", "watch", "create", "update", "patch" ]
+  - apiGroups: [ "snapshot.storage.k8s.io" ]
+    resources: [ "volumesnapshots" ]
+    verbs: [ "get", "list" ]
+  - apiGroups: [ "snapshot.storage.k8s.io" ]
+    resources: [ "volumesnapshotcontents" ]
+    verbs: [ "get", "list" ]
+  - apiGroups: [ "storage.k8s.io" ]
+    resources: [ "csinodes" ]
+    verbs: [ "get", "list", "watch" ]
+  - apiGroups: [ "" ]
+    resources: [ "nodes" ]
+    verbs: [ "get", "list", "watch" ]
+  - apiGroups: [ "coordination.k8s.io" ]
+    resources: [ "leases" ]
+    verbs: [ "get", "watch", "list", "delete", "update", "create" ]
+  - apiGroups: [ "storage.k8s.io" ]
+    resources: [ "volumeattachments" ]
+    verbs: [ "get", "list", "watch" ]

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-resizer.yaml

@@ -0,0 +1,33 @@
+{{- if .Values.enableVolumeResizing }}
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: ebs-external-resizer-role
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+rules:
+  # The following rule should be uncommented for plugins that require secrets
+  # for provisioning.
+  # - apiGroups: [""]
+  #   resources: ["secrets"]
+  #   verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["persistentvolumes"]
+    verbs: ["get", "list", "watch", "update", "patch"]
+  - apiGroups: [""]
+    resources: ["persistentvolumeclaims"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["persistentvolumeclaims/status"]
+    verbs: ["update", "patch"]
+  - apiGroups: ["storage.k8s.io"]
+    resources: ["storageclasses"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["events"]
+    verbs: ["list", "watch", "create", "update", "patch"]
+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["get", "list", "watch"]
+{{- end}}

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-snapshot-controller.yaml

@@ -0,0 +1,35 @@
+{{- if .Values.enableVolumeSnapshot }}
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: ebs-snapshot-controller-role
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+rules:
+  - apiGroups: [""]
+    resources: ["persistentvolumes"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["persistentvolumeclaims"]
+    verbs: ["get", "list", "watch", "update"]
+  - apiGroups: ["storage.k8s.io"]
+    resources: ["storageclasses"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["events"]
+    verbs: ["list", "watch", "create", "update", "patch"]
+  - apiGroups: ["snapshot.storage.k8s.io"]
+    resources: ["volumesnapshotclasses"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["snapshot.storage.k8s.io"]
+    resources: ["volumesnapshotcontents"]
+    verbs: ["create", "get", "list", "watch", "update", "delete"]
+  - apiGroups: ["snapshot.storage.k8s.io"]
+    resources: ["volumesnapshots"]
+    verbs: ["get", "list", "watch", "update"]
+  - apiGroups: ["snapshot.storage.k8s.io"]
+    resources: ["volumesnapshots/status"]
+    verbs: ["update"]
+
+{{- end }}

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-snapshotter.yaml

@@ -0,0 +1,25 @@
+{{- if .Values.enableVolumeSnapshot }}
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: ebs-external-snapshotter-role
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+rules:
+  - apiGroups: [""]
+    resources: ["events"]
+    verbs: ["list", "watch", "create", "update", "patch"]
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get", "list"]
+  - apiGroups: ["snapshot.storage.k8s.io"]
+    resources: ["volumesnapshotclasses"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["snapshot.storage.k8s.io"]
+    resources: ["volumesnapshotcontents"]
+    verbs: ["create", "get", "list", "watch", "update", "delete"]
+  - apiGroups: ["snapshot.storage.k8s.io"]
+    resources: ["volumesnapshotcontents/status"]
+    verbs: ["update"]
+{{- end }}

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-attacher.yaml

@@ -0,0 +1,15 @@
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: ebs-csi-attacher-binding
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+subjects:
+  - kind: ServiceAccount
+    name: {{ .Values.serviceAccount.controller.name }}
+    namespace: {{ .Release.Namespace }}
+roleRef:
+  kind: ClusterRole
+  name: ebs-external-attacher-role
+  apiGroup: rbac.authorization.k8s.io

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-provisioner.yaml

@@ -0,0 +1,15 @@
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: ebs-csi-provisioner-binding
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+subjects:
+  - kind: ServiceAccount
+    name: {{ .Values.serviceAccount.controller.name }}
+    namespace: {{ .Release.Namespace }}
+roleRef:
+  kind: ClusterRole
+  name: ebs-external-provisioner-role
+  apiGroup: rbac.authorization.k8s.io

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-resizer.yaml

@@ -0,0 +1,18 @@
+{{- if .Values.enableVolumeResizing }}
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: ebs-csi-resizer-binding
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+subjects:
+  - kind: ServiceAccount
+    name: {{ .Values.serviceAccount.controller.name }}
+    namespace: {{ .Release.Namespace }}
+roleRef:
+  kind: ClusterRole
+  name: ebs-external-resizer-role
+  apiGroup: rbac.authorization.k8s.io
+
+{{- end}}

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-snapshot-controller.yaml

@@ -0,0 +1,18 @@
+{{- if .Values.enableVolumeSnapshot }}
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: ebs-csi-snapshot-controller-binding
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+subjects:
+  - kind: ServiceAccount
+    name: {{ .Values.serviceAccount.snapshot.name }}
+    namespace: {{ .Release.Namespace }}
+roleRef:
+  kind: ClusterRole
+  name: ebs-snapshot-controller-role
+  apiGroup: rbac.authorization.k8s.io
+
+{{- end }}

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-snapshotter.yaml

@@ -0,0 +1,18 @@
+{{- if .Values.enableVolumeSnapshot }}
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: ebs-csi-snapshotter-binding
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+subjects:
+  - kind: ServiceAccount
+    name: {{ .Values.serviceAccount.controller.name }}
+    namespace: {{ .Release.Namespace }}
+roleRef:
+  kind: ClusterRole
+  name: ebs-external-snapshotter-role
+  apiGroup: rbac.authorization.k8s.io
+
+{{- end }}

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/controller.yaml

@@ -0,0 +1,245 @@
+# Controller Service
+kind: Deployment
+apiVersion: apps/v1
+metadata:
+  name: ebs-csi-controller
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+spec:
+  replicas: {{ .Values.replicaCount }}
+  selector:
+    matchLabels:
+      app: ebs-csi-controller
+      {{- include "aws-ebs-csi-driver.selectorLabels" . | nindent 6 }}
+  template:
+    metadata:
+      labels:
+        app: ebs-csi-controller
+        {{- include "aws-ebs-csi-driver.labels" . | nindent 8 }}
+      {{- if .Values.podAnnotations }}
+      annotations: {{ toYaml .Values.podAnnotations | nindent 8 }}
+      {{- end }}
+    spec:
+      nodeSelector:
+        kubernetes.io/os: linux
+        {{- with .Values.nodeSelector }}
+{{ toYaml . | indent 8 }}
+        {{- end }}
+      serviceAccountName: {{ .Values.serviceAccount.controller.name }}
+      priorityClassName: {{ .Values.priorityClassName | default "system-cluster-critical" }}
+      {{- with .Values.affinity }}
+      affinity: {{ toYaml . | nindent 8 }}
+      {{- end }}
+      tolerations:
+        {{- if .Values.tolerateAllTaints }}
+        - operator: Exists
+        {{- else }}
+        - key: CriticalAddonsOnly
+          operator: Exists
+        - operator: Exists
+          effect: NoExecute
+          tolerationSeconds: 300
+        {{- end }}
+        {{- with .Values.tolerations }}
+{{ toYaml . | indent 8 }}
+        {{- end }}
+{{- if .Values.topologySpreadConstraints }}
+{{- $tscLabelSelector := dict "labelSelector" ( dict "matchLabels" ( dict "app" "ebs-csi-controller" ) ) }}
+      topologySpreadConstraints:
+        {{- range .Values.topologySpreadConstraints }}
+        - {{ mergeOverwrite . $tscLabelSelector | toJson }}
+        {{- end }}
+{{- end }}
+      containers:
+        - name: ebs-plugin
+          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
+          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          args:
+            {{- if ne .Release.Name "kustomize" }}
+            - controller
+            {{- else }}
+            # - {all,controller,node} # specify the driver mode
+            {{- end }}
+            - --endpoint=$(CSI_ENDPOINT)
+            {{- if .Values.extraVolumeTags }}
+              {{- include "aws-ebs-csi-driver.extra-volume-tags" . | nindent 12 }}
+            {{- end }}
+            {{- if .Values.k8sTagClusterId }}
+            - --k8s-tag-cluster-id={{ .Values.k8sTagClusterId }}
+            {{- end }}
+            - --logtostderr
+            - --v={{ .Values.logLevel }}
+          env:
+            - name: CSI_ENDPOINT
+              value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
+            - name: AWS_ACCESS_KEY_ID
+              valueFrom:
+                secretKeyRef:
+                  name: aws-secret
+                  key: key_id
+                  optional: true
+            - name: AWS_SECRET_ACCESS_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: aws-secret
+                  key: access_key
+                  optional: true
+            {{- if .Values.region }}
+            - name: AWS_REGION
+              value: {{ .Values.region }}
+            {{- end }}
+{{- if .Values.proxy.http_proxy }}
+            - name: HTTP_PROXY
+              value: {{ .Values.proxy.http_proxy | quote }}
+            - name: HTTPS_PROXY
+              value: {{ .Values.proxy.http_proxy | quote }}
+            - name: NO_PROXY
+              value: {{ .Values.proxy.no_proxy | quote }}
+{{- end }}
+          volumeMounts:
+            - name: socket-dir
+              mountPath: /var/lib/csi/sockets/pluginproxy/
+          ports:
+            - name: healthz
+              containerPort: 9808
+              protocol: TCP
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: healthz
+            initialDelaySeconds: 10
+            timeoutSeconds: 3
+            periodSeconds: 10
+            failureThreshold: 5
+          readinessProbe:
+            httpGet:
+              path: /healthz
+              port: healthz
+            initialDelaySeconds: 10
+            timeoutSeconds: 3
+            periodSeconds: 10
+            failureThreshold: 5
+          {{- with .Values.resources }}
+          resources: {{ toYaml . | nindent 12 }}
+          {{- end }}
+        - name: csi-provisioner
+          image: {{ printf "%s:%s" .Values.sidecars.provisionerImage.repository .Values.sidecars.provisionerImage.tag }}
+          args:
+            - --csi-address=$(ADDRESS)
+            - --v={{ .Values.logLevel }}
+            {{- if .Values.enableVolumeScheduling }}
+            - --feature-gates=Topology=true
+            {{- end}}
+            {{- if .Values.extraCreateMetadata }}
+            - --extra-create-metadata
+            {{- end}}
+            - --leader-election={{ ternary "true" "false" ( gt (.Values.replicaCount|int) 1 ) }}
+            - --default-fstype=ext4
+          env:
+            - name: ADDRESS
+              value: /var/lib/csi/sockets/pluginproxy/csi.sock
+{{- if .Values.proxy.http_proxy }}
+            - name: HTTP_PROXY
+              value: {{ .Values.proxy.http_proxy | quote }}
+            - name: HTTPS_PROXY
+              value: {{ .Values.proxy.http_proxy | quote }}
+            - name: NO_PROXY
+              value: {{ .Values.proxy.no_proxy | quote }}
+{{- end }}
+          volumeMounts:
+            - name: socket-dir
+              mountPath: /var/lib/csi/sockets/pluginproxy/
+          {{- with .Values.resources }}
+          resources: {{ toYaml . | nindent 12 }}
+          {{- end }}
+        - name: csi-attacher
+          image: {{ printf "%s:%s" .Values.sidecars.attacherImage.repository .Values.sidecars.attacherImage.tag }}
+          args:
+            - --csi-address=$(ADDRESS)
+            - --v={{ .Values.logLevel }}
+            - --leader-election={{ ternary "true" "false" ( gt (.Values.replicaCount|int) 1 ) }}
+          env:
+            - name: ADDRESS
+              value: /var/lib/csi/sockets/pluginproxy/csi.sock
+{{- if .Values.proxy.http_proxy }}
+            - name: HTTP_PROXY
+              value: {{ .Values.proxy.http_proxy | quote }}
+            - name: HTTPS_PROXY
+              value: {{ .Values.proxy.http_proxy | quote }}
+            - name: NO_PROXY
+              value: {{ .Values.proxy.no_proxy | quote }}
+{{- end }}
+          volumeMounts:
+            - name: socket-dir
+              mountPath: /var/lib/csi/sockets/pluginproxy/
+          {{- with .Values.resources }}
+          resources: {{ toYaml . | nindent 12 }}
+          {{- end }}
+        {{- if .Values.enableVolumeSnapshot }}
+        - name: csi-snapshotter
+          image: {{ printf "%s:%s" .Values.sidecars.snapshotterImage.repository .Values.sidecars.snapshotterImage.tag }}
+          args:
+            - --csi-address=$(ADDRESS)
+            - --leader-election={{ ternary "true" "false" ( gt (.Values.replicaCount|int) 1 ) }}
+          env:
+            - name: ADDRESS
+              value: /var/lib/csi/sockets/pluginproxy/csi.sock
+{{- if .Values.proxy.http_proxy }}
+            - name: HTTP_PROXY
+              value: {{ .Values.proxy.http_proxy | quote }}
+            - name: HTTPS_PROXY
+              value: {{ .Values.proxy.http_proxy | quote }}
+            - name: NO_PROXY
+              value: {{ .Values.proxy.no_proxy | quote }}
+{{- end }}
+          volumeMounts:
+            - name: socket-dir
+              mountPath: /var/lib/csi/sockets/pluginproxy/
+          {{- with .Values.resources }}
+          resources: {{ toYaml . | nindent 12 }}
+          {{- end }}
+        {{- end }}
+        {{- if .Values.enableVolumeResizing }}
+        - name: csi-resizer
+          image: {{ printf "%s:%s" .Values.sidecars.resizerImage.repository .Values.sidecars.resizerImage.tag }}
+          imagePullPolicy: Always
+          args:
+            - --csi-address=$(ADDRESS)
+            - --v={{ .Values.logLevel }}
+          env:
+            - name: ADDRESS
+              value: /var/lib/csi/sockets/pluginproxy/csi.sock
+{{- if .Values.proxy.http_proxy }}
+            - name: HTTP_PROXY
+              value: {{ .Values.proxy.http_proxy | quote }}
+            - name: HTTPS_PROXY
+              value: {{ .Values.proxy.http_proxy | quote }}
+            - name: NO_PROXY
+              value: {{ .Values.proxy.no_proxy | quote }}
+{{- end }}
+          volumeMounts:
+            - name: socket-dir
+              mountPath: /var/lib/csi/sockets/pluginproxy/
+          {{- with .Values.resources }}
+          resources: {{ toYaml . | nindent 12 }}
+          {{- end }}
+        {{- end }}
+        - name: liveness-probe
+          image: {{ printf "%s:%s" .Values.sidecars.livenessProbeImage.repository .Values.sidecars.livenessProbeImage.tag }}
+          args:
+            - --csi-address=/csi/csi.sock
+          volumeMounts:
+            - name: socket-dir
+              mountPath: /csi
+          {{- with .Values.resources }}
+          resources: {{ toYaml . | nindent 12 }}
+          {{- end }}
+      {{- if .Values.imagePullSecrets }}
+      imagePullSecrets:
+      {{- range .Values.imagePullSecrets }}
+        - name: {{ . }}
+      {{- end }}
+      {{- end }}
+      volumes:
+        - name: socket-dir
+          emptyDir: {}

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/csidriver.yaml

@@ -0,0 +1,9 @@
+apiVersion: storage.k8s.io/v1beta1
+kind: CSIDriver
+metadata:
+  name: ebs.csi.aws.com
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+spec:
+  attachRequired: true
+  podInfoOnMount: false

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/node.yaml

@@ -0,0 +1,176 @@
+# Node Service
+kind: DaemonSet
+apiVersion: apps/v1
+metadata:
+  name: ebs-csi-node
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+spec:
+  selector:
+    matchLabels:
+      app: ebs-csi-node
+      {{- include "aws-ebs-csi-driver.selectorLabels" . | nindent 6 }}
+  template:
+    metadata:
+      labels:
+        app: ebs-csi-node
+        {{- include "aws-ebs-csi-driver.labels" . | nindent 8 }}
+      {{- if .Values.node.podAnnotations }}
+      annotations: {{ toYaml .Values.node.podAnnotations | nindent 8 }}
+      {{- end }}
+    spec:
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+            - matchExpressions:
+              - key: eks.amazonaws.com/compute-type
+                operator: NotIn
+                values:
+                - fargate
+      nodeSelector:
+        kubernetes.io/os: linux
+        {{- with .Values.node.nodeSelector }}
+{{ toYaml . | indent 8 }}
+        {{- end }}
+      hostNetwork: true
+      serviceAccountName: {{ .Values.serviceAccount.node.name }}
+      priorityClassName: {{ .Values.node.priorityClassName | default "system-cluster-critical" }}
+      tolerations:
+        {{- if .Values.node.tolerateAllTaints }}
+        - operator: Exists
+        {{- else }}
+        - key: CriticalAddonsOnly
+          operator: Exists
+        - operator: Exists
+          effect: NoExecute
+          tolerationSeconds: 300
+        {{- end }}
+        {{- with .Values.node.tolerations }}
+{{ toYaml . | indent 8 }}
+        {{- end }}
+      containers:
+        - name: ebs-plugin
+          securityContext:
+            privileged: true
+          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
+          args:
+            - node
+            - --endpoint=$(CSI_ENDPOINT)
+            {{- if .Values.volumeAttachLimit }}
+            - --volume-attach-limit={{ .Values.volumeAttachLimit }}
+            {{- end }}
+            - --logtostderr
+            - --v={{ .Values.logLevel }}
+          env:
+            - name: CSI_ENDPOINT
+              value: unix:/csi/csi.sock
+{{- if .Values.proxy.http_proxy }}
+            - name: HTTP_PROXY
+              value: {{ .Values.proxy.http_proxy | quote }}
+            - name: HTTPS_PROXY
+              value: {{ .Values.proxy.http_proxy | quote }}
+            - name: NO_PROXY
+              value: {{ .Values.proxy.no_proxy | quote }}
+{{- end }}
+          volumeMounts:
+            - name: kubelet-dir
+              mountPath: /var/lib/kubelet
+              mountPropagation: "Bidirectional"
+            - name: plugin-dir
+              mountPath: /csi
+            - name: device-dir
+              mountPath: /dev
+          ports:
+            - name: healthz
+              containerPort: 9808
+              protocol: TCP
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: healthz
+            initialDelaySeconds: 10
+            timeoutSeconds: 3
+            periodSeconds: 10
+            failureThreshold: 5
+          {{- if .Values.node.resources }}
+          {{- with .Values.node.resources }}
+          resources: {{ toYaml . | nindent 12 }}
+          {{- end }}
+          {{- else }}
+          {{- with .Values.resources }}
+          resources: {{ toYaml . | nindent 12 }}
+          {{- end }}
+          {{- end }}
+        - name: node-driver-registrar
+          image: {{ printf "%s:%s" .Values.sidecars.nodeDriverRegistrarImage.repository .Values.sidecars.nodeDriverRegistrarImage.tag }}
+          args:
+            - --csi-address=$(ADDRESS)
+            - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
+            - --v={{ .Values.logLevel }}
+          env:
+            - name: ADDRESS
+              value: /csi/csi.sock
+            - name: DRIVER_REG_SOCK_PATH
+              value: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
+{{- if .Values.proxy.http_proxy }}
+            - name: HTTP_PROXY
+              value: {{ .Values.proxy.http_proxy | quote }}
+            - name: HTTPS_PROXY
+              value: {{ .Values.proxy.http_proxy | quote }}
+            - name: NO_PROXY
+              value: {{ .Values.proxy.no_proxy | quote }}
+{{- end }}
+          volumeMounts:
+            - name: plugin-dir
+              mountPath: /csi
+            - name: registration-dir
+              mountPath: /registration
+          {{- if .Values.node.resources }}
+          {{- with .Values.node.resources }}
+          resources: {{ toYaml . | nindent 12 }}
+          {{- end }}
+          {{- else }}
+          {{- with .Values.resources }}
+          resources: {{ toYaml . | nindent 12 }}
+          {{- end }}
+          {{- end }}
+        - name: liveness-probe
+          image: {{ printf "%s:%s" .Values.sidecars.livenessProbeImage.repository .Values.sidecars.livenessProbeImage.tag }}
+          args:
+            - --csi-address=/csi/csi.sock
+          volumeMounts:
+            - name: plugin-dir
+              mountPath: /csi
+          {{- if .Values.node.resources }}
+          {{- with .Values.node.resources }}
+          resources: {{ toYaml . | nindent 12 }}
+          {{- end }}
+          {{- else }}
+          {{- with .Values.resources }}
+          resources: {{ toYaml . | nindent 12 }}
+          {{- end }}
+          {{- end }}
+      {{- if .Values.imagePullSecrets }}
+      imagePullSecrets:
+      {{- range .Values.imagePullSecrets }}
+        - name: {{ . }}
+      {{- end }}
+      {{- end }}
+      volumes:
+        - name: kubelet-dir
+          hostPath:
+            path: /var/lib/kubelet
+            type: Directory
+        - name: plugin-dir
+          hostPath:
+            path: /var/lib/kubelet/plugins/ebs.csi.aws.com/
+            type: DirectoryOrCreate
+        - name: registration-dir
+          hostPath:
+            path: /var/lib/kubelet/plugins_registry/
+            type: Directory
+        - name: device-dir
+          hostPath:
+            path: /dev
+            type: Directory

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/role-snapshot-controller-leaderelection.yaml

@@ -0,0 +1,14 @@
+{{- if .Values.enableVolumeSnapshot }}
+---
+kind: Role
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: ebs-snapshot-controller-leaderelection
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+rules:
+  - apiGroups: ["coordination.k8s.io"]
+    resources: ["leases"]
+    verbs: ["get", "watch", "list", "delete", "update", "create"]
+
+{{- end }}

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/rolebinding-snapshot-controller-leaderelection.yaml

@@ -0,0 +1,18 @@
+{{- if .Values.enableVolumeSnapshot }}
+---
+kind: RoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: ebs-snapshot-controller-leaderelection
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+subjects:
+  - kind: ServiceAccount
+    name: {{ .Values.serviceAccount.snapshot.name }}
+    namespace: {{ .Release.Namespace }}
+roleRef:
+  kind: Role
+  name: ebs-snapshot-controller-leaderelection
+  apiGroup: rbac.authorization.k8s.io
+
+{{- end }}

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/serviceaccount-csi-controller.yaml

@@ -0,0 +1,16 @@
+{{- if .Values.serviceAccount.controller.create -}}
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: {{ .Values.serviceAccount.controller.name }}
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+  {{- with .Values.serviceAccount.controller.annotations }}
+  annotations: {{ toYaml . | nindent 4 }}
+  {{- end }}
+  {{- if eq .Release.Name "kustomize" }}
+  #Enable if EKS IAM for SA is used
+  #annotations:
+  #  eks.amazonaws.com/role-arn: arn:aws:iam::586565787010:role/ebs-csi-role
+  {{- end }}
+{{- end -}}

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/serviceaccount-csi-node.yaml

@@ -0,0 +1,11 @@
+{{- if .Values.serviceAccount.node.create -}}
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: {{ .Values.serviceAccount.node.name }}
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+  {{- with .Values.serviceAccount.node.annotations }}
+  annotations: {{ toYaml . | nindent 4 }}
+  {{- end }}
+{{- end -}}

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/serviceaccount-snapshot-controller.yaml

@@ -0,0 +1,14 @@
+{{- if .Values.enableVolumeSnapshot }}
+{{- if .Values.serviceAccount.snapshot.create }}
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: {{ .Values.serviceAccount.snapshot.name }}
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+  {{- with .Values.serviceAccount.snapshot.annotations }}
+  annotations: {{ toYaml . | nindent 4 }}
+  {{- end }}
+{{- end }}
+{{- end }}

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/statefulset.yaml

@@ -0,0 +1,60 @@
+{{- if .Values.enableVolumeSnapshot }}
+#Snapshot controller
+kind: StatefulSet
+apiVersion: apps/v1
+metadata:
+  name: ebs-snapshot-controller
+  labels:
+    {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }}
+spec:
+  serviceName: ebs-snapshot-controller
+  replicas: 1
+  selector:
+    matchLabels:
+      app: ebs-snapshot-controller
+      {{- include "aws-ebs-csi-driver.selectorLabels" . | nindent 6 }}
+  template:
+    metadata:
+      labels:
+        app: ebs-snapshot-controller
+        {{- include "aws-ebs-csi-driver.labels" . | nindent 8 }}
+    spec:
+      serviceAccountName: {{ .Values.serviceAccount.snapshot.name }}
+      nodeSelector:
+        kubernetes.io/os: linux
+        {{- with .Values.nodeSelector }}
+{{ toYaml . | indent 8 }}
+        {{- end }}
+      priorityClassName: {{ .Values.priorityClassName | default "system-cluster-critical" }}
+      {{- with .Values.affinity }}
+      affinity: {{ toYaml . | nindent 8 }}
+      {{- end }}
+      tolerations:
+        {{- if .Values.tolerateAllTaints }}
+        - operator: Exists
+        {{- end }}
+        {{- with .Values.tolerations }}
+{{ toYaml . | indent 8 }}
+        {{- end }}
+      containers:
+        - name: snapshot-controller
+          image: {{ printf "%s:%s" .Values.snapshotController.repository .Values.snapshotController.tag }}
+          env:
+{{- if .Values.proxy.http_proxy }}
+          - name: HTTP_PROXY
+            value: {{ .Values.proxy.http_proxy | quote }}
+          - name: HTTPS_PROXY
+            value: {{ .Values.proxy.http_proxy | quote }}
+          - name: NO_PROXY
+            value: {{ .Values.proxy.no_proxy | quote }}
+{{- end }}
+          args:
+            - --v={{ .Values.logLevel }}
+            - --leader-election=false
+      {{- if .Values.imagePullSecrets }}
+      imagePullSecrets:
+      {{- range .Values.imagePullSecrets }}
+        - name: {{ . }}
+      {{- end }}
+      {{- end }}
+{{- end }}

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/storageclass.yaml

@@ -0,0 +1,15 @@
+{{- range .Values.storageClasses }}
+---
+kind: StorageClass
+apiVersion: storage.k8s.io/v1
+metadata:
+  name: {{ .name }}
+  {{- if .annotations }}
+  annotations: {{- .annotations | toYaml | trim | nindent 4 }}
+  {{- end }}
+  {{- if .labels }}
+  labels: {{- .labels | toYaml | trim | nindent 4 }}
+  {{- end }}
+provisioner: ebs.csi.aws.com
+{{ omit (dict "volumeBindingMode" "WaitForFirstConsumer" | merge .) "name" "annotations" "labels" | toYaml }}
+{{- end }}

charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/values.yaml

@@ -0,0 +1,148 @@
+# Default values for aws-ebs-csi-driver.
+# This is a YAML-formatted file.
+# Declare variables to be passed into your templates.
+
+replicaCount: 2
+
+image:
+  repository: k8s.gcr.io/provider-aws/aws-ebs-csi-driver
+  tag: "v0.9.0"
+  pullPolicy: IfNotPresent
+
+logLevel: 5
+
+sidecars:
+  provisionerImage:
+    repository: k8s.gcr.io/sig-storage/csi-provisioner
+    tag: "v2.0.2"
+  attacherImage:
+    repository: k8s.gcr.io/sig-storage/csi-attacher
+    tag: "v3.0.0"
+  snapshotterImage:
+    repository: k8s.gcr.io/sig-storage/csi-snapshotter
+    tag: "v3.0.3"
+  livenessProbeImage:
+    repository: k8s.gcr.io/sig-storage/livenessprobe
+    tag: "v2.2.0"
+  resizerImage:
+    repository: k8s.gcr.io/sig-storage/csi-resizer
+    tag: "v1.0.0"
+  nodeDriverRegistrarImage:
+    repository: k8s.gcr.io/sig-storage/csi-node-driver-registrar
+    tag: "v2.0.1"
+
+snapshotController:
+  repository: k8s.gcr.io/sig-storage/snapshot-controller
+  tag: "v3.0.3"
+
+proxy: {}
+#  http_proxy:
+#  no_proxy:
+
+imagePullSecrets: []
+nameOverride: ""
+fullnameOverride: ""
+
+podAnnotations: {}
+
+# True if enable volume scheduling for dynamic volume provisioning
+enableVolumeScheduling: true
+
+# True if enable volume resizing
+enableVolumeResizing: false
+
+# True if enable volume snapshot
+enableVolumeSnapshot: false
+
+# The "maximum number of attachable volumes" per node
+volumeAttachLimit: ""
+
+resources:
+  {}
+  # We usually recommend not to specify default resources and to leave this as a conscious
+  # choice for the user. This also increases chances charts run on environments with little
+  # resources, such as Minikube. If you do want to specify resources, uncomment the following
+  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
+  # limits:
+  #   cpu: 100m
+  #   memory: 128Mi
+  # requests:
+  #   cpu: 100m
+  #   memory: 128Mi
+
+priorityClassName: ""
+nodeSelector: {}
+tolerateAllTaints: true
+tolerations: []
+affinity: {}
+
+# TSCs without the label selector stanza
+#
+# Example:
+#
+# topologySpreadConstraints:
+#  - maxSkew: 1
+#    topologyKey: topology.kubernetes.io/zone
+#    whenUnsatisfiable: ScheduleAnyway
+#  - maxSkew: 1
+#    topologyKey: kubernetes.io/hostname
+#    whenUnsatisfiable: ScheduleAnyway
+
+topologySpreadConstraints: []
+
+# Extra volume tags to attach to each dynamically provisioned volume.
+# ---
+# extraVolumeTags:
+#   key1: value1
+#   key2: value2
+extraVolumeTags: {}
+
+# If set, add pv/pvc metadata to plugin create requests as parameters.
+extraCreateMetadata: false
+
+# ID of the Kubernetes cluster used for tagging provisioned EBS volumes (optional).
+k8sTagClusterId: ""
+
+# AWS region to use. If not specified then the region will be looked up via the AWS EC2 metadata
+# service.
+# ---
+# region: us-east-1
+region: ""
+
+node:
+  priorityClassName: ""
+  nodeSelector: {}
+  podAnnotations: {}
+  tolerateAllTaints: true
+  tolerations: []
+  resources: {}
+
+serviceAccount:
+  controller:
+    create: true # A service account will be created for you if set to true. Set to false if you want to use your own.
+    name: ebs-csi-controller-sa # Name of the service-account to be used/created.
+    annotations: {}
+  snapshot:
+    create: true
+    name: ebs-snapshot-controller
+    annotations: {}
+  node:
+    create: true
+    name: ebs-csi-node-sa
+    annotations: {}
+
+storageClasses: []
+# Add StorageClass resources like:
+# - name: ebs-sc
+#   # annotation metadata
+#   annotations:
+#     storageclass.kubernetes.io/is-default-class: "true"
+#   # label metadata
+#   labels:
+#     my-label-is: supercool
+#   # defaults to WaitForFirstConsumer
+#   volumeBindingMode: WaitForFirstConsumer
+#   # defaults to Delete
+#   reclaimPolicy: Retain
+#   parameters:
+#     encrypted: "true"

charts/kubezero-aws-ebs-csi-driver/crds/crd_snapshotter.yaml

@@ -0,0 +1,420 @@
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+    api-approved.kubernetes.io: "https://github.com/kubernetes-csi/external-snapshotter/pull/139"
+  creationTimestamp: null
+  name: volumesnapshotclasses.snapshot.storage.k8s.io
+spec:
+  group: snapshot.storage.k8s.io
+  names:
+    kind: VolumeSnapshotClass
+    listKind: VolumeSnapshotClassList
+    plural: volumesnapshotclasses
+    singular: volumesnapshotclass
+  scope: Cluster
+  preserveUnknownFields: false
+  validation:
+    openAPIV3Schema:
+      description: VolumeSnapshotClass specifies parameters that a underlying storage
+        system uses when creating a volume snapshot. A specific VolumeSnapshotClass
+        is used by specifying its name in a VolumeSnapshot object. VolumeSnapshotClasses
+        are non-namespaced
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
+          type: string
+        deletionPolicy:
+          description: deletionPolicy determines whether a VolumeSnapshotContent created
+            through the VolumeSnapshotClass should be deleted when its bound VolumeSnapshot
+            is deleted. Supported values are "Retain" and "Delete". "Retain" means
+            that the VolumeSnapshotContent and its physical snapshot on underlying
+            storage system are kept. "Delete" means that the VolumeSnapshotContent
+            and its physical snapshot on underlying storage system are deleted. Required.
+          enum:
+          - Delete
+          - Retain
+          type: string
+        driver:
+          description: driver is the name of the storage driver that handles this
+            VolumeSnapshotClass. Required.
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
+          type: string
+        parameters:
+          additionalProperties:
+            type: string
+          description: parameters is a key-value map with storage driver specific
+            parameters for creating snapshots. These values are opaque to Kubernetes.
+          type: object
+      required:
+      - deletionPolicy
+      - driver
+      type: object
+  version: v1beta1
+  versions:
+  - name: v1beta1
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []
+
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+    api-approved.kubernetes.io: "https://github.com/kubernetes-csi/external-snapshotter/pull/139"
+  creationTimestamp: null
+  name: volumesnapshotcontents.snapshot.storage.k8s.io
+spec:
+  group: snapshot.storage.k8s.io
+  names:
+    kind: VolumeSnapshotContent
+    listKind: VolumeSnapshotContentList
+    plural: volumesnapshotcontents
+    singular: volumesnapshotcontent
+  scope: Cluster
+  subresources:
+    status: {}
+  preserveUnknownFields: false
+  validation:
+    openAPIV3Schema:
+      description: VolumeSnapshotContent represents the actual "on-disk" snapshot
+        object in the underlying storage system
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
+          type: string
+        spec:
+          description: spec defines properties of a VolumeSnapshotContent created
+            by the underlying storage system. Required.
+          properties:
+            deletionPolicy:
+              description: deletionPolicy determines whether this VolumeSnapshotContent
+                and its physical snapshot on the underlying storage system should
+                be deleted when its bound VolumeSnapshot is deleted. Supported values
+                are "Retain" and "Delete". "Retain" means that the VolumeSnapshotContent
+                and its physical snapshot on underlying storage system are kept. "Delete"
+                means that the VolumeSnapshotContent and its physical snapshot on
+                underlying storage system are deleted. In dynamic snapshot creation
+                case, this field will be filled in with the "DeletionPolicy" field
+                defined in the VolumeSnapshotClass the VolumeSnapshot refers to. For
+                pre-existing snapshots, users MUST specify this field when creating
+                the VolumeSnapshotContent object. Required.
+              enum:
+              - Delete
+              - Retain
+              type: string
+            driver:
+              description: driver is the name of the CSI driver used to create the
+                physical snapshot on the underlying storage system. This MUST be the
+                same as the name returned by the CSI GetPluginName() call for that
+                driver. Required.
+              type: string
+            source:
+              description: source specifies from where a snapshot will be created.
+                This field is immutable after creation. Required.
+              properties:
+                snapshotHandle:
+                  description: snapshotHandle specifies the CSI "snapshot_id" of a
+                    pre-existing snapshot on the underlying storage system. This field
+                    is immutable.
+                  type: string
+                volumeHandle:
+                  description: volumeHandle specifies the CSI "volume_id" of the volume
+                    from which a snapshot should be dynamically taken from. This field
+                    is immutable.
+                  type: string
+              type: object
+            volumeSnapshotClassName:
+              description: name of the VolumeSnapshotClass to which this snapshot
+                belongs.
+              type: string
+            volumeSnapshotRef:
+              description: volumeSnapshotRef specifies the VolumeSnapshot object to
+                which this VolumeSnapshotContent object is bound. VolumeSnapshot.Spec.VolumeSnapshotContentName
+                field must reference to this VolumeSnapshotContent's name for the
+                bidirectional binding to be valid. For a pre-existing VolumeSnapshotContent
+                object, name and namespace of the VolumeSnapshot object MUST be provided
+                for binding to happen. This field is immutable after creation. Required.
+              properties:
+                apiVersion:
+                  description: API version of the referent.
+                  type: string
+                fieldPath:
+                  description: 'If referring to a piece of an object instead of an
+                    entire object, this string should contain a valid JSON/Go field
+                    access statement, such as desiredState.manifest.containers[2].
+                    For example, if the object reference is to a container within
+                    a pod, this would take on a value like: "spec.containers{name}"
+                    (where "name" refers to the name of the container that triggered
+                    the event) or if no container name is specified "spec.containers[2]"
+                    (container with index 2 in this pod). This syntax is chosen only
+                    to have some well-defined way of referencing a part of an object.
+                    TODO: this design is not final and this field is subject to change
+                    in the future.'
+                  type: string
+                kind:
+                  description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
+                  type: string
+                name:
+                  description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
+                  type: string
+                namespace:
+                  description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
+                  type: string
+                resourceVersion:
+                  description: 'Specific resourceVersion to which this reference is
+                    made, if any. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#concurrency-control-and-consistency'
+                  type: string
+                uid:
+                  description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
+                  type: string
+              type: object
+          required:
+          - deletionPolicy
+          - driver
+          - source
+          - volumeSnapshotRef
+          type: object
+        status:
+          description: status represents the current information of a snapshot.
+          properties:
+            creationTime:
+              description: creationTime is the timestamp when the point-in-time snapshot
+                is taken by the underlying storage system. In dynamic snapshot creation
+                case, this field will be filled in with the "creation_time" value
+                returned from CSI "CreateSnapshotRequest" gRPC call. For a pre-existing
+                snapshot, this field will be filled with the "creation_time" value
+                returned from the CSI "ListSnapshots" gRPC call if the driver supports
+                it. If not specified, it indicates the creation time is unknown. The
+                format of this field is a Unix nanoseconds time encoded as an int64.
+                On Unix, the command `date +%s%N` returns the current time in nanoseconds
+                since 1970-01-01 00:00:00 UTC.
+              format: int64
+              type: integer
+            error:
+              description: error is the latest observed error during snapshot creation,
+                if any.
+              properties:
+                message:
+                  description: 'message is a string detailing the encountered error
+                    during snapshot creation if specified. NOTE: message may be logged,
+                    and it should not contain sensitive information.'
+                  type: string
+                time:
+                  description: time is the timestamp when the error was encountered.
+                  format: date-time
+                  type: string
+              type: object
+            readyToUse:
+              description: readyToUse indicates if a snapshot is ready to be used
+                to restore a volume. In dynamic snapshot creation case, this field
+                will be filled in with the "ready_to_use" value returned from CSI
+                "CreateSnapshotRequest" gRPC call. For a pre-existing snapshot, this
+                field will be filled with the "ready_to_use" value returned from the
+                CSI "ListSnapshots" gRPC call if the driver supports it, otherwise,
+                this field will be set to "True". If not specified, it means the readiness
+                of a snapshot is unknown.
+              type: boolean
+            restoreSize:
+              description: restoreSize represents the complete size of the snapshot
+                in bytes. In dynamic snapshot creation case, this field will be filled
+                in with the "size_bytes" value returned from CSI "CreateSnapshotRequest"
+                gRPC call. For a pre-existing snapshot, this field will be filled
+                with the "size_bytes" value returned from the CSI "ListSnapshots"
+                gRPC call if the driver supports it. When restoring a volume from
+                this snapshot, the size of the volume MUST NOT be smaller than the
+                restoreSize if it is specified, otherwise the restoration will fail.
+                If not specified, it indicates that the size is unknown.
+              format: int64
+              minimum: 0
+              type: integer
+            snapshotHandle:
+              description: snapshotHandle is the CSI "snapshot_id" of a snapshot on
+                the underlying storage system. If not specified, it indicates that
+                dynamic snapshot creation has either failed or it is still in progress.
+              type: string
+          type: object
+      required:
+      - spec
+      type: object
+  version: v1beta1
+  versions:
+  - name: v1beta1
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []
+
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: (devel)
+    api-approved.kubernetes.io: "https://github.com/kubernetes-csi/external-snapshotter/pull/139"
+  creationTimestamp: null
+  name: volumesnapshots.snapshot.storage.k8s.io
+spec:
+  group: snapshot.storage.k8s.io
+  names:
+    kind: VolumeSnapshot
+    listKind: VolumeSnapshotList
+    plural: volumesnapshots
+    singular: volumesnapshot
+  scope: Namespaced
+  subresources:
+    status: {}
+  preserveUnknownFields: false
+  validation:
+    openAPIV3Schema:
+      description: VolumeSnapshot is a user's request for either creating a point-in-time
+        snapshot of a persistent volume, or binding to a pre-existing snapshot.
+      properties:
+        apiVersion:
+          description: 'APIVersion defines the versioned schema of this representation
+            of an object. Servers should convert recognized schemas to the latest
+            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
+          type: string
+        kind:
+          description: 'Kind is a string value representing the REST resource this
+            object represents. Servers may infer this from the endpoint the client
+            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
+          type: string
+        spec:
+          description: 'spec defines the desired characteristics of a snapshot requested
+            by a user. More info: https://kubernetes.io/docs/concepts/storage/volume-snapshots#volumesnapshots
+            Required.'
+          properties:
+            source:
+              description: source specifies where a snapshot will be created from.
+                This field is immutable after creation. Required.
+              properties:
+                persistentVolumeClaimName:
+                  description: persistentVolumeClaimName specifies the name of the
+                    PersistentVolumeClaim object in the same namespace as the VolumeSnapshot
+                    object where the snapshot should be dynamically taken from. This
+                    field is immutable.
+                  type: string
+                volumeSnapshotContentName:
+                  description: volumeSnapshotContentName specifies the name of a pre-existing
+                    VolumeSnapshotContent object. This field is immutable.
+                  type: string
+              type: object
+            volumeSnapshotClassName:
+              description: 'volumeSnapshotClassName is the name of the VolumeSnapshotClass
+                requested by the VolumeSnapshot. If not specified, the default snapshot
+                class will be used if one exists. If not specified, and there is no
+                default snapshot class, dynamic snapshot creation will fail. Empty
+                string is not allowed for this field. TODO(xiangqian): a webhook validation
+                on empty string. More info: https://kubernetes.io/docs/concepts/storage/volume-snapshot-classes'
+              type: string
+          required:
+          - source
+          type: object
+        status:
+          description: 'status represents the current information of a snapshot. NOTE:
+            status can be modified by sources other than system controllers, and must
+            not be depended upon for accuracy. Controllers should only use information
+            from the VolumeSnapshotContent object after verifying that the binding
+            is accurate and complete.'
+          properties:
+            boundVolumeSnapshotContentName:
+              description: 'boundVolumeSnapshotContentName represents the name of
+                the VolumeSnapshotContent object to which the VolumeSnapshot object
+                is bound. If not specified, it indicates that the VolumeSnapshot object
+                has not been successfully bound to a VolumeSnapshotContent object
+                yet. NOTE: Specified boundVolumeSnapshotContentName alone does not
+                mean binding       is valid. Controllers MUST always verify bidirectional
+                binding between       VolumeSnapshot and VolumeSnapshotContent to
+                avoid possible security issues.'
+              type: string
+            creationTime:
+              description: creationTime is the timestamp when the point-in-time snapshot
+                is taken by the underlying storage system. In dynamic snapshot creation
+                case, this field will be filled in with the "creation_time" value
+                returned from CSI "CreateSnapshotRequest" gRPC call. For a pre-existing
+                snapshot, this field will be filled with the "creation_time" value
+                returned from the CSI "ListSnapshots" gRPC call if the driver supports
+                it. If not specified, it indicates that the creation time of the snapshot
+                is unknown.
+              format: date-time
+              type: string
+            error:
+              description: error is the last observed error during snapshot creation,
+                if any. This field could be helpful to upper level controllers(i.e.,
+                application controller) to decide whether they should continue on
+                waiting for the snapshot to be created based on the type of error
+                reported.
+              properties:
+                message:
+                  description: 'message is a string detailing the encountered error
+                    during snapshot creation if specified. NOTE: message may be logged,
+                    and it should not contain sensitive information.'
+                  type: string
+                time:
+                  description: time is the timestamp when the error was encountered.
+                  format: date-time
+                  type: string
+              type: object
+            readyToUse:
+              description: readyToUse indicates if a snapshot is ready to be used
+                to restore a volume. In dynamic snapshot creation case, this field
+                will be filled in with the "ready_to_use" value returned from CSI
+                "CreateSnapshotRequest" gRPC call. For a pre-existing snapshot, this
+                field will be filled with the "ready_to_use" value returned from the
+                CSI "ListSnapshots" gRPC call if the driver supports it, otherwise,
+                this field will be set to "True". If not specified, it means the readiness
+                of a snapshot is unknown.
+              type: boolean
+            restoreSize:
+              description: restoreSize represents the complete size of the snapshot
+                in bytes. In dynamic snapshot creation case, this field will be filled
+                in with the "size_bytes" value returned from CSI "CreateSnapshotRequest"
+                gRPC call. For a pre-existing snapshot, this field will be filled
+                with the "size_bytes" value returned from the CSI "ListSnapshots"
+                gRPC call if the driver supports it. When restoring a volume from
+                this snapshot, the size of the volume MUST NOT be smaller than the
+                restoreSize if it is specified, otherwise the restoration will fail.
+                If not specified, it indicates that the size is unknown.
+              type: string
+          type: object
+      required:
+      - spec
+      type: object
+  version: v1beta1
+  versions:
+  - name: v1beta1
+    served: true
+    storage: true
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []

charts/kubezero-aws-ebs-csi-driver/loglevel_leader.patch

@@ -0,0 +1,103 @@
+diff -rtubN aws-ebs-csi-driver.orig/templates/controller.yaml aws-ebs-csi-driver/templates/controller.yaml
+--- aws-ebs-csi-driver.orig/templates/controller.yaml	2021-03-05 03:10:41.000000000 +0100
++++ aws-ebs-csi-driver/templates/controller.yaml	2021-03-05 10:29:31.878615411 +0100
+@@ -68,7 +68,7 @@
+             - --k8s-tag-cluster-id={{ .Values.k8sTagClusterId }}
+             {{- end }}
+             - --logtostderr
+-            - --v=5
++            - --v={{ .Values.logLevel }}
+           env:
+             - name: CSI_ENDPOINT
+               value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
+@@ -126,14 +126,14 @@
+           image: {{ printf "%s:%s" .Values.sidecars.provisionerImage.repository .Values.sidecars.provisionerImage.tag }}
+           args:
+             - --csi-address=$(ADDRESS)
+-            - --v=5
++            - --v={{ .Values.logLevel }}
+             {{- if .Values.enableVolumeScheduling }}
+             - --feature-gates=Topology=true
+             {{- end}}
+             {{- if .Values.extraCreateMetadata }}
+             - --extra-create-metadata
+             {{- end}}
+-            - --leader-election=true
++            - --leader-election={{ ternary "true" "false" ( gt (.Values.replicaCount|int) 1 ) }}
+             - --default-fstype=ext4
+           env:
+             - name: ADDRESS
+@@ -156,8 +156,8 @@
+           image: {{ printf "%s:%s" .Values.sidecars.attacherImage.repository .Values.sidecars.attacherImage.tag }}
+           args:
+             - --csi-address=$(ADDRESS)
+-            - --v=5
+-            - --leader-election=true
++            - --v={{ .Values.logLevel }}
++            - --leader-election={{ ternary "true" "false" ( gt (.Values.replicaCount|int) 1 ) }}
+           env:
+             - name: ADDRESS
+               value: /var/lib/csi/sockets/pluginproxy/csi.sock
+@@ -180,7 +180,7 @@
+           image: {{ printf "%s:%s" .Values.sidecars.snapshotterImage.repository .Values.sidecars.snapshotterImage.tag }}
+           args:
+             - --csi-address=$(ADDRESS)
+-            - --leader-election=true
++            - --leader-election={{ ternary "true" "false" ( gt (.Values.replicaCount|int) 1 ) }}
+           env:
+             - name: ADDRESS
+               value: /var/lib/csi/sockets/pluginproxy/csi.sock
+@@ -205,7 +205,7 @@
+           imagePullPolicy: Always
+           args:
+             - --csi-address=$(ADDRESS)
+-            - --v=5
++            - --v={{ .Values.logLevel }}
+           env:
+             - name: ADDRESS
+               value: /var/lib/csi/sockets/pluginproxy/csi.sock
+diff -rtubN aws-ebs-csi-driver.orig/templates/node.yaml aws-ebs-csi-driver/templates/node.yaml
+--- aws-ebs-csi-driver.orig/templates/node.yaml	2021-03-05 03:10:41.000000000 +0100
++++ aws-ebs-csi-driver/templates/node.yaml	2021-03-05 10:30:07.391950366 +0100
+@@ -61,7 +61,7 @@
+             - --volume-attach-limit={{ .Values.volumeAttachLimit }}
+             {{- end }}
+             - --logtostderr
+-            - --v=5
++            - --v={{ .Values.logLevel }}
+           env:
+             - name: CSI_ENDPOINT
+               value: unix:/csi/csi.sock
+@@ -107,7 +107,7 @@
+           args:
+             - --csi-address=$(ADDRESS)
+             - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
+-            - --v=5
++            - --v={{ .Values.logLevel }}
+           env:
+             - name: ADDRESS
+               value: /csi/csi.sock
+diff -rtubN aws-ebs-csi-driver.orig/templates/statefulset.yaml aws-ebs-csi-driver/templates/statefulset.yaml
+--- aws-ebs-csi-driver.orig/templates/statefulset.yaml	2021-03-05 03:10:41.000000000 +0100
++++ aws-ebs-csi-driver/templates/statefulset.yaml	2021-03-05 10:29:31.881948744 +0100
+@@ -49,7 +49,7 @@
+             value: {{ .Values.proxy.no_proxy | quote }}
+ {{- end }}
+           args:
+-            - --v=5
++            - --v={{ .Values.logLevel }}
+             - --leader-election=false
+       {{- if .Values.imagePullSecrets }}
+       imagePullSecrets:
+diff -rtubN aws-ebs-csi-driver.orig/values.yaml aws-ebs-csi-driver/values.yaml
+--- aws-ebs-csi-driver.orig/values.yaml	2021-03-05 03:10:41.000000000 +0100
++++ aws-ebs-csi-driver/values.yaml	2021-03-05 10:29:31.881948744 +0100
+@@ -9,6 +9,8 @@
+   tag: "v0.9.0"
+   pullPolicy: IfNotPresent
+ 
++logLevel: 5
++
+ sidecars:
+   provisionerImage:
+     repository: k8s.gcr.io/sig-storage/csi-provisioner

charts/kubezero-aws-ebs-csi-driver/templates/storage-class.yaml

@@ -1,61 +0,0 @@
-{{- if .Values.StorageClass.create }}
-kind: StorageClass
-apiVersion: storage.k8s.io/v1
-metadata:
-  name: ebs-sc-gp2-xfs
-  labels:
-{{ include "kubezero-lib.labels" . | indent 4 }}
-provisioner: ebs.csi.aws.com
-volumeBindingMode: WaitForFirstConsumer
-parameters:
-  csi.storage.k8s.io/fstype: xfs
-  type: gp2
-  encrypted: "true"
-{{- if index .Values "aws-ebs-csi-driver" "enableVolumeResizing" }}
-allowVolumeExpansion: true
-{{- end }}
----
-kind: StorageClass
-apiVersion: storage.k8s.io/v1
-metadata:
-  name: ebs-sc-gp3-xfs
-  labels:
-{{ include "kubezero-lib.labels" . | indent 4 }}
-  {{- if .Values.StorageClass.default }}
-  annotations:
-    storageclass.kubernetes.io/is-default-class: "true"
-  {{- end }}
-provisioner: ebs.csi.aws.com
-volumeBindingMode: WaitForFirstConsumer
-parameters:
-  csi.storage.k8s.io/fstype: xfs
-  type: gp3
-  encrypted: "true"
-{{- if index .Values "aws-ebs-csi-driver" "enableVolumeResizing" }}
-allowVolumeExpansion: true
-{{- end }}
-
-{{- range .Values.StorageClass.Zones }}
----
-kind: StorageClass
-apiVersion: storage.k8s.io/v1
-metadata:
-  name: ebs-sc-gp2-xfs-{{ . }}
-  labels:
-{{ include "kubezero-lib.labels" . | indent 4 }}
-provisioner: ebs.csi.aws.com
-volumeBindingMode: WaitForFirstConsumer
-parameters:
-  csi.storage.k8s.io/fstype: xfs
-  type: gp2
-  encrypted: "true"
-{{- if index .Values "aws-ebs-csi-driver" "enableVolumeResizing" }}
-allowVolumeExpansion: true
-{{- end }}
-allowedTopologies:
-- matchLabelExpressions:
-  - key: failure-domain.beta.kubernetes.io/zone
-    values:
-    - {{ . }}
-{{- end }}
-{{- end }}

charts/kubezero-aws-ebs-csi-driver/update.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+
+VERSION=0.9.14
+
+rm -rf charts/aws-ebs-csi-driver
+curl -L -s -o - https://github.com/kubernetes-sigs/aws-ebs-csi-driver/releases/download/helm-chart-aws-ebs-csi-driver-${VERSION}/aws-ebs-csi-driver-${VERSION}.tgz | tar xfz - -C charts
+
+patch -d charts -i ../loglevel_leader.patch -p0 --no-backup-if-mismatch

charts/kubezero-aws-ebs-csi-driver/values.yaml

@@ -1,9 +1,10 @@
 aws-ebs-csi-driver:
   replicaCount: 1
+  logLevel: 1
 
   enableVolumeScheduling: true
-  enableVolumeResizing: false
+  enableVolumeResizing: true
-  enableVolumeSnapshot: false
+  enableVolumeSnapshot: true
 
   nodeSelector:
     node-role.kubernetes.io/master: ""
@@ -20,6 +21,28 @@ aws-ebs-csi-driver:
   extraVolumeTags: {}
   #  Name: KubeZero-Cluster
 
-StorageClass:
+  resources:
-  create: true
+    requests:
-  default: true
+      cpu: 10m
+      memory: 24Mi
+    limits:
+      # cpu: 50m
+      memory: 40Mi
+
+  storageClasses:
+  - name: ebs-sc-gp2-xfs
+    volumeBindingMode: WaitForFirstConsumer
+    allowVolumeExpansion: true
+    parameters:
+      csi.storage.k8s.io/fstype: xfs
+      type: gp2
+      encrypted: "true"
+  - name: ebs-sc-gp3-xfs
+    annotations:
+      storageclass.kubernetes.io/is-default-class: "true"
+    volumeBindingMode: WaitForFirstConsumer
+    allowVolumeExpansion: true
+    parameters:
+      csi.storage.k8s.io/fstype: xfs
+      type: gp3
+      encrypted: "true"

charts/kubezero-aws-efs-csi-driver/Chart.yaml

@@ -1,9 +1,9 @@
 apiVersion: v2
 name: kubezero-aws-efs-csi-driver
 description: KubeZero Umbrella Chart for aws-efs-csi-driver
-version: 0.2.0
+version: 0.3.2
-appVersion: 1.0.0
+appVersion: 1.2.0
-kubeVersion: ">=1.16.0-0"
+kubeVersion: ">=1.18.0-0"
 home: https://kubezero.com
 icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png
 sources:
@@ -20,7 +20,6 @@ dependencies:
   - name: kubezero-lib
     version: ">= 0.1.3"
     repository: https://zero-down-time.github.io/kubezero/
-# Once they properly update upstream
+  - name: aws-efs-csi-driver
-#  - name: aws-ebs-csi-driver
+    version: 1.2.1
-#    version: 1.0.0
+    # repository: https://kubernetes-sigs.github.io/aws-efs-csi-driver/
-#    repository: https://kubernetes-sigs.github.io/aws-efs-csi-driver

charts/kubezero-aws-efs-csi-driver/README.md

@@ -1,6 +1,6 @@
 # kubezero-aws-efs-csi-driver
 
-![Version: 0.2.0](https://img.shields.io/badge/Version-0.2.0-informational?style=flat-square) ![AppVersion: 1.0.0](https://img.shields.io/badge/AppVersion-1.0.0-informational?style=flat-square)
+![Version: 0.3.2](https://img.shields.io/badge/Version-0.3.2-informational?style=flat-square) ![AppVersion: 1.2.0](https://img.shields.io/badge/AppVersion-1.2.0-informational?style=flat-square)
 
 KubeZero Umbrella Chart for aws-efs-csi-driver
 
@@ -19,10 +19,11 @@ KubeZero Umbrella Chart for aws-efs-csi-driver
 
 ## Requirements
 
-Kubernetes: `>=1.16.0-0`
+Kubernetes: `>=1.18.0-0`
 
 | Repository | Name | Version |
 |------------|------|---------|
+|  | aws-efs-csi-driver | 1.2.1 |
 | https://zero-down-time.github.io/kubezero/ | kubezero-lib | >= 0.1.3 |
 
 ## Storage Class
@@ -38,4 +39,9 @@ Details also see: [Reserve PV](https://kubernetes.io/docs/concepts/storage/persi
 |-----|------|---------|-------------|
 | StorageClass.create | bool | `true` |  |
 | StorageClass.default | bool | `false` |  |
-| aws-efs-csi-driver.nodeSelector | object | `{}` |  |
+| aws-efs-csi-driver.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key | string | `"node.kubernetes.io/csi.efs.fs"` |  |
+| aws-efs-csi-driver.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].operator | string | `"Exists"` |  |
+| aws-efs-csi-driver.logLevel | int | `1` |  |
+| aws-efs-csi-driver.resources.limits.memory | string | `"128Mi"` |  |
+| aws-efs-csi-driver.resources.requests.cpu | string | `"20m"` |  |
+| aws-efs-csi-driver.resources.requests.memory | string | `"64Mi"` |  |

charts/kubezero-aws-efs-csi-driver/affinity_resources.patch

@@ -0,0 +1,53 @@
+diff -tubNr charts/aws-efs-csi-driver/templates/node-daemonset.yaml charts/aws-efs-csi-driver.zdt/templates/node-daemonset.yaml
+--- charts/aws-efs-csi-driver/templates/node-daemonset.yaml	2021-03-19 00:22:54.000000000 +0100
++++ charts/aws-efs-csi-driver.zdt/templates/node-daemonset.yaml	2021-03-24 13:06:56.105762451 +0100
+@@ -40,15 +40,10 @@
+         {{- with .Values.nodeSelector }}
+         {{- . | toYaml | nindent 8 }}
+         {{- end }}
++      {{- with .Values.affinity }}
+       affinity:
+-        nodeAffinity:
+-          requiredDuringSchedulingIgnoredDuringExecution:
+-            nodeSelectorTerms:
+-            - matchExpressions:
+-              - key: eks.amazonaws.com/compute-type
+-                operator: NotIn
+-                values:
+-                - fargate
++        {{- . | toYaml | nindent 8 }}
++      {{- end }}
+       hostNetwork: true
+       {{- if .Values.dnsPolicy }}
+       dnsPolicy: "{{ .Values.dnsPolicy }}"
+@@ -99,6 +94,10 @@
+             timeoutSeconds: 3
+             periodSeconds: 2
+             failureThreshold: 5
++          {{- with .Values.resources }}
++          resources:
++            {{- . | toYaml | nindent 12 }}
++          {{- end }}
+         - name: csi-driver-registrar
+           image: {{ printf "%s:%s" .Values.sidecars.nodeDriverRegistrarImage.repository .Values.sidecars.nodeDriverRegistrarImage.tag }}
+           args:
+diff -tubNr charts/aws-efs-csi-driver/values.yaml charts/aws-efs-csi-driver.zdt/values.yaml
+--- charts/aws-efs-csi-driver/values.yaml	2021-03-19 00:22:54.000000000 +0100
++++ charts/aws-efs-csi-driver.zdt/values.yaml	2021-03-24 13:06:56.105762451 +0100
+@@ -43,7 +43,15 @@
+ 
+ tolerations: []
+ 
+-affinity: {}
++affinity:
++  nodeAffinity:
++    requiredDuringSchedulingIgnoredDuringExecution:
++      nodeSelectorTerms:
++      - matchExpressions:
++        - key: eks.amazonaws.com/compute-type
++          operator: NotIn
++          values:
++          - fargate
+ 
+ node:
+   podAnnotations: {}

charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/.helmignore

@@ -0,0 +1,22 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/

charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/Chart.yaml

@@ -1,15 +1,18 @@
 apiVersion: v1
-appVersion: "1.0.0"
+appVersion: 1.2.0
-name: aws-efs-csi-driver
 description: A Helm chart for AWS EFS CSI Driver
-version: 0.2.0
-kubeVersion: ">=1.14.0-0"
 home: https://github.com/kubernetes-sigs/aws-efs-csi-driver
-sources:
-  - https://github.com/kubernetes-sigs/aws-efs-csi-driver
 keywords:
-  - aws
+- aws
-  - efs
+- efs
-  - csi
+- csi
+kubeVersion: '>=1.17.0-0'
 maintainers:
-  - name: leakingtapan
+- name: leakingtapan
+  url: https://github.com/leakingtapan
+- name: krmichel
+  url: https://github.com/krmichel
+name: aws-efs-csi-driver
+sources:
+- https://github.com/kubernetes-sigs/aws-efs-csi-driver
+version: 1.2.1

charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/NOTES.txt

@@ -1,3 +1,3 @@
 To verify that aws-efs-csi-driver has started, run:
 
-    kubectl get pod -n kube-system -l "app.kubernetes.io/name={{ include "aws-efs-csi-driver.name" . }},app.kubernetes.io/instance={{ .Release.Name }}"
+    kubectl get pod -n {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "aws-efs-csi-driver.name" . }},app.kubernetes.io/instance={{ .Release.Name }}"

charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/_helpers.tpl

@@ -43,3 +43,14 @@ app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
 {{- end }}
 app.kubernetes.io/managed-by: {{ .Release.Service }}
 {{- end -}}
+
+{{/*
+Create the name of the service account to use
+*/}}
+{{- define "aws-efs-csi-driver.serviceAccountName" -}}
+{{- if .Values.serviceAccount.controller.create -}}
+    {{ default (include "aws-efs-csi-driver.fullname" .) .Values.serviceAccount.controller.name }}
+{{- else -}}
+    {{ default "default" .Values.serviceAccount.controller.name }}
+{{- end -}}
+{{- end -}}

charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/controller-deployment.yaml

@@ -0,0 +1,100 @@
+{{- if .Values.controller.create }}
+# Controller Service
+kind: Deployment
+apiVersion: apps/v1
+metadata:
+  name: efs-csi-controller
+  labels:
+    app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }}
+spec:
+  replicas: {{ .Values.replicaCount }}
+  selector:
+    matchLabels:
+      app: efs-csi-controller
+      app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }}
+      app.kubernetes.io/instance: {{ .Release.Name }}
+  template:
+    metadata:
+      labels:
+        app: efs-csi-controller
+        app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }}
+        app.kubernetes.io/instance: {{ .Release.Name }}
+      {{- if .Values.node.podAnnotations }}
+      annotations: {{ toYaml .Values.node.podAnnotations | nindent 8 }}
+      {{- end }}
+    spec:
+      {{- if .Values.imagePullSecrets }}
+      imagePullSecrets:
+        {{- range .Values.imagePullSecrets }}
+        - name: {{ . }}
+      {{- end }}
+      {{- end }}
+      nodeSelector:
+        kubernetes.io/os: linux
+        {{- with .Values.nodeSelector }}
+        {{- . | toYaml | nindent 8 }}
+        {{- end }}
+      {{- if .Values.serviceAccount.controller.create }}
+      serviceAccountName: {{ include "aws-efs-csi-driver.serviceAccountName" . }}
+      {{- end }}
+      priorityClassName: system-cluster-critical
+      tolerations:
+        - operator: Exists
+        {{- with .Values.node.tolerations }}
+        {{- . | toYaml | nindent 8 }}
+        {{- end }}
+      containers:
+        - name: efs-plugin
+          securityContext:
+            privileged: true
+          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+          imagePullPolicy: IfNotPresent
+          args:
+            - --endpoint=$(CSI_ENDPOINT)
+            - --logtostderr
+            - --v=5
+            # Uncomment below line to allow access point root directory to be deleted by controller.
+            #- --delete-access-point-root-dir
+          env:
+            - name: CSI_ENDPOINT
+              value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
+          volumeMounts:
+            - name: socket-dir
+              mountPath: /var/lib/csi/sockets/pluginproxy/
+          ports:
+            - name: healthz
+              containerPort: 9808
+              protocol: TCP
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: healthz
+            initialDelaySeconds: 10
+            timeoutSeconds: 3
+            periodSeconds: 10
+            failureThreshold: 5
+        - name: csi-provisioner
+          image: {{ printf "%s:%s" .Values.sidecars.csiProvisionerImage.repository .Values.sidecars.csiProvisionerImage.tag }}
+          args:
+            - --csi-address=$(ADDRESS)
+            - --v=5
+            - --feature-gates=Topology=true
+            - --leader-election
+          env:
+            - name: ADDRESS
+              value: /var/lib/csi/sockets/pluginproxy/csi.sock
+          volumeMounts:
+            - name: socket-dir
+              mountPath: /var/lib/csi/sockets/pluginproxy/
+        - name: liveness-probe
+          image: {{ printf "%s:%s" .Values.sidecars.livenessProbeImage.repository .Values.sidecars.livenessProbeImage.tag }}
+          args:
+            - --csi-address=/csi/csi.sock
+            - --health-port=9808
+          volumeMounts:
+            - name: socket-dir
+              mountPath: /csi
+      volumes:
+        - name: socket-dir
+          emptyDir: {}
+{{- end }}

charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/controller-serviceaccount.yaml

@@ -0,0 +1,59 @@
+{{- if .Values.serviceAccount.controller.create -}}
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: {{ include "aws-efs-csi-driver.serviceAccountName" . }}
+  labels:
+    app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }}
+  {{- with .Values.serviceAccount.controller.annotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+---
+
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: efs-csi-external-provisioner-role
+  labels:
+    app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }}
+rules:
+  - apiGroups: [""]
+    resources: ["persistentvolumes"]
+    verbs: ["get", "list", "watch", "create", "delete"]
+  - apiGroups: [""]
+    resources: ["persistentvolumeclaims"]
+    verbs: ["get", "list", "watch", "update"]
+  - apiGroups: ["storage.k8s.io"]
+    resources: ["storageclasses"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["events"]
+    verbs: ["list", "watch", "create"]
+  - apiGroups: ["storage.k8s.io"]
+    resources: ["csinodes"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["nodes"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["coordination.k8s.io"]
+    resources: ["leases"]
+    verbs: ["get", "watch", "list", "delete", "update", "create"]
+
+---
+
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: efs-csi-provisioner-binding
+  labels:
+    app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }}
+subjects:
+  - kind: ServiceAccount
+    name: {{ include "aws-efs-csi-driver.serviceAccountName" . }}
+    namespace: {{ .Release.Namespace }}
+roleRef:
+  kind: ClusterRole
+  name: efs-csi-external-provisioner-role
+  apiGroup: rbac.authorization.k8s.io
+{{- end }}

charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/csidriver.yaml

@@ -2,5 +2,9 @@ apiVersion: storage.k8s.io/v1beta1
 kind: CSIDriver
 metadata:
   name: efs.csi.aws.com
+  annotations:
+    "helm.sh/hook": pre-install, pre-upgrade
+    "helm.sh/hook-delete-policy": before-hook-creation
+    "helm.sh/resource-policy": keep
 spec:
   attachRequired: false

charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/node-daemonset.yaml

@@ -3,7 +3,8 @@ kind: DaemonSet
 apiVersion: apps/v1
 metadata:
   name: efs-csi-node
-  namespace: kube-system
+  labels:
+    app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }}
 spec:
   selector:
     matchLabels:
@@ -20,12 +21,37 @@ spec:
       annotations: {{ toYaml .Values.node.podAnnotations | nindent 8 }}
       {{- end }}
     spec:
+    {{- if .Values.hostAliases }}
+      hostAliases:
+      {{- range $k, $v := .Values.hostAliases }}
+        - ip: {{ $v.ip }}
+          hostnames:
+            - {{ $k }}.efs.{{ $v.region }}.amazonaws.com
+      {{- end }}
+    {{- end }}
+    {{- if .Values.imagePullSecrets }}
+      imagePullSecrets:
+      {{- range .Values.imagePullSecrets }}
+        - name: {{ . }}
+      {{- end }}
+    {{- end }}
       nodeSelector:
         beta.kubernetes.io/os: linux
         {{- with .Values.nodeSelector }}
         {{- . | toYaml | nindent 8 }}
         {{- end }}
+      {{- with .Values.affinity }}
+      affinity:
+        {{- . | toYaml | nindent 8 }}
+      {{- end }}
       hostNetwork: true
+      {{- if .Values.dnsPolicy }}
+      dnsPolicy: "{{ .Values.dnsPolicy }}"
+      {{- end }}
+      {{- with .Values.dnsConfig }}
+      dnsConfig:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
       priorityClassName: system-node-critical
       tolerations:
         - operator: Exists
@@ -40,7 +66,7 @@ spec:
           args:
             - --endpoint=$(CSI_ENDPOINT)
             - --logtostderr
-            - --v=5
+            - --v={{ .Values.logLevel }}
           env:
             - name: CSI_ENDPOINT
               value: unix:/csi/csi.sock
@@ -53,7 +79,9 @@ spec:
             - name: efs-state-dir
               mountPath: /var/run/efs
             - name: efs-utils-config
-              mountPath: /etc/amazon/efs
+              mountPath: /var/amazon/efs
+            - name: efs-utils-config-legacy
+              mountPath: /etc/amazon/efs-legacy
           ports:
             - name: healthz
               containerPort: 9809
@@ -66,12 +94,16 @@ spec:
             timeoutSeconds: 3
             periodSeconds: 2
             failureThreshold: 5
-        - name: cs-driver-registrar
+          {{- with .Values.resources }}
+          resources:
+            {{- . | toYaml | nindent 12 }}
+          {{- end }}
+        - name: csi-driver-registrar
           image: {{ printf "%s:%s" .Values.sidecars.nodeDriverRegistrarImage.repository .Values.sidecars.nodeDriverRegistrarImage.tag }}
           args:
             - --csi-address=$(ADDRESS)
             - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
-            - --v=5
+            - --v={{ .Values.logLevel }}
           env:
             - name: ADDRESS
               value: /csi/csi.sock
@@ -91,6 +123,7 @@ spec:
           args:
             - --csi-address=/csi/csi.sock
             - --health-port=9809
+            - --v={{ .Values.logLevel }}
           volumeMounts:
             - name: plugin-dir
               mountPath: /csi
@@ -112,6 +145,10 @@ spec:
             path: /var/run/efs
             type: DirectoryOrCreate
         - name: efs-utils-config
+          hostPath:
+            path: /var/amazon/efs
+            type: DirectoryOrCreate
+        - name: efs-utils-config-legacy
           hostPath:
             path: /etc/amazon/efs
             type: DirectoryOrCreate

charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/storageclass.yaml

@@ -0,0 +1,15 @@
+{{- range .Values.storageClasses }}
+kind: StorageClass
+apiVersion: storage.k8s.io/v1
+metadata:
+  name: {{ .name }}
+provisioner: efs.csi.aws.com
+{{- with .mountOptions }}
+mountOptions: 
+{{ toYaml . }}
+{{- end }}
+{{- with .parameters }}
+parameters:
+{{ toYaml . | indent 2 }}
+{{- end }}
+{{- end }}

charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/values.yaml

@@ -6,16 +6,19 @@ replicaCount: 2
 
 image:
   repository: amazon/aws-efs-csi-driver
-  tag: "v1.0.0"
+  tag: "v1.2.0"
   pullPolicy: IfNotPresent
 
 sidecars:
   livenessProbeImage:
-    repository: quay.io/k8scsi/livenessprobe
+    repository: public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe
-    tag: "v2.0.0"
+    tag: v2.1.0-eks-1-18-1
   nodeDriverRegistrarImage:
-    repository: quay.io/k8scsi/csi-node-driver-registrar
+    repository: public.ecr.aws/eks-distro/kubernetes-csi/node-driver-registrar
-    tag: "v1.3.0"
+    tag: v2.0.1-eks-1-18-1
+  csiProvisionerImage:
+    repository: public.ecr.aws/eks-distro/kubernetes-csi/external-provisioner
+    tag: v2.0.3-eks-1-18-1
 
 imagePullSecrets: []
 nameOverride: ""
@@ -23,7 +26,8 @@ fullnameOverride: ""
 
 podAnnotations: {}
 
-resources: {}
+resources:
+  {}
   # We usually recommend not to specify default resources and to leave this as a conscious
   # choice for the user. This also increases chances charts run on environments with little
   # resources, such as Minikube. If you do want to specify resources, uncomment the following
@@ -39,8 +43,63 @@ nodeSelector: {}
 
 tolerations: []
 
-affinity: {}
+affinity:
+  nodeAffinity:
+    requiredDuringSchedulingIgnoredDuringExecution:
+      nodeSelectorTerms:
+      - matchExpressions:
+        - key: eks.amazonaws.com/compute-type
+          operator: NotIn
+          values:
+          - fargate
 
 node:
   podAnnotations: {}
   tolerations: []
+
+logLevel: 5
+
+hostAliases:
+  {}
+  # for cross VPC EFS, you need to poison or overwrite the DNS for the efs volume as per
+  # https://docs.aws.amazon.com/efs/latest/ug/efs-different-vpc.html#wt6-efs-utils-step3
+  # implementing the suggested solution found here:
+  # https://github.com/kubernetes-sigs/aws-efs-csi-driver/issues/240#issuecomment-676849346
+  # EFS Vol ID, IP, Region
+  # "fs-01234567":
+  #   ip: 10.10.2.2
+  #   region: us-east-2
+
+dnsPolicy: ""
+dnsConfig:
+  {}
+  # Example config which uses the AWS nameservers
+  # dnsPolicy: "None"
+  # dnsConfig:
+  #   nameservers:
+  #     - 169.254.169.253
+
+serviceAccount:
+  controller:
+    # Specifies whether a service account should be created
+    create: true
+    annotations: {}
+    ## Enable if EKS IAM for SA is used
+    #  eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/efs-csi-role
+    name: efs-csi-controller-sa
+
+controller:
+  create: true
+
+storageClasses: []
+# Add StorageClass resources like:
+# - name: efs-sc
+#   mountOptions:
+#   - tls
+#   parameters:
+#     provisioningMode: efs-ap
+#     fileSystemId: fs-92107410
+#     directoryPerms: "700"
+#     gidRangeStart: "1000"
+#     gidRangeEnd: "2000"
+#     basePath: "/dynamic_provisioning"

charts/kubezero-aws-efs-csi-driver/update.sh

@@ -1,14 +1,8 @@
 #!/bin/bash
-set -ex
 
-# Upstream doesnt have proper Helm repo yet so we just download latest release and stuff it into charts
+VERSION=1.2.1
 
-REPO="kubernetes-sigs/aws-efs-csi-driver"
+rm -rf charts/aws-efs-csi-driver
-[ -z "$RELEASE" ] && RELEASE=$(curl -sL -s https://api.github.com/repos/${REPO}/releases | grep '"tag_name":' | cut -d'"' -f4 | grep -v -E "(alpha|beta|rc)" | sort -t"." -k 1,1 -k 2,2 -k 3,3 -k 4,4 | tail -n 1)
+curl -L -s -o - https://github.com/kubernetes-sigs/aws-efs-csi-driver/releases/download/helm-chart-aws-efs-csi-driver-${VERSION}/aws-efs-csi-driver-${VERSION}.tgz | tar xfz - -C charts
 
-rm -rf git
+patch -i affinity_resources.patch -p0 --no-backup-if-mismatch
-git clone https://github.com/$REPO.git git
-cd git && git checkout $RELEASE && cd -
-
-rm -rf charts/aws-efs-csi-driver && mkdir -p charts/aws-efs-csi-driver
-mv git/helm/* charts/aws-efs-csi-driver

charts/kubezero-aws-efs-csi-driver/values.yaml

@@ -1,6 +1,20 @@
 aws-efs-csi-driver:
-  nodeSelector:  {}
+  logLevel: 1
-    # node.kubernetes.io/csi.efs.fs: EFS-FS_ID
+
+  resources:
+    requests:
+      cpu: 20m
+      memory: 64Mi
+    limits:
+      memory: 128Mi
+
+  affinity:
+    nodeAffinity:
+      requiredDuringSchedulingIgnoredDuringExecution:
+        nodeSelectorTerms:
+        - matchExpressions:
+          - key: node.kubernetes.io/csi.efs.fs
+            operator: Exists
 
 StorageClass:
   create: true

charts/kubezero-aws-node-termination-handler/Chart.yaml

@@ -0,0 +1,20 @@
+apiVersion: v2
+name: kubezero-aws-node-termination-handler
+description: Umbrella chart for all aws-node-termination-handler
+type: application
+version: 0.1.2
+home: https://kubezero.com
+icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png
+keywords:
+  - kubezero
+  - aws-node-termination-handler
+maintainers:
+  - name: Quarky9
+dependencies:
+  - name: kubezero-lib
+    version: ">= 0.1.3"
+    repository: https://zero-down-time.github.io/kubezero/
+  - name: aws-node-termination-handler
+    version: ">= 0.14.1"
+    repository: https://aws.github.io/eks-charts
+kubeVersion: ">= 1.18.0"

charts/kubezero-aws-node-termination-handler/README.md

@@ -0,0 +1,48 @@
+# kubezero-aws-node-termination-handler
+
+![Version: 0.1.2](https://img.shields.io/badge/Version-0.1.2-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square)
+
+Umbrella chart for all aws-node-termination-handler
+
+**Homepage:** <https://kubezero.com>
+
+## Maintainers
+
+| Name | Email | Url |
+| ---- | ------ | --- |
+| Quarky9 |  |  |
+
+## Requirements
+
+Kubernetes: `>= 1.18.0`
+
+| Repository | Name | Version |
+|------------|------|---------|
+| https://aws.github.io/eks-charts | aws-node-termination-handler | >= 0.14.1 |
+| https://zero-down-time.github.io/kubezero/ | kubezero-lib | >= 0.1.3 |
+
+## Values
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| aws-node-termination-handler.deleteLocalData | bool | `true` |  |
+| aws-node-termination-handler.enablePrometheusServer | bool | `false` |  |
+| aws-node-termination-handler.enableSqsTerminationDraining | bool | `true` |  |
+| aws-node-termination-handler.jsonLogging | bool | `true` |  |
+| aws-node-termination-handler.nodeSelector."node-role.kubernetes.io/master" | string | `""` |  |
+| aws-node-termination-handler.podMonitor.create | bool | `false` |  |
+| aws-node-termination-handler.podMonitor.labels.release | string | `"metrics"` |  |
+| aws-node-termination-handler.taintNode | bool | `true` |  |
+| aws-node-termination-handler.tolerations[0].effect | string | `"NoSchedule"` |  |
+| aws-node-termination-handler.tolerations[0].key | string | `"node-role.kubernetes.io/master"` |  |
+
+## KubeZero default configuration
+
+- enable SQS Mode
+- allow draining of localdata
+- enable prometheus
+
+## Resources
+
+- https://github.com/aws/aws-node-termination-handler
+- https://github.com/aws/eks-charts/tree/master/stable/aws-node-termination-handler

charts/kubezero-aws-node-termination-handler/README.md.gotmpl

@@ -0,0 +1,27 @@
+{{ template "chart.header" . }}
+{{ template "chart.deprecationWarning" . }}
+
+{{ template "chart.versionBadge" . }}{{ template "chart.typeBadge" . }}{{ template "chart.appVersionBadge" . }}
+
+{{ template "chart.description" . }}
+
+{{ template "chart.homepageLine" . }}
+
+{{ template "chart.maintainersSection" . }}
+
+{{ template "chart.sourcesSection" . }}
+
+{{ template "chart.requirementsSection" . }}
+
+{{ template "chart.valuesSection" . }}
+
+## KubeZero default configuration
+
+- enable SQS Mode
+- allow draining of localdata
+- enable prometheus
+
+## Resources
+
+- https://github.com/aws/aws-node-termination-handler
+- https://github.com/aws/eks-charts/tree/master/stable/aws-node-termination-handler

charts/kubezero-aws-node-termination-handler/values.yaml

@@ -0,0 +1,19 @@
+aws-node-termination-handler:
+  enableSqsTerminationDraining: true
+  # queueURL: <SQS queue ARN from kube controller stack>
+  deleteLocalData: true
+  taintNode: true
+
+  enablePrometheusServer: false
+  podMonitor:
+    create: false
+    labels:
+      release: metrics
+
+  jsonLogging: true
+
+  tolerations:
+  - key: node-role.kubernetes.io/master
+    effect: NoSchedule
+  nodeSelector:
+    node-role.kubernetes.io/master: ""

charts/kubezero-calico/Chart.yaml

@@ -2,7 +2,7 @@ apiVersion: v2
 name: kubezero-calico
 description: KubeZero Umbrella Chart for Calico
 type: application
-version: 0.2.1
+version: 0.2.2
 appVersion: v3.16.5
 home: https://kubezero.com
 icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png

charts/kubezero-calico/README.md

@@ -1,6 +1,6 @@
 # kubezero-calico
 
-![Version: 0.2.1](https://img.shields.io/badge/Version-0.2.1-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v3.16.5](https://img.shields.io/badge/AppVersion-v3.16.5-informational?style=flat-square)
+![Version: 0.2.2](https://img.shields.io/badge/Version-0.2.2-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v3.16.5](https://img.shields.io/badge/AppVersion-v3.16.5-informational?style=flat-square)
 
 KubeZero Umbrella Chart for Calico
 

charts/kubezero-calico/templates/calico.yaml

@@ -482,7 +482,7 @@ spec:
             privileged: true
           resources:
             requests:
-              cpu: 250m
+              cpu: 50m
           livenessProbe:
             exec:
               command:
@@ -619,6 +619,9 @@ spec:
               command:
               - /usr/bin/check-status
               - -r
+          resources:
+            requests:
+              cpu: 50m
 
 ---
 

charts/kubezero-cert-manager/Chart.yaml

@@ -2,7 +2,7 @@ apiVersion: v2
 name: kubezero-cert-manager
 description: KubeZero Umbrella Chart for cert-manager
 type: application
-version: 0.4.1
+version: 0.5.0
 home: https://kubezero.com
 icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png
 keywords:
@@ -15,7 +15,7 @@ dependencies:
     version: ">= 0.1.3"
     repository: https://zero-down-time.github.io/kubezero/
   - name: cert-manager
-    version: 1.1.0
+    version: 1.2.0
     repository: https://charts.jetstack.io
     condition: cert-manager.enabled
-kubeVersion: ">= 1.16.0"
+kubeVersion: ">= 1.18.0"

charts/kubezero-cert-manager/README.md

@@ -1,6 +1,6 @@
 # kubezero-cert-manager
 
-![Version: 0.4.1](https://img.shields.io/badge/Version-0.4.1-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square)
+![Version: 0.5.0](https://img.shields.io/badge/Version-0.5.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square)
 
 KubeZero Umbrella Chart for cert-manager
 
@@ -14,11 +14,11 @@ KubeZero Umbrella Chart for cert-manager
 
 ## Requirements
 
-Kubernetes: `>= 1.16.0`
+Kubernetes: `>= 1.18.0`
 
 | Repository | Name | Version |
 |------------|------|---------|
-| https://charts.jetstack.io | cert-manager | 1.1.0 |
+| https://charts.jetstack.io | cert-manager | 1.2.0 |
 | https://zero-down-time.github.io/kubezero/ | kubezero-lib | >= 0.1.3 |
 
 ## AWS - IAM Role

charts/kubezero-istio-ingress/Chart.yaml

@@ -2,8 +2,8 @@ apiVersion: v2
 name: kubezero-istio-ingress
 description: KubeZero Umbrella Chart for Istio based Ingress
 type: application
-version: 0.1.1
+version: 0.5.0
-appVersion: 1.8.1
+appVersion: 1.9.1
 home: https://kubezero.com
 icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png
 keywords:
@@ -16,9 +16,9 @@ dependencies:
     version: ">= 0.1.3"
     repository: https://zero-down-time.github.io/kubezero/
   - name: istio-ingress
-    version: 1.1.0
+    version: 1.9.1
     condition: istio-ingress.enabled
   - name: istio-private-ingress
-    version: 1.1.0
+    version: 1.9.1
     condition: istio-private-ingress.enabled
-kubeVersion: ">= 1.16.0"
+kubeVersion: ">= 1.18.0"

charts/kubezero-istio-ingress/README.md

@@ -1,6 +1,6 @@
 # kubezero-istio-ingress
 
-![Version: 0.1.1](https://img.shields.io/badge/Version-0.1.1-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.8.1](https://img.shields.io/badge/AppVersion-1.8.1-informational?style=flat-square)
+![Version: 0.5.0](https://img.shields.io/badge/Version-0.5.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.9.1](https://img.shields.io/badge/AppVersion-1.9.1-informational?style=flat-square)
 
 KubeZero Umbrella Chart for Istio based Ingress
 
@@ -16,12 +16,12 @@ Installs Istio Ingress Gateways, requires kubezero-istio to be installed !
 
 ## Requirements
 
-Kubernetes: `>= 1.16.0`
+Kubernetes: `>= 1.18.0`
 
 | Repository | Name | Version |
 |------------|------|---------|
-|  | istio-ingress | 1.1.0 |
+|  | istio-ingress | 1.9.1 |
-|  | istio-private-ingress | 1.1.0 |
+|  | istio-private-ingress | 1.9.1 |
 | https://zero-down-time.github.io/kubezero/ | kubezero-lib | >= 0.1.3 |
 
 ## Values
@@ -30,11 +30,9 @@ Kubernetes: `>= 1.16.0`
 |-----|------|---------|-------------|
 | global.arch.amd64 | int | `2` |  |
 | global.defaultPodDisruptionBudget.enabled | bool | `false` |  |
-| global.hub | string | `"docker.io/istio"` |  |
 | global.jwtPolicy | string | `"first-party-jwt"` |  |
 | global.logAsJson | bool | `true` |  |
 | global.priorityClassName | string | `"system-cluster-critical"` |  |
-| global.tag | string | `"1.8.1"` |  |
 | istio-ingress.dnsNames | list | `[]` |  |
 | istio-ingress.enabled | bool | `false` |  |
 | istio-ingress.gateways.istio-ingressgateway.autoscaleEnabled | bool | `false` |  |
@@ -45,16 +43,19 @@ Kubernetes: `>= 1.16.0`
 | istio-ingress.gateways.istio-ingressgateway.podAntiAffinityLabelSelector[0].operator | string | `"In"` |  |
 | istio-ingress.gateways.istio-ingressgateway.podAntiAffinityLabelSelector[0].topologyKey | string | `"kubernetes.io/hostname"` |  |
 | istio-ingress.gateways.istio-ingressgateway.podAntiAffinityLabelSelector[0].values | string | `"istio-ingressgateway"` |  |
-| istio-ingress.gateways.istio-ingressgateway.ports[0].name | string | `"http-status"` |  |
+| istio-ingress.gateways.istio-ingressgateway.ports[0].name | string | `"status-port"` |  |
 | istio-ingress.gateways.istio-ingressgateway.ports[0].nodePort | int | `30021` |  |
 | istio-ingress.gateways.istio-ingressgateway.ports[0].port | int | `15021` |  |
+| istio-ingress.gateways.istio-ingressgateway.ports[0].protocol | string | `"TCP"` |  |
 | istio-ingress.gateways.istio-ingressgateway.ports[1].name | string | `"http2"` |  |
 | istio-ingress.gateways.istio-ingressgateway.ports[1].nodePort | int | `30080` |  |
 | istio-ingress.gateways.istio-ingressgateway.ports[1].port | int | `80` |  |
+| istio-ingress.gateways.istio-ingressgateway.ports[1].protocol | string | `"TCP"` |  |
 | istio-ingress.gateways.istio-ingressgateway.ports[1].targetPort | int | `8080` |  |
 | istio-ingress.gateways.istio-ingressgateway.ports[2].name | string | `"https"` |  |
 | istio-ingress.gateways.istio-ingressgateway.ports[2].nodePort | int | `30443` |  |
 | istio-ingress.gateways.istio-ingressgateway.ports[2].port | int | `443` |  |
+| istio-ingress.gateways.istio-ingressgateway.ports[2].protocol | string | `"TCP"` |  |
 | istio-ingress.gateways.istio-ingressgateway.ports[2].targetPort | int | `8443` |  |
 | istio-ingress.gateways.istio-ingressgateway.replicaCount | int | `1` |  |
 | istio-ingress.gateways.istio-ingressgateway.resources.limits.memory | string | `"256Mi"` |  |
@@ -73,17 +74,30 @@ Kubernetes: `>= 1.16.0`
 | istio-private-ingress.gateways.istio-ingressgateway.podAntiAffinityLabelSelector[0].operator | string | `"In"` |  |
 | istio-private-ingress.gateways.istio-ingressgateway.podAntiAffinityLabelSelector[0].topologyKey | string | `"kubernetes.io/hostname"` |  |
 | istio-private-ingress.gateways.istio-ingressgateway.podAntiAffinityLabelSelector[0].values | string | `"istio-private-ingressgateway"` |  |
-| istio-private-ingress.gateways.istio-ingressgateway.ports[0].name | string | `"http-status"` |  |
+| istio-private-ingress.gateways.istio-ingressgateway.ports[0].name | string | `"status-port"` |  |
 | istio-private-ingress.gateways.istio-ingressgateway.ports[0].nodePort | int | `31021` |  |
 | istio-private-ingress.gateways.istio-ingressgateway.ports[0].port | int | `15021` |  |
+| istio-private-ingress.gateways.istio-ingressgateway.ports[0].protocol | string | `"TCP"` |  |
 | istio-private-ingress.gateways.istio-ingressgateway.ports[1].name | string | `"http2"` |  |
 | istio-private-ingress.gateways.istio-ingressgateway.ports[1].nodePort | int | `31080` |  |
 | istio-private-ingress.gateways.istio-ingressgateway.ports[1].port | int | `80` |  |
+| istio-private-ingress.gateways.istio-ingressgateway.ports[1].protocol | string | `"TCP"` |  |
 | istio-private-ingress.gateways.istio-ingressgateway.ports[1].targetPort | int | `8080` |  |
 | istio-private-ingress.gateways.istio-ingressgateway.ports[2].name | string | `"https"` |  |
 | istio-private-ingress.gateways.istio-ingressgateway.ports[2].nodePort | int | `31443` |  |
 | istio-private-ingress.gateways.istio-ingressgateway.ports[2].port | int | `443` |  |
+| istio-private-ingress.gateways.istio-ingressgateway.ports[2].protocol | string | `"TCP"` |  |
 | istio-private-ingress.gateways.istio-ingressgateway.ports[2].targetPort | int | `8443` |  |
+| istio-private-ingress.gateways.istio-ingressgateway.ports[3].name | string | `"tcp-istiod"` |  |
+| istio-private-ingress.gateways.istio-ingressgateway.ports[3].nodePort | int | `31012` |  |
+| istio-private-ingress.gateways.istio-ingressgateway.ports[3].port | int | `15012` |  |
+| istio-private-ingress.gateways.istio-ingressgateway.ports[3].protocol | string | `"TCP"` |  |
+| istio-private-ingress.gateways.istio-ingressgateway.ports[3].targetPort | int | `15012` |  |
+| istio-private-ingress.gateways.istio-ingressgateway.ports[4].name | string | `"tls"` |  |
+| istio-private-ingress.gateways.istio-ingressgateway.ports[4].nodePort | int | `31044` |  |
+| istio-private-ingress.gateways.istio-ingressgateway.ports[4].port | int | `15443` |  |
+| istio-private-ingress.gateways.istio-ingressgateway.ports[4].protocol | string | `"TCP"` |  |
+| istio-private-ingress.gateways.istio-ingressgateway.ports[4].targetPort | int | `15443` |  |
 | istio-private-ingress.gateways.istio-ingressgateway.replicaCount | int | `1` |  |
 | istio-private-ingress.gateways.istio-ingressgateway.resources.limits.memory | string | `"256Mi"` |  |
 | istio-private-ingress.gateways.istio-ingressgateway.resources.requests.cpu | string | `"100m"` |  |

charts/kubezero-istio-ingress/charts/istio-ingress/Chart.yaml

@@ -1,6 +1,6 @@
 apiVersion: v1
 name: istio-ingress
-version: 1.1.0
+version: 1.9.1
 tillerVersion: ">=2.7.2"
 description: Helm chart for deploying Istio gateways
 keywords:

charts/kubezero-istio-ingress/charts/istio-ingress/templates/_affinity.tpl

@@ -1,11 +1,11 @@
 {{/* affinity - https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ */}}
 
-{{- define "nodeaffinity" }}
+{{ define "nodeaffinity" }}
-  nodeAffinity:
+nodeAffinity:
-    requiredDuringSchedulingIgnoredDuringExecution:
+  requiredDuringSchedulingIgnoredDuringExecution:
-    {{- include "nodeAffinityRequiredDuringScheduling" . }}
+  {{- include "nodeAffinityRequiredDuringScheduling" . }}
-    preferredDuringSchedulingIgnoredDuringExecution:
+  preferredDuringSchedulingIgnoredDuringExecution:
-    {{- include "nodeAffinityPreferredDuringScheduling" . }}
+  {{- include "nodeAffinityPreferredDuringScheduling" . }}
 {{- end }}
 
 {{- define "nodeAffinityRequiredDuringScheduling" }}

charts/kubezero-istio-ingress/charts/istio-ingress/templates/deployment.yaml

@@ -41,6 +41,7 @@ spec:
         istio.io/rev: {{ .Values.revision | default "default" }}
         install.operator.istio.io/owning-resource: {{ .Values.ownerName | default "unknown" }}
         operator.istio.io/component: "IngressGateways"
+        sidecar.istio.io/inject: "false"
       annotations:
         {{- if .Values.meshConfig.enablePrometheusMerge }}
         prometheus.io/port: "15020"
@@ -101,12 +102,6 @@ spec:
             - containerPort: {{ $val.targetPort | default $val.port }}
               protocol: {{ $val.protocol | default "TCP" }}
             {{- end }}
-            {{- if $.Values.global.meshExpansion.enabled }}
-            {{- range $key, $val := $gateway.meshExpansionPorts }}
-            - containerPort: {{ $val.targetPort | default $val.port }}
-              protocol: {{ $val.protocol | default "TCP" }}
-            {{- end }}
-            {{- end }}
             - containerPort: 15090
               protocol: TCP
               name: http-envoy-prom
@@ -220,6 +215,10 @@ spec:
           - name: TRUST_DOMAIN
             value: "{{ .Values.global.trustDomain | default (index .Values.meshConfig "trustDomain") }}"
           {{- end }}
+          {{- if not $gateway.runAsRoot }}
+          - name: ISTIO_META_UNPRIVILEGED_POD
+            value: "true"
+          {{- end }}
           {{- range $key, $val := $gateway.env }}
           - name: {{ $key }}
             value: {{ $val }}
@@ -228,10 +227,10 @@ spec:
           - name: {{ $key }}
             value: "{{ $value }}"
           {{- end }}
-          {{ $network_set := index $gateway.env "ISTIO_META_NETWORK" }}
+          {{- $network_set := index $gateway.env "ISTIO_META_NETWORK" }}
           {{- if and (not $network_set) .Values.global.network }}
           - name: ISTIO_META_NETWORK
-            value: {{ .Values.global.network }}
+            value: "{{ .Values.global.network }}"
           {{- end }}
 {{- if $gateway.podAnnotations }}
           - name: "ISTIO_METAJSON_ANNOTATIONS"
@@ -254,8 +253,6 @@ spec:
             mountPath: /var/run/secrets/tokens
             readOnly: true
 {{- end }}
-          - name: gatewaysdsudspath
-            mountPath: /var/run/ingress_gateway
           {{- if .Values.global.mountMtlsCerts }}
           # Use the key and cert mounted to /etc/certs/ for the in-cluster mTLS communications.
           - name: istio-certs
@@ -296,10 +293,18 @@ spec:
             - path: "annotations"
               fieldRef:
                 fieldPath: metadata.annotations
+            - path: "cpu-limit"
+              resourceFieldRef:
+                containerName: istio-proxy
+                resource: limits.cpu
+                divisor: 1m
+            - path: "cpu-request"
+              resourceFieldRef:
+                containerName: istio-proxy
+                resource: requests.cpu
+                divisor: 1m
       - name: istio-envoy
         emptyDir: {}
-      - name: gatewaysdsudspath
-        emptyDir: {}
       - name: istio-data
         emptyDir: {}
 {{- if eq .Values.global.jwtPolicy "third-party-jwt" }}
@@ -335,7 +340,7 @@ spec:
           optional: true
       {{- end }}
       affinity:
-      {{- include "nodeaffinity" (dict "global" .Values.global "nodeSelector" $gateway.nodeSelector) | indent 6 }}
+{{ include "nodeaffinity" (dict "global" .Values.global "nodeSelector" $gateway.nodeSelector) | trim | indent 8 }}
       {{- include "podAntiAffinity" $gateway | indent 6 }}
 {{- if $gateway.tolerations }}
       tolerations:

charts/kubezero-istio-ingress/charts/istio-ingress/templates/meshexpansion.yaml

@@ -1,79 +0,0 @@
-{{- if .Values.global.meshExpansion.enabled }}
-apiVersion: networking.istio.io/v1alpha3
-kind: Gateway
-metadata:
-  name: meshexpansion-gateway
-  namespace: {{ .Release.Namespace }}
-  labels:
-    release: {{ .Release.Name }}
-    istio.io/rev: {{ .Values.revision | default "default" }}
-    install.operator.istio.io/owning-resource: {{ .Values.ownerName | default "unknown" }}
-    operator.istio.io/component: "IngressGateways"
-spec:
-  selector:
-    istio: ingressgateway
-  servers:
-    - port:
-        number: 15012
-        protocol: TCP
-        name: tcp-istiod
-      hosts:
-        - "*"
-    - port:
-        number: 15017
-        protocol: TCP
-        name: tcp-istiodwebhook
-      hosts:
-        - "*"
----
-
-apiVersion: networking.istio.io/v1alpha3
-kind: VirtualService
-metadata:
-  name: meshexpansion-vs-istiod
-  namespace: {{ .Release.Namespace }}
-  labels:
-    release: {{ .Release.Name }}
-spec:
-  hosts:
-  - istiod.{{ .Values.global.istioNamespace }}.svc.{{ .Values.global.proxy.clusterDomain }}
-  gateways:
-  - meshexpansion-gateway
-  tcp:
-  - match:
-    - port: 15012
-    route:
-    - destination:
-        host: istiod.{{ .Values.global.istioNamespace }}.svc.{{ .Values.global.proxy.clusterDomain }}
-        port:
-          number: 15012
-  - match:
-    - port: 15017
-    route:
-    - destination:
-        host: istiod.{{ .Release.Namespace }}.svc.{{ .Values.global.proxy.clusterDomain }}
-        port:
-          number: 443
----
-
-apiVersion: networking.istio.io/v1alpha3
-kind: DestinationRule
-metadata:
-  name: meshexpansion-dr-istiod
-  namespace: {{ .Release.Namespace }}
-  labels:
-    release: {{ .Release.Name }}
-spec:
-  host: istiod.{{ .Release.Namespace }}.svc.{{ .Values.global.proxy.clusterDomain }}
-  trafficPolicy:
-    portLevelSettings:
-    - port:
-        number: 15012
-      tls:
-        mode: DISABLE
-    - port:
-        number: 15017
-      tls:
-        mode: DISABLE
-
-{{- end }}

charts/kubezero-istio-ingress/charts/istio-ingress/templates/service.yaml

@@ -38,14 +38,6 @@ spec:
       {{- end }}
     {{- end }}
 
-    {{- if $.Values.global.meshExpansion.enabled }}
-    {{- range $key, $val := $gateway.meshExpansionPorts }}
-    -
-      {{- range $pkey, $pval := $val }}
-      {{ $pkey}}: {{ $pval }}
-      {{- end }}
-    {{- end }}
-    {{- end }}
   {{ range $app := $gateway.ingressPorts }}
     -
       port: {{ $app.port }}

charts/kubezero-istio-ingress/charts/istio-ingress/values.yaml

@@ -24,7 +24,11 @@ gateways:
       targetPort: 8443
       name: https
       protocol: TCP
-      # This is the port where sni routing happens
+    - port: 15012
+      targetPort: 15012
+      name: tcp-istiod
+      protocol: TCP
+    # This is the port where sni routing happens
     - port: 15443
       targetPort: 15443
       name: tls
@@ -66,18 +70,6 @@ gateways:
     podAnnotations: {}
     type: LoadBalancer #change to NodePort, ClusterIP or LoadBalancer if need be
 
-    #### MESH EXPANSION PORTS  ########
-    # Pilot and Citadel MTLS ports are enabled in gateway - but will only redirect
-    # to pilot/citadel if global.meshExpansion settings are enabled.
-    # Delete these ports if mesh expansion is not enabled, to avoid
-    # exposing unnecessary ports on the web.
-    # You can remove these ports if you are not using mesh expansion
-    meshExpansionPorts:
-    - port: 15012
-      targetPort: 15012
-      name: tcp-istiod
-    ####### end MESH EXPANSION PORTS ######
-
     ##############
     secretVolumes:
     - name: ingressgateway-certs
@@ -99,7 +91,7 @@ gateways:
       # A gateway with this mode ensures that pilot generates an additional
       # set of clusters for internal services but without Istio mTLS, to
       # enable cross cluster routing.
-      ISTIO_META_ROUTER_MODE: "sni-dnat"
+      ISTIO_META_ROUTER_MODE: "standard"
 
     nodeSelector: {}
     tolerations: []
@@ -142,6 +134,12 @@ global:
   # . implies these objects are visible to only to sidecars in the same namespace, or if imported as a Sidecar.egress.host
   defaultConfigVisibilitySettings: []
 
+  # Default node selector to be applied to all deployments so that all pods can be
+  # constrained to run a particular nodes. Each component can overwrite these default
+  # values by adding its node selector block in the relevant section below and setting
+  # the desired values.
+  defaultNodeSelector: {}
+
   # enable pod disruption budget for the control plane, which is used to
   # ensure Istio control plane components are gradually upgraded or recovered.
   defaultPodDisruptionBudget:
@@ -170,10 +168,10 @@ global:
   # Default hub for Istio images.
   # Releases are published to docker hub under 'istio' project.
   # Dev builds from prow are on gcr.io
-  hub: gcr.io/istio-testing
+  hub: docker.io/istio
 
   # Default tag for Istio images.
-  tag: latest
+  tag: 1.9.1
 
   # Specify image pull policy if default behavior isn't desired.
   # Default behavior: latest images will be Always else IfNotPresent.
@@ -206,14 +204,6 @@ global:
   logging:
     level: "default:info"
 
-  # If set to true, the pilot and citadel mtls will be exposed on the
-  # ingress gateway
-  meshExpansion:
-    enabled: false
-    # If set to true, the pilot and citadel mtls and the plain text pilot ports
-    # will be exposed on an internal gateway
-    useILB: false
-
   # Kubernetes >=v1.11.0 will create two PriorityClass, including system-cluster-critical and
   # system-node-critical, it is better to configure this in order to make sure your Istio pods
   # will not be killed because of low priority class.
@@ -318,11 +308,10 @@ global:
     servicePort: 0
 
   # Deprecated, use meshConfig.trustDomain
-#  trustDomain: ""
+  trustDomain: ""
 
 meshConfig:
   enablePrometheusMerge: true
-#  trustDomain: ""
   defaultConfig:
     proxyMetadata: {}
     tracing:

charts/kubezero-istio-ingress/charts/istio-private-ingress/Chart.yaml

@@ -1,6 +1,6 @@
 apiVersion: v1
 name: istio-private-ingress
-version: 1.1.0
+version: 1.9.1
 tillerVersion: ">=2.7.2"
 description: Helm chart for deploying Istio gateways
 keywords:

charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/_affinity.tpl

@@ -1,11 +1,11 @@
 {{/* affinity - https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ */}}
 
-{{- define "nodeaffinity" }}
+{{ define "nodeaffinity" }}
-  nodeAffinity:
+nodeAffinity:
-    requiredDuringSchedulingIgnoredDuringExecution:
+  requiredDuringSchedulingIgnoredDuringExecution:
-    {{- include "nodeAffinityRequiredDuringScheduling" . }}
+  {{- include "nodeAffinityRequiredDuringScheduling" . }}
-    preferredDuringSchedulingIgnoredDuringExecution:
+  preferredDuringSchedulingIgnoredDuringExecution:
-    {{- include "nodeAffinityPreferredDuringScheduling" . }}
+  {{- include "nodeAffinityPreferredDuringScheduling" . }}
 {{- end }}
 
 {{- define "nodeAffinityRequiredDuringScheduling" }}

charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/deployment.yaml

@@ -41,6 +41,7 @@ spec:
         istio.io/rev: {{ .Values.revision | default "default" }}
         install.operator.istio.io/owning-resource: {{ .Values.ownerName | default "unknown" }}
         operator.istio.io/component: "IngressGateways"
+        sidecar.istio.io/inject: "false"
       annotations:
         {{- if .Values.meshConfig.enablePrometheusMerge }}
         prometheus.io/port: "15020"
@@ -101,12 +102,6 @@ spec:
             - containerPort: {{ $val.targetPort | default $val.port }}
               protocol: {{ $val.protocol | default "TCP" }}
             {{- end }}
-            {{- if $.Values.global.meshExpansion.enabled }}
-            {{- range $key, $val := $gateway.meshExpansionPorts }}
-            - containerPort: {{ $val.targetPort | default $val.port }}
-              protocol: {{ $val.protocol | default "TCP" }}
-            {{- end }}
-            {{- end }}
             - containerPort: 15090
               protocol: TCP
               name: http-envoy-prom
@@ -220,6 +215,10 @@ spec:
           - name: TRUST_DOMAIN
             value: "{{ .Values.global.trustDomain | default (index .Values.meshConfig "trustDomain") }}"
           {{- end }}
+          {{- if not $gateway.runAsRoot }}
+          - name: ISTIO_META_UNPRIVILEGED_POD
+            value: "true"
+          {{- end }}
           {{- range $key, $val := $gateway.env }}
           - name: {{ $key }}
             value: {{ $val }}
@@ -228,10 +227,10 @@ spec:
           - name: {{ $key }}
             value: "{{ $value }}"
           {{- end }}
-          {{ $network_set := index $gateway.env "ISTIO_META_NETWORK" }}
+          {{- $network_set := index $gateway.env "ISTIO_META_NETWORK" }}
           {{- if and (not $network_set) .Values.global.network }}
           - name: ISTIO_META_NETWORK
-            value: {{ .Values.global.network }}
+            value: "{{ .Values.global.network }}"
           {{- end }}
 {{- if $gateway.podAnnotations }}
           - name: "ISTIO_METAJSON_ANNOTATIONS"
@@ -254,8 +253,6 @@ spec:
             mountPath: /var/run/secrets/tokens
             readOnly: true
 {{- end }}
-          - name: gatewaysdsudspath
-            mountPath: /var/run/ingress_gateway
           {{- if .Values.global.mountMtlsCerts }}
           # Use the key and cert mounted to /etc/certs/ for the in-cluster mTLS communications.
           - name: istio-certs
@@ -296,10 +293,18 @@ spec:
             - path: "annotations"
               fieldRef:
                 fieldPath: metadata.annotations
+            - path: "cpu-limit"
+              resourceFieldRef:
+                containerName: istio-proxy
+                resource: limits.cpu
+                divisor: 1m
+            - path: "cpu-request"
+              resourceFieldRef:
+                containerName: istio-proxy
+                resource: requests.cpu
+                divisor: 1m
       - name: istio-envoy
         emptyDir: {}
-      - name: gatewaysdsudspath
-        emptyDir: {}
       - name: istio-data
         emptyDir: {}
 {{- if eq .Values.global.jwtPolicy "third-party-jwt" }}
@@ -335,7 +340,7 @@ spec:
           optional: true
       {{- end }}
       affinity:
-      {{- include "nodeaffinity" (dict "global" .Values.global "nodeSelector" $gateway.nodeSelector) | indent 6 }}
+{{ include "nodeaffinity" (dict "global" .Values.global "nodeSelector" $gateway.nodeSelector) | trim | indent 8 }}
       {{- include "podAntiAffinity" $gateway | indent 6 }}
 {{- if $gateway.tolerations }}
       tolerations:

charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/meshexpansion.yaml

@@ -1,79 +0,0 @@
-{{- if .Values.global.meshExpansion.enabled }}
-apiVersion: networking.istio.io/v1alpha3
-kind: Gateway
-metadata:
-  name: meshexpansion-gateway
-  namespace: {{ .Release.Namespace }}
-  labels:
-    release: {{ .Release.Name }}
-    istio.io/rev: {{ .Values.revision | default "default" }}
-    install.operator.istio.io/owning-resource: {{ .Values.ownerName | default "unknown" }}
-    operator.istio.io/component: "IngressGateways"
-spec:
-  selector:
-    istio: ingressgateway
-  servers:
-    - port:
-        number: 15012
-        protocol: TCP
-        name: tcp-istiod
-      hosts:
-        - "*"
-    - port:
-        number: 15017
-        protocol: TCP
-        name: tcp-istiodwebhook
-      hosts:
-        - "*"
----
-
-apiVersion: networking.istio.io/v1alpha3
-kind: VirtualService
-metadata:
-  name: meshexpansion-vs-istiod
-  namespace: {{ .Release.Namespace }}
-  labels:
-    release: {{ .Release.Name }}
-spec:
-  hosts:
-  - istiod.{{ .Values.global.istioNamespace }}.svc.{{ .Values.global.proxy.clusterDomain }}
-  gateways:
-  - meshexpansion-gateway
-  tcp:
-  - match:
-    - port: 15012
-    route:
-    - destination:
-        host: istiod.{{ .Values.global.istioNamespace }}.svc.{{ .Values.global.proxy.clusterDomain }}
-        port:
-          number: 15012
-  - match:
-    - port: 15017
-    route:
-    - destination:
-        host: istiod.{{ .Release.Namespace }}.svc.{{ .Values.global.proxy.clusterDomain }}
-        port:
-          number: 443
----
-
-apiVersion: networking.istio.io/v1alpha3
-kind: DestinationRule
-metadata:
-  name: meshexpansion-dr-istiod
-  namespace: {{ .Release.Namespace }}
-  labels:
-    release: {{ .Release.Name }}
-spec:
-  host: istiod.{{ .Release.Namespace }}.svc.{{ .Values.global.proxy.clusterDomain }}
-  trafficPolicy:
-    portLevelSettings:
-    - port:
-        number: 15012
-      tls:
-        mode: DISABLE
-    - port:
-        number: 15017
-      tls:
-        mode: DISABLE
-
-{{- end }}

charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/service.yaml

@@ -38,14 +38,6 @@ spec:
       {{- end }}
     {{- end }}
 
-    {{- if $.Values.global.meshExpansion.enabled }}
-    {{- range $key, $val := $gateway.meshExpansionPorts }}
-    -
-      {{- range $pkey, $pval := $val }}
-      {{ $pkey}}: {{ $pval }}
-      {{- end }}
-    {{- end }}
-    {{- end }}
   {{ range $app := $gateway.ingressPorts }}
     -
       port: {{ $app.port }}

charts/kubezero-istio-ingress/charts/istio-private-ingress/values.yaml

@@ -24,7 +24,11 @@ gateways:
       targetPort: 8443
       name: https
       protocol: TCP
-      # This is the port where sni routing happens
+    - port: 15012
+      targetPort: 15012
+      name: tcp-istiod
+      protocol: TCP
+    # This is the port where sni routing happens
     - port: 15443
       targetPort: 15443
       name: tls
@@ -66,18 +70,6 @@ gateways:
     podAnnotations: {}
     type: LoadBalancer #change to NodePort, ClusterIP or LoadBalancer if need be
 
-    #### MESH EXPANSION PORTS  ########
-    # Pilot and Citadel MTLS ports are enabled in gateway - but will only redirect
-    # to pilot/citadel if global.meshExpansion settings are enabled.
-    # Delete these ports if mesh expansion is not enabled, to avoid
-    # exposing unnecessary ports on the web.
-    # You can remove these ports if you are not using mesh expansion
-    meshExpansionPorts:
-    - port: 15012
-      targetPort: 15012
-      name: tcp-istiod
-    ####### end MESH EXPANSION PORTS ######
-
     ##############
     secretVolumes:
     - name: ingressgateway-certs
@@ -99,7 +91,7 @@ gateways:
       # A gateway with this mode ensures that pilot generates an additional
       # set of clusters for internal services but without Istio mTLS, to
       # enable cross cluster routing.
-      ISTIO_META_ROUTER_MODE: "sni-dnat"
+      ISTIO_META_ROUTER_MODE: "standard"
 
     nodeSelector: {}
     tolerations: []
@@ -142,6 +134,12 @@ global:
   # . implies these objects are visible to only to sidecars in the same namespace, or if imported as a Sidecar.egress.host
   defaultConfigVisibilitySettings: []
 
+  # Default node selector to be applied to all deployments so that all pods can be
+  # constrained to run a particular nodes. Each component can overwrite these default
+  # values by adding its node selector block in the relevant section below and setting
+  # the desired values.
+  defaultNodeSelector: {}
+
   # enable pod disruption budget for the control plane, which is used to
   # ensure Istio control plane components are gradually upgraded or recovered.
   defaultPodDisruptionBudget:
@@ -170,10 +168,10 @@ global:
   # Default hub for Istio images.
   # Releases are published to docker hub under 'istio' project.
   # Dev builds from prow are on gcr.io
-  hub: gcr.io/istio-testing
+  hub: docker.io/istio
 
   # Default tag for Istio images.
-  tag: latest
+  tag: 1.9.1
 
   # Specify image pull policy if default behavior isn't desired.
   # Default behavior: latest images will be Always else IfNotPresent.
@@ -206,14 +204,6 @@ global:
   logging:
     level: "default:info"
 
-  # If set to true, the pilot and citadel mtls will be exposed on the
-  # ingress gateway
-  meshExpansion:
-    enabled: false
-    # If set to true, the pilot and citadel mtls and the plain text pilot ports
-    # will be exposed on an internal gateway
-    useILB: false
-
   # Kubernetes >=v1.11.0 will create two PriorityClass, including system-cluster-critical and
   # system-node-critical, it is better to configure this in order to make sure your Istio pods
   # will not be killed because of low priority class.
@@ -318,11 +308,10 @@ global:
     servicePort: 0
 
   # Deprecated, use meshConfig.trustDomain
-#  trustDomain: ""
+  trustDomain: ""
 
 meshConfig:
   enablePrometheusMerge: true
-#  trustDomain: ""
   defaultConfig:
     proxyMetadata: {}
     tracing:

charts/kubezero-istio-ingress/templates/ingress-gateway.yaml

@@ -1,3 +1,5 @@
+# https://istio.io/latest/docs/tasks/traffic-management/ingress/secure-ingress/#configure-a-tls-ingress-gateway-for-multiple-hosts
+
 {{- if and (index .Values "istio-ingress" "enabled") (index .Values "istio-ingress" "dnsNames") }}
 apiVersion: networking.istio.io/v1beta1
 kind: Gateway

charts/kubezero-istio-ingress/values.yaml

@@ -1,7 +1,7 @@
 # Make sure these values match kuberzero-istio !!!
 global:
-  hub: docker.io/istio
+  #hub: docker.io/istio
-  tag: 1.8.1
+  #tag: 1.9.0
 
   logAsJson: true
   jwtPolicy: first-party-jwt
@@ -42,17 +42,32 @@ istio-ingress:
       nodeSelector:
         node.kubernetes.io/ingress.public: "30080_30443"
       ports:
-      - name: http-status
+      - name: status-port
         port: 15021
         nodePort: 30021
+        protocol: TCP
       - name: http2
         port: 80
         targetPort: 8080
         nodePort: 30080
+        protocol: TCP
       - name: https
         port: 443
         targetPort: 8443
         nodePort: 30443
+        protocol: TCP
+     ## multi-cluster - disabled on public LBs
+     #- name: tcp-istiod
+     #  port: 15012
+     #  targetPort: 15012
+     #  nodePort: 30012
+     #  protocol: TCP
+     ## multi-cluster sni east-west
+     #- name: tls
+     #  port: 15443
+     #  targetPort: 15443
+     #  nodePort: 30044
+     #  protocol: TCP
 
   dnsNames: []
 # - '*.example.com'
@@ -93,17 +108,32 @@ istio-private-ingress:
         #nodeSelector: "31080_31443_31671_31672_31224"
 
       ports:
-      - name: http-status
+      - name: status-port
         port: 15021
         nodePort: 31021
+        protocol: TCP
       - name: http2
         port: 80
         targetPort: 8080
         nodePort: 31080
+        protocol: TCP
       - name: https
         port: 443
         targetPort: 8443
         nodePort: 31443
+        protocol: TCP
+      # multi-cluster
+      - name: tcp-istiod
+        port: 15012
+        targetPort: 15012
+        nodePort: 31012
+        protocol: TCP
+      # multi-cluster sni east-west
+      - name: tls
+        port: 15443
+        targetPort: 15443
+        nodePort: 31044
+        protocol: TCP
       #- name: fluentd-forward
       #  port: 24224
       #  nodePort: 31224

charts/kubezero-istio/Chart.yaml

@@ -2,8 +2,8 @@ apiVersion: v2
 name: kubezero-istio
 description: KubeZero Umbrella Chart for Istio
 type: application
-version: 0.4.1
+version: 0.5.0
-appVersion: 1.8.1
+appVersion: 1.9.1
 home: https://kubezero.com
 icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png
 keywords:
@@ -16,7 +16,7 @@ dependencies:
     version: ">= 0.1.3"
     repository: https://zero-down-time.github.io/kubezero/
   - name: base
-    version: 1.1.0
+    version: 1.9.1
   - name: istio-discovery
-    version: 1.2.0
+    version: 1.9.1
-kubeVersion: ">= 1.16.0"
+kubeVersion: ">= 1.18.0"

charts/kubezero-istio/README.md

@@ -1,6 +1,6 @@
 # kubezero-istio
 
-![Version: 0.4.1](https://img.shields.io/badge/Version-0.4.1-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.8.1](https://img.shields.io/badge/AppVersion-1.8.1-informational?style=flat-square)
+![Version: 0.5.0](https://img.shields.io/badge/Version-0.5.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.9.1](https://img.shields.io/badge/AppVersion-1.9.1-informational?style=flat-square)
 
 KubeZero Umbrella Chart for Istio
 
@@ -16,12 +16,12 @@ Installs the Istio control plane
 
 ## Requirements
 
-Kubernetes: `>= 1.16.0`
+Kubernetes: `>= 1.18.0`
 
 | Repository | Name | Version |
 |------------|------|---------|
-|  | base | 1.1.0 |
+|  | base | 1.9.1 |
-|  | istio-discovery | 1.2.0 |
+|  | istio-discovery | 1.9.1 |
 | https://zero-down-time.github.io/kubezero/ | kubezero-lib | >= 0.1.3 |
 
 ## Values
@@ -29,11 +29,9 @@ Kubernetes: `>= 1.16.0`
 | Key | Type | Default | Description |
 |-----|------|---------|-------------|
 | global.defaultPodDisruptionBudget.enabled | bool | `false` |  |
-| global.hub | string | `"docker.io/istio"` |  |
 | global.jwtPolicy | string | `"first-party-jwt"` |  |
 | global.logAsJson | bool | `true` |  |
 | global.priorityClassName | string | `"system-cluster-critical"` |  |
-| global.tag | string | `"1.8.0"` |  |
 | istio-discovery.meshConfig.accessLogEncoding | string | `"JSON"` |  |
 | istio-discovery.meshConfig.accessLogFile | string | `"/dev/stdout"` |  |
 | istio-discovery.meshConfig.h2UpgradePolicy | string | `"DO_NOT_UPGRADE"` |  |

charts/kubezero-istio/charts/base/Chart.yaml

@@ -1,6 +1,6 @@
 apiVersion: v1
 name: base
-version: 1.1.0
+version: 1.9.1
 tillerVersion: ">=2.7.2"
 description: Helm chart for deploying Istio cluster resources and CRDs
 keywords: