diff --git a/.gitignore b/.gitignore index 04f73a7..3bb249e 100644 --- a/.gitignore +++ b/.gitignore @@ -1,7 +1,6 @@ # Vim *.swp -output .vscode .DS_Store .idea diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..b0168fd --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,46 @@ +# Changelog + +## KubeZero - 2.18 ( Argoless ) + +### High level / Admin changes +- ArgoCD is now optional and NOT required nor used during initial cluster bootstrap +- the bootstrap process now uses the same config and templates as the optional ArgoCD applications later on +- the bootstrap is can now be restarted at any time and considerably faster +- the top level KubeZero config for the ArgoCD app-of-apps is now also maintained via the gitops workflow. Changes can be applied by a simple git push rather than manual scripts + +### Calico +- version bump + +### Cert-manager +- local issuers are now cluster issuer to allow them being used across namespaces +- all cert-manager resources moved into the cert-manager namespace +- version bump to 1.10 + +### Kiam +- set priorty class to cluster essential +- certificates are now issued by the cluster issuer + +### EBS / EFS +- version bump + +### Istio +- istio operator removed, deployment migrated to helm, various cleanups +- version bump to 1.8 +- all ingress resources are now in the dedicated new namespace istio-ingress ( deployed via separate kubezero chart istio-ingress) +- set priorty class of ingress components to cluster essential + +### Logging +- ES/Kibana version bump to 7.10 +- ECK operator is now installed on demand in logging ns +- Custom event fields configurable via new fluent-bit chart + e.g. clustername could be added to each event allowing easy filtering in case multiple clusters stream events into a single central ES cluster + +### ArgoCD +- version bump, new app of app architecure + +### Metrics +- version bump +- all servicemonitor resources are now in the same namespaces as the respective apps to avoid deployments across multiple namespaces + +### upstream Kubernetes 1.18 +https://sysdig.com/blog/whats-new-kubernetes-1-18/ diff --git a/CHANGES.md b/CHANGES.md deleted file mode 100644 index 02ab072..0000000 --- a/CHANGES.md +++ /dev/null @@ -1,15 +0,0 @@ -# CFN / Platform -- Kube to 1.17 -- Kube-proxy uses ipvs -- metrics support for kube-proxy -- no reliance on custom resource for S3 buckets anymore - - -# Kubezero -- fully automated one command bootstrap incl. all kubezero components -- migrated from kube-prometheuss to prometheus-operator helm charts for metrics -- latest Grafana incl. peristence -- kube-prometheus adapter improvements / customizations -- integrated EFS CSI driver into Kubezero -- prometheus itself can be exposed via istio ingress on demand to ease development of custom metrics -- backup script to export all cert-manager items between clusters diff --git a/Makefile b/Makefile index bc4204f..ceadf7e 100644 --- a/Makefile +++ b/Makefile @@ -2,16 +2,12 @@ BUCKET ?= zero-downtime BUCKET_PREFIX ?= /cloudbender/distfiles FILES ?= distfiles.txt -.PHONY: clean update docs +.PHONY: update docs all: update - -clean: - rm -f kube*.tgz - update: - ./script/update_helm.sh + ./scripts/update_helm.sh docs: for c in charts/*; do \ @@ -19,3 +15,6 @@ docs: [[ $$c =~ "kubeadm" ]] && continue ; \ helm-docs -c $$c ; \ done + +publish: + ./scripts/publish.sh diff --git a/README.md b/README.md index ec6247b..8f48dbf 100644 --- a/README.md +++ b/README.md @@ -1,43 +1,47 @@ KubeZero - Zero Down Time Kubernetes platform ======================== -KubeZero is a pre-configured collection of components deployed onto a bare Kubernetes cluster. -All chosen components are 100% organic OpenSource. +KubeZero is a Kubernetes distribution providing an integrated container platform so you can focus on your applications. -# Design goals +# Design philosophy -- Cloud provider agnostic, bare-metal / self-hosted possible -- No vendor lock in -- No closed source solutions +- Cloud provider agnostic, bare-metal/self-hosted +- No vendor lock in, most components are optional and could be exchanged +- Organic Open Source / open and permissive licenses over closed-source solutions - No premium services / subscriptions required -- Staying to upstream projects as close as possible -- Minimal custom code -- Work within each community / give back +- Staying and contributing back to upstream projects as much as possible + # Version / Support Matrix -| KubeZero \ Kubernetes Version | v1.17 | v1.18 | v1.19 | v1.20 | EOL | -|----------------------------------------|-------|-------|-------|-------|-------------| -| master branch | no | yes | beta | no | | -| stable branch | no | yes | no | no | | -| v2.18.0 | no | yes | no | no | 30 Apr 2021 | -| v1 | yes | no | no | no | 30 Jan 2021 | +| KubeZero \ Kubernetes Version | v1.18 | v1.19 | v1.20 | EOL | +|----------------------------------------|-------|-------|-------|-------------| +| master branch | yes | yes | beta | | +| stable branch | yes | yes | no | | +| v2.19.0 | yes | yes | no | 30 Jun 2021 | +| v2.18.0 | yes | no | no | 30 Apr 2021 | + +# Architecure +![aws_architecture](docs/aws_architecture.png) -## General -- Container runtime cri-o rather than Docker for improved security and performance +# Components +## Container runtime +- cri-o rather than Docker for improved security and performance ## Control plane - support for single node control plane for small clusters / test environments to reduce costs - access to control plane from within the VPC only by default ( VPN access required for Admin tasks ) - controller nodes are used for various platform admin controllers / operators to reduce costs and noise on worker nodes -- integrated ArgoCD Gitops controller + +## GitOps +- full ArgoCD support and integration (optional) ## AWS IAM access control - Kiam allowing IAM roles per pod - IAM roles are assumed / requested and cached on controller nodes for improved security -- blocking access to meta-data service on all nodes -- IAM roles are maintained/ automated and tracked via CFN templates +- access to meta-data services is blocked / controlled on all nodes +- core IAM roles are maintained via CFN templates ## Network - Calico using VxLAN incl. increased MTU @@ -66,7 +70,7 @@ All chosen components are 100% organic OpenSource. ## Logging - all container logs are enhanced with Kubernetes metadata to provide context for each message -- flexible ElasticSearch setup via ECK operator to ease maintenance and reduce required admin knowledge, incl automated backups to S3 -- Kibana allowing easy search and dashboards for all logs, incl. pre configured index templates and index management to reduce costs -- fluentd central log ingress service allowing additional parsing and queuing to improved reliability +- flexible ElasticSearch setup, leveraging the ECK operator, for easy maintenance & minimal admin knowledge required, incl. automated backups to S3 +- Kibana allowing easy search and dashboards for all logs, incl. pre configured index templates and index management +- central fluentd service providing queuing during highload as well as additional parsing options - lightweight fluent-bit agents on each node requiring minimal resources forwarding logs secure via SSL to fluentd diff --git a/charts/kubeadm/Chart.yaml b/charts/kubeadm/Chart.yaml index 99424f1..ba8a347 100644 --- a/charts/kubeadm/Chart.yaml +++ b/charts/kubeadm/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: kubeadm description: KubeZero Kubeadm golden config type: application -version: 1.18.14 +version: 1.19.8 home: https://kubezero.com icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png keywords: @@ -10,4 +10,4 @@ keywords: - kubeadm maintainers: - name: Quarky9 -kubeVersion: ">= 1.16.0" +kubeVersion: ">= 1.18.0" diff --git a/charts/kubeadm/README.md.gotmpl b/charts/kubeadm/README.md.gotmpl index d29076a..fce4c94 100644 --- a/charts/kubeadm/README.md.gotmpl +++ b/charts/kubeadm/README.md.gotmpl @@ -17,6 +17,14 @@ Installs the Istio control plane {{ template "chart.valuesSection" . }} +## Changes for 1.19 + +### Logging to json of control plane components +- https://github.com/kubernetes/website/blob/dev-1.19/content/en/docs/concepts/cluster-administration/system-logs.md + +### PodTopologySpread +- https://kubernetes.io/blog/2020/05/introducing-podtopologyspread/#podtopologyspread-defaults + ## Resources - https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/ diff --git a/charts/kubeadm/templates/ClusterConfiguration.yaml b/charts/kubeadm/templates/ClusterConfiguration.yaml index f7c4238..0bfa0b8 100644 --- a/charts/kubeadm/templates/ClusterConfiguration.yaml +++ b/charts/kubeadm/templates/ClusterConfiguration.yaml @@ -10,30 +10,31 @@ networking: etcd: local: extraArgs: - listen-metrics-urls: "http://0.0.0.0:2381" + listen-metrics-urls: "http://{{ .Values.listenAddress }}:2381" + logger: "zap" + # log-level: "warn" {{- with .Values.etcdExtraArgs }} {{- toYaml . | nindent 6 }} {{- end }} controllerManager: extraArgs: profiling: "false" - bind-address: 0.0.0.0 + bind-address: {{ .Values.listenAddress }} terminated-pod-gc-threshold: "300" leader-elect: {{ .Values.clusterHighAvailable | quote }} - # Default anyways but make kube-bench happy - feature-gates: "RotateKubeletServerCertificate=true" + logging-format: json scheduler: extraArgs: profiling: "false" - bind-address: 0.0.0.0 + bind-address: {{ .Values.listenAddress }} leader-elect: {{ .Values.clusterHighAvailable | quote }} + logging-format: json apiServer: certSANs: - {{ regexSplit ":" .Values.apiEndpoint -1 | first }} extraArgs: etcd-servers: {{ .Values.allEtcdEndpoints }} profiling: "false" - feature-gates: "CSIMigration=true,CSIMigrationAWS=true,CSIMigrationAWSComplete=true" audit-log-path: "/var/log/kubernetes/audit.log" audit-policy-file: /etc/kubernetes/apiserver/audit-policy.yaml audit-log-maxage: "7" @@ -41,11 +42,17 @@ apiServer: audit-log-maxbackup: "3" tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384" admission-control-config-file: /etc/kubernetes/apiserver/admission-configuration.yaml + {{- if eq .Values.platform "aws" }} authentication-token-webhook-config-file: /etc/kubernetes/apiserver/aws-iam-authenticator.yaml + feature-gates: "CSIMigrationAWS=true,CSIMigrationAWSComplete=true,DefaultPodTopologySpread=true" + {{- else }} + feature-gates: "DefaultPodTopologySpread=true" + {{- end }} enable-admission-plugins: NodeRestriction,EventRateLimit {{- if .Values.clusterHighAvailable }} - goaway-chance: ".001" + # goaway-chance: ".001" {{- end }} + logging-format: json {{- with .Values.apiExtraArgs }} {{- toYaml . | nindent 4 }} {{- end }} diff --git a/charts/kubeadm/templates/JoinConfiguration.yaml b/charts/kubeadm/templates/JoinConfiguration.yaml index 6ad03cc..6d3d44a 100644 --- a/charts/kubeadm/templates/JoinConfiguration.yaml +++ b/charts/kubeadm/templates/JoinConfiguration.yaml @@ -1,18 +1,16 @@ apiVersion: kubeadm.k8s.io/v1beta2 kind: JoinConfiguration -metadata: - name: kubezero-joinconfiguration discovery: - bootstrapToken: - apiServerEndpoint: {{ .Values.apiEndpoint }} - token: {{ .Values.joinToken }} - caCertHashes: - - "{{ .Values.caCertHash }}" + file: + kubeConfigPath: /root/.kube/config +controlPlane: + localAPIEndpoint: + advertiseAddress: IP_ADDRESS + bindPort: {{ regexSplit ":" .Values.apiEndpoint -1 | last }} nodeRegistration: ignorePreflightErrors: - DirAvailable--var-lib-etcd + - FileAvailable--etc-kubernetes-pki-ca.crt - Swap -controlPlane: - localAPIEndpoint: - advertiseAddress: {{ .Values.ipAddress }} - bindPort: {{ regexSplit ":" .Values.apiEndpoint -1 | last }} + kubeletExtraArgs: + node-labels: {{ .Values.nodeLabels | quote }} diff --git a/charts/kubeadm/templates/KubeProxyConfiguration.yaml b/charts/kubeadm/templates/KubeProxyConfiguration.yaml index 6ca5945..8d30d72 100644 --- a/charts/kubeadm/templates/KubeProxyConfiguration.yaml +++ b/charts/kubeadm/templates/KubeProxyConfiguration.yaml @@ -2,5 +2,5 @@ apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration metadata: name: kubezero-kubeproxyconfiguration -metricsBindAddress: "0.0.0.0:10249" +metricsBindAddress: "{{ .Values.listenAddress }}:10249" mode: "" diff --git a/charts/kubeadm/templates/KubeletConfiguration.yaml b/charts/kubeadm/templates/KubeletConfiguration.yaml index 3568211..04617ad 100644 --- a/charts/kubeadm/templates/KubeletConfiguration.yaml +++ b/charts/kubeadm/templates/KubeletConfiguration.yaml @@ -3,17 +3,35 @@ kind: KubeletConfiguration metadata: name: kubezero-kubeletconfiguration failSwapOn: false +cgroupDriver: cgroupfs +logging: + # We already have syslog-ng logging json + # format: json hairpinMode: hairpin-veth +{{- if .Values.systemd }} resolvConf: /run/systemd/resolve/resolv.conf -protectKernelDefaults: true +{{- end }} +protectKernelDefaults: {{ .Values.protectKernelDefaults }} eventRecordQPS: 0 # Breaks kubelet at boot time # tlsCertFile: /var/lib/kubelet/pki/kubelet.crt # tlsPrivateKeyFile: /var/lib/kubelet/pki/kubelet.key tlsCipherSuites: [TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256] featureGates: - # Default anyways but make kube-bench happy - RotateKubeletServerCertificate: true - CSIMigration: true + CustomCPUCFSQuotaPeriod: true +{{- if eq .Values.platform "aws" }} CSIMigrationAWS: true CSIMigrationAWSComplete: true +{{- end }} +# Minimal unit is 50m per pod +podsPerCore: 20 +# Basic OS on Ubuntu 20.04 incl. crio +systemReserved: + memory: 256Mi +# This should be dynamic based on number of maxpods and available cores +# https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#memory_cpu +# Below are default minimal for 2 cores and minimum kubelet +kubeReserved: + cpu: 70m + memory: 128Mi +# cpuCFSQuotaPeriod: 10ms diff --git a/charts/kubeadm/templates/admin-kubectl.yaml b/charts/kubeadm/templates/admin-kubectl.yaml new file mode 100644 index 0000000..bdbf724 --- /dev/null +++ b/charts/kubeadm/templates/admin-kubectl.yaml @@ -0,0 +1,27 @@ +{{- if eq .Values.platform "aws" }} +apiVersion: v1 +kind: Config +clusters: +- cluster: + server: https://{{ .Values.apiEndpoint }} + name: {{ .Values.clusterName }} +contexts: +- context: + cluster: {{ .Values.clusterName }} + user: kubernetes-admin + name: kubernetes-admin@{{ .Values.clusterName }} +current-context: kubernetes-admin@{{ .Values.clusterName }} +preferences: {} +users: +- name: kubernetes-admin + user: + exec: + apiVersion: client.authentication.k8s.io/v1alpha1 + command: aws-iam-authenticator + args: + - "token" + - "-i" + - "{{ .Values.clusterName }}" + - "-r" + - "{{ .Values.kubeAdminRole }}" +{{- end }} diff --git a/charts/kubeadm/templates/aws-iam-authenticator/crds.yaml b/charts/kubeadm/templates/aws-iam-authenticator/crds.yaml index 7ff85f7..c1977a8 100644 --- a/charts/kubeadm/templates/aws-iam-authenticator/crds.yaml +++ b/charts/kubeadm/templates/aws-iam-authenticator/crds.yaml @@ -1,3 +1,4 @@ +{{- if eq .Values.platform "aws" }} apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: @@ -30,3 +31,4 @@ spec: type: array items: type: string +{{- end }} diff --git a/charts/kubeadm/templates/aws-iam-authenticator/deployment.yaml b/charts/kubeadm/templates/aws-iam-authenticator/deployment.yaml index 6d6d235..7224642 100644 --- a/charts/kubeadm/templates/aws-iam-authenticator/deployment.yaml +++ b/charts/kubeadm/templates/aws-iam-authenticator/deployment.yaml @@ -1,3 +1,4 @@ +{{- if eq .Values.platform "aws" }} kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: @@ -51,7 +52,7 @@ metadata: --- kind: ClusterRoleBinding -apiVersion: rbac.authorization.k8s.io/v1beta1 +apiVersion: rbac.authorization.k8s.io/v1 metadata: name: aws-iam-authenticator namespace: kube-system @@ -116,7 +117,7 @@ spec: containers: - name: aws-iam-authenticator - image: public.ecr.aws/x8h8t2o1/aws-iam-authenticator:v0.5.2 + image: public.ecr.aws/zero-downtime/aws-iam-authenticator:v0.5.2 args: - server - --backend-mode=CRD,MountedFile @@ -136,7 +137,7 @@ spec: cpu: 10m limits: memory: 20Mi - cpu: 100m + #cpu: 100m volumeMounts: - name: config @@ -151,14 +152,4 @@ spec: - name: state hostPath: path: /var/aws-iam-authenticator/ ---- -apiVersion: iamauthenticator.k8s.aws/v1alpha1 -kind: IAMIdentityMapping -metadata: - name: kubernetes-admin -spec: - # Arn of the User or Role to be allowed to authenticate - arn: {{ .Values.kubeAdminRole }} - username: kubernetes-admin - groups: - - system:masters +{{- end }} diff --git a/charts/kubeadm/templates/aws-iam-authenticator/mappings.yaml b/charts/kubeadm/templates/aws-iam-authenticator/mappings.yaml new file mode 100644 index 0000000..70ba334 --- /dev/null +++ b/charts/kubeadm/templates/aws-iam-authenticator/mappings.yaml @@ -0,0 +1,26 @@ +{{- if eq .Values.platform "aws" }} +# Controller role for consistency, similar to kubeadm admin.conf +apiVersion: iamauthenticator.k8s.aws/v1alpha1 +kind: IAMIdentityMapping +metadata: + name: kubezero-worker-nodes +spec: + arn: {{ .Values.WorkerNodeRole }} + username: system:node:{{ "{{" }}EC2PrivateDNSName{{ "}}" }} + groups: + # For now use masters, define properly with 1.20 + - system:masters + - system:nodes + - system:bootstrappers +--- +# Admin Role for remote access +apiVersion: iamauthenticator.k8s.aws/v1alpha1 +kind: IAMIdentityMapping +metadata: + name: kubernetes-admin +spec: + arn: {{ .Values.kubeAdminRole }} + username: kubernetes-admin + groups: + - system:masters +{{- end }} diff --git a/charts/kubeadm/templates/device-plugins/fuse-device-plugin.yaml b/charts/kubeadm/templates/device-plugins/fuse-device-plugin.yaml new file mode 100644 index 0000000..c5a40e8 --- /dev/null +++ b/charts/kubeadm/templates/device-plugins/fuse-device-plugin.yaml @@ -0,0 +1,30 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: fuse-device-plugin + namespace: kube-system +spec: + selector: + matchLabels: + name: fuse-device-plugin + template: + metadata: + labels: + name: fuse-device-plugin + spec: + hostNetwork: true + containers: + - image: public.ecr.aws/zero-downtime/fuse-device-plugin:v1.0 + # imagePullPolicy: Always + name: fuse-device-plugin + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + volumeMounts: + - name: device-plugin + mountPath: /var/lib/kubelet/device-plugins + volumes: + - name: device-plugin + hostPath: + path: /var/lib/kubelet/device-plugins diff --git a/charts/kubeadm/templates/patches/coredns0.yaml b/charts/kubeadm/templates/patches/coredns0.yaml new file mode 100644 index 0000000..ef42b4e --- /dev/null +++ b/charts/kubeadm/templates/patches/coredns0.yaml @@ -0,0 +1,24 @@ +spec: + replicas: {{ ternary 3 1 .Values.clusterHighAvailable }} + template: + spec: + containers: + - name: coredns + resources: + requests: + cpu: 100m + memory: 32Mi + limits: + memory: 128Mi + nodeSelector: + node-role.kubernetes.io/master: "" + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: k8s-app + operator: In + values: + - kube-dns + topologyKey: "kubernetes.io/hostname" diff --git a/charts/kubeadm/templates/patches/etcd0.yaml b/charts/kubeadm/templates/patches/etcd0.yaml new file mode 100644 index 0000000..b17a0e5 --- /dev/null +++ b/charts/kubeadm/templates/patches/etcd0.yaml @@ -0,0 +1,7 @@ +spec: + containers: + - name: etcd + resources: + requests: + cpu: 250m + memory: 192Mi diff --git a/charts/kubeadm/templates/patches/kube-apiserver0.yaml b/charts/kubeadm/templates/patches/kube-apiserver0.yaml new file mode 100644 index 0000000..05c2a18 --- /dev/null +++ b/charts/kubeadm/templates/patches/kube-apiserver0.yaml @@ -0,0 +1,7 @@ +spec: + containers: + - name: kube-apiserver + resources: + requests: + cpu: 250m + memory: 1Gi diff --git a/charts/kubeadm/templates/patches/kube-controller-manager0.yaml b/charts/kubeadm/templates/patches/kube-controller-manager0.yaml new file mode 100644 index 0000000..cb08409 --- /dev/null +++ b/charts/kubeadm/templates/patches/kube-controller-manager0.yaml @@ -0,0 +1,7 @@ +spec: + containers: + - name: kube-controller-manager + resources: + requests: + cpu: 200m + memory: 128Mi diff --git a/charts/kubeadm/templates/patches/kube-scheduler0.yaml b/charts/kubeadm/templates/patches/kube-scheduler0.yaml new file mode 100644 index 0000000..837fd63 --- /dev/null +++ b/charts/kubeadm/templates/patches/kube-scheduler0.yaml @@ -0,0 +1,7 @@ +spec: + containers: + - name: kube-scheduler + resources: + requests: + cpu: 100m + memory: 64Mi diff --git a/charts/kubeadm/values.yaml b/charts/kubeadm/values.yaml index b4d6148..6798ff8 100644 --- a/charts/kubeadm/values.yaml +++ b/charts/kubeadm/values.yaml @@ -1,11 +1,17 @@ -clusterVersion: 1.18.0 +clusterVersion: 1.19.0 +listenAddress: 0.0.0.0 clusterName: pleasechangeme apiEndpoint: kube-api.changeme.org:6443 etcdExtraArgs: {} - # Enable for > 1.18 - # unsafe-no-fsync: "true" apiExtraArgs: {} clusterHighAvailable: false allEtcdEndpoints: "" -joinToken: "" -caCertHash: "" +# supported values aws,bare-metal +platform: "aws" +# Set to false for openrc, eg. on Gentoo or Alpine +systemd: true +protectKernelDefaults: true + +WorkerNodeRole: "arn:aws:iam::000000000000:role/KubernetesNode" +WorkerIamRole: "arn:aws:iam::000000000000:role/KubernetesNode" +KubeAdminRole: "arn:aws:iam::000000000000:role/KubernetesNode" diff --git a/charts/kubezero-argocd/Chart.yaml b/charts/kubezero-argocd/Chart.yaml index 52e6369..64c2eb8 100644 --- a/charts/kubezero-argocd/Chart.yaml +++ b/charts/kubezero-argocd/Chart.yaml @@ -1,7 +1,7 @@ apiVersion: v2 description: KubeZero ArgoCD Helm chart to install ArgoCD itself and the KubeZero ArgoCD Application name: kubezero-argocd -version: 0.6.1 +version: 0.7.1 home: https://kubezero.com icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png keywords: @@ -15,6 +15,6 @@ dependencies: version: ">= 0.1.3" repository: https://zero-down-time.github.io/kubezero/ - name: argo-cd - version: 2.9.5 + version: 2.17.4 repository: https://argoproj.github.io/argo-helm -kubeVersion: ">= 1.16.0" +kubeVersion: ">= 1.18.0" diff --git a/charts/kubezero-argocd/README.md b/charts/kubezero-argocd/README.md index 81b8cfe..f972fd2 100644 --- a/charts/kubezero-argocd/README.md +++ b/charts/kubezero-argocd/README.md @@ -1,6 +1,6 @@ # kubezero-argocd -![Version: 0.6.1](https://img.shields.io/badge/Version-0.6.1-informational?style=flat-square) +![Version: 0.7.1](https://img.shields.io/badge/Version-0.7.1-informational?style=flat-square) KubeZero ArgoCD Helm chart to install ArgoCD itself and the KubeZero ArgoCD Application @@ -14,11 +14,11 @@ KubeZero ArgoCD Helm chart to install ArgoCD itself and the KubeZero ArgoCD Appl ## Requirements -Kubernetes: `>= 1.16.0` +Kubernetes: `>= 1.18.0` | Repository | Name | Version | |------------|------|---------| -| https://argoproj.github.io/argo-helm | argo-cd | 2.9.5 | +| https://argoproj.github.io/argo-helm | argo-cd | 2.17.4 | | https://zero-down-time.github.io/kubezero/ | kubezero-lib | >= 0.1.3 | ## Values @@ -35,7 +35,7 @@ Kubernetes: `>= 1.16.0` | argo-cd.controller.resources.requests.cpu | string | `"100m"` | | | argo-cd.controller.resources.requests.memory | string | `"256Mi"` | | | argo-cd.dex.enabled | bool | `false` | | -| argo-cd.global.image.tag | string | `"v1.7.10"` | | +| argo-cd.global.image.tag | string | `"v1.8.7"` | | | argo-cd.installCRDs | bool | `false` | | | argo-cd.repoServer.logFormat | string | `"json"` | | | argo-cd.repoServer.metrics.enabled | bool | `false` | | diff --git a/charts/kubezero-argocd/values.yaml b/charts/kubezero-argocd/values.yaml index 481fd85..ecba336 100644 --- a/charts/kubezero-argocd/values.yaml +++ b/charts/kubezero-argocd/values.yaml @@ -42,7 +42,7 @@ argo-cd: global: image: - tag: v1.7.10 + tag: v1.8.7 controller: args: diff --git a/charts/kubezero-aws-ebs-csi-driver/Chart.yaml b/charts/kubezero-aws-ebs-csi-driver/Chart.yaml index a13400a..0f0e48a 100644 --- a/charts/kubezero-aws-ebs-csi-driver/Chart.yaml +++ b/charts/kubezero-aws-ebs-csi-driver/Chart.yaml @@ -2,8 +2,8 @@ apiVersion: v2 name: kubezero-aws-ebs-csi-driver description: KubeZero Umbrella Chart for aws-ebs-csi-driver type: application -version: 0.3.4 -appVersion: 0.8.0 +version: 0.4.4 +appVersion: 0.9.0 home: https://kubezero.com icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png sources: @@ -18,9 +18,9 @@ maintainers: - name: Quarky9 dependencies: - name: aws-ebs-csi-driver - version: 0.7.1 + version: 0.9.14 repository: https://kubernetes-sigs.github.io/aws-ebs-csi-driver - name: kubezero-lib version: ">= 0.1.3" repository: https://zero-down-time.github.io/kubezero/ -kubeVersion: ">= 1.16.0" +kubeVersion: ">= 1.18.0" diff --git a/charts/kubezero-aws-ebs-csi-driver/README.md b/charts/kubezero-aws-ebs-csi-driver/README.md index fc14ab9..a76dc2c 100644 --- a/charts/kubezero-aws-ebs-csi-driver/README.md +++ b/charts/kubezero-aws-ebs-csi-driver/README.md @@ -1,6 +1,6 @@ # kubezero-aws-ebs-csi-driver -![Version: 0.3.4](https://img.shields.io/badge/Version-0.3.4-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 0.8.0](https://img.shields.io/badge/AppVersion-0.8.0-informational?style=flat-square) +![Version: 0.4.4](https://img.shields.io/badge/Version-0.4.4-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 0.9.0](https://img.shields.io/badge/AppVersion-0.9.0-informational?style=flat-square) KubeZero Umbrella Chart for aws-ebs-csi-driver @@ -19,11 +19,11 @@ KubeZero Umbrella Chart for aws-ebs-csi-driver ## Requirements -Kubernetes: `>= 1.16.0` +Kubernetes: `>= 1.18.0` | Repository | Name | Version | |------------|------|---------| -| https://kubernetes-sigs.github.io/aws-ebs-csi-driver | aws-ebs-csi-driver | 0.7.1 | +| https://kubernetes-sigs.github.io/aws-ebs-csi-driver | aws-ebs-csi-driver | 0.9.14 | | https://zero-down-time.github.io/kubezero/ | kubezero-lib | >= 0.1.3 | ## IAM Role @@ -41,14 +41,29 @@ This class is by default also set as default storage class. | Key | Type | Default | Description | |-----|------|---------|-------------| -| StorageClass.create | bool | `true` | | -| StorageClass.default | bool | `true` | | -| aws-ebs-csi-driver.enableVolumeResizing | bool | `false` | | +| aws-ebs-csi-driver.enableVolumeResizing | bool | `true` | | | aws-ebs-csi-driver.enableVolumeScheduling | bool | `true` | | -| aws-ebs-csi-driver.enableVolumeSnapshot | bool | `false` | | +| aws-ebs-csi-driver.enableVolumeSnapshot | bool | `true` | | | aws-ebs-csi-driver.extraVolumeTags | object | `{}` | Optional tags to be added to each EBS volume | +| aws-ebs-csi-driver.logLevel | int | `1` | | | aws-ebs-csi-driver.nodeSelector."node-role.kubernetes.io/master" | string | `""` | | | aws-ebs-csi-driver.podAnnotations | object | `{}` | iam.amazonaws.com/role: to assume | | aws-ebs-csi-driver.replicaCount | int | `1` | | +| aws-ebs-csi-driver.resources.limits.memory | string | `"40Mi"` | | +| aws-ebs-csi-driver.resources.requests.cpu | string | `"10m"` | | +| aws-ebs-csi-driver.resources.requests.memory | string | `"24Mi"` | | +| aws-ebs-csi-driver.storageClasses[0].allowVolumeExpansion | bool | `true` | | +| aws-ebs-csi-driver.storageClasses[0].name | string | `"ebs-sc-gp2-xfs"` | | +| aws-ebs-csi-driver.storageClasses[0].parameters."csi.storage.k8s.io/fstype" | string | `"xfs"` | | +| aws-ebs-csi-driver.storageClasses[0].parameters.encrypted | string | `"true"` | | +| aws-ebs-csi-driver.storageClasses[0].parameters.type | string | `"gp2"` | | +| aws-ebs-csi-driver.storageClasses[0].volumeBindingMode | string | `"WaitForFirstConsumer"` | | +| aws-ebs-csi-driver.storageClasses[1].allowVolumeExpansion | bool | `true` | | +| aws-ebs-csi-driver.storageClasses[1].annotations."storageclass.kubernetes.io/is-default-class" | string | `"true"` | | +| aws-ebs-csi-driver.storageClasses[1].name | string | `"ebs-sc-gp3-xfs"` | | +| aws-ebs-csi-driver.storageClasses[1].parameters."csi.storage.k8s.io/fstype" | string | `"xfs"` | | +| aws-ebs-csi-driver.storageClasses[1].parameters.encrypted | string | `"true"` | | +| aws-ebs-csi-driver.storageClasses[1].parameters.type | string | `"gp3"` | | +| aws-ebs-csi-driver.storageClasses[1].volumeBindingMode | string | `"WaitForFirstConsumer"` | | | aws-ebs-csi-driver.tolerations[0].effect | string | `"NoSchedule"` | | | aws-ebs-csi-driver.tolerations[0].key | string | `"node-role.kubernetes.io/master"` | | diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/.helmignore b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/.helmignore new file mode 100644 index 0000000..50af031 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/.helmignore @@ -0,0 +1,22 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*~ +# Various IDEs +.project +.idea/ +*.tmproj +.vscode/ diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/Chart.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/Chart.yaml new file mode 100644 index 0000000..9a5a2b9 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/Chart.yaml @@ -0,0 +1,18 @@ +apiVersion: v1 +appVersion: 0.9.0 +description: A Helm chart for AWS EBS CSI Driver +home: https://github.com/kubernetes-sigs/aws-ebs-csi-driver +keywords: +- aws +- ebs +- csi +kubeVersion: '>=1.17.0-0' +maintainers: +- email: chengpan@amazon.com + name: leakingtapan +- name: krmichel + url: https://github.com/krmichel +name: aws-ebs-csi-driver +sources: +- https://github.com/kubernetes-sigs/aws-ebs-csi-driver +version: 0.9.14 diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/NOTES.txt b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/NOTES.txt new file mode 100644 index 0000000..3717647 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/NOTES.txt @@ -0,0 +1,3 @@ +To verify that aws-ebs-csi-driver has started, run: + + kubectl get pod -n {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "aws-ebs-csi-driver.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/_helpers.tpl b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/_helpers.tpl new file mode 100644 index 0000000..fdc77c4 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/_helpers.tpl @@ -0,0 +1,69 @@ +{{/* vim: set filetype=mustache: */}} +{{/* +Expand the name of the chart. +*/}} +{{- define "aws-ebs-csi-driver.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{/* +Create a default fully qualified app name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +If release name contains chart name it will be used as a full name. +*/}} +{{- define "aws-ebs-csi-driver.fullname" -}} +{{- if .Values.fullnameOverride -}} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- $name := default .Chart.Name .Values.nameOverride -}} +{{- if contains $name .Release.Name -}} +{{- .Release.Name | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} +{{- end -}} +{{- end -}} +{{- end -}} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "aws-ebs-csi-driver.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{/* +Common labels +*/}} +{{- define "aws-ebs-csi-driver.labels" -}} +{{ include "aws-ebs-csi-driver.selectorLabels" . }} +{{- if ne .Release.Name "kustomize" }} +helm.sh/chart: {{ include "aws-ebs-csi-driver.chart" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} +{{- end -}} + +{{/* +Common selector labels +*/}} +{{- define "aws-ebs-csi-driver.selectorLabels" -}} +app.kubernetes.io/name: {{ include "aws-ebs-csi-driver.name" . }} +{{- if ne .Release.Name "kustomize" }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} +{{- end -}} + +{{/* +Convert the `--extra-volume-tags` command line arg from a map. +*/}} +{{- define "aws-ebs-csi-driver.extra-volume-tags" -}} +{{- $result := dict "pairs" (list) -}} +{{- range $key, $value := .Values.extraVolumeTags -}} +{{- $noop := printf "%s=%s" $key $value | append $result.pairs | set $result "pairs" -}} +{{- end -}} +{{- if gt (len $result.pairs) 0 -}} +{{- printf "%s=%s" "- --extra-volume-tags" (join "," $result.pairs) -}} +{{- end -}} +{{- end -}} diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-attacher.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-attacher.yaml new file mode 100644 index 0000000..e0919ce --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-attacher.yaml @@ -0,0 +1,23 @@ +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: ebs-external-attacher-role + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} +rules: + - apiGroups: [""] + resources: ["persistentvolumes"] + verbs: ["get", "list", "watch", "update", "patch"] + - apiGroups: [""] + resources: ["nodes"] + verbs: ["get", "list", "watch"] + - apiGroups: ["csi.storage.k8s.io"] + resources: ["csinodeinfos"] + verbs: ["get", "list", "watch"] + - apiGroups: ["storage.k8s.io"] + resources: ["volumeattachments"] + verbs: ["get", "list", "watch", "update", "patch"] + - apiGroups: [ "storage.k8s.io" ] + resources: [ "volumeattachments/status" ] + verbs: [ "patch" ] diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-provisioner.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-provisioner.yaml new file mode 100644 index 0000000..0fb7ded --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-provisioner.yaml @@ -0,0 +1,38 @@ +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: ebs-external-provisioner-role + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} +rules: + - apiGroups: [ "" ] + resources: [ "persistentvolumes" ] + verbs: [ "get", "list", "watch", "create", "delete" ] + - apiGroups: [ "" ] + resources: [ "persistentvolumeclaims" ] + verbs: [ "get", "list", "watch", "update" ] + - apiGroups: [ "storage.k8s.io" ] + resources: [ "storageclasses" ] + verbs: [ "get", "list", "watch" ] + - apiGroups: [ "" ] + resources: [ "events" ] + verbs: [ "list", "watch", "create", "update", "patch" ] + - apiGroups: [ "snapshot.storage.k8s.io" ] + resources: [ "volumesnapshots" ] + verbs: [ "get", "list" ] + - apiGroups: [ "snapshot.storage.k8s.io" ] + resources: [ "volumesnapshotcontents" ] + verbs: [ "get", "list" ] + - apiGroups: [ "storage.k8s.io" ] + resources: [ "csinodes" ] + verbs: [ "get", "list", "watch" ] + - apiGroups: [ "" ] + resources: [ "nodes" ] + verbs: [ "get", "list", "watch" ] + - apiGroups: [ "coordination.k8s.io" ] + resources: [ "leases" ] + verbs: [ "get", "watch", "list", "delete", "update", "create" ] + - apiGroups: [ "storage.k8s.io" ] + resources: [ "volumeattachments" ] + verbs: [ "get", "list", "watch" ] diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-resizer.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-resizer.yaml new file mode 100644 index 0000000..9d85b97 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-resizer.yaml @@ -0,0 +1,33 @@ +{{- if .Values.enableVolumeResizing }} +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: ebs-external-resizer-role + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} +rules: + # The following rule should be uncommented for plugins that require secrets + # for provisioning. + # - apiGroups: [""] + # resources: ["secrets"] + # verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["persistentvolumes"] + verbs: ["get", "list", "watch", "update", "patch"] + - apiGroups: [""] + resources: ["persistentvolumeclaims"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["persistentvolumeclaims/status"] + verbs: ["update", "patch"] + - apiGroups: ["storage.k8s.io"] + resources: ["storageclasses"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["events"] + verbs: ["list", "watch", "create", "update", "patch"] + - apiGroups: [""] + resources: ["pods"] + verbs: ["get", "list", "watch"] +{{- end}} diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-snapshot-controller.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-snapshot-controller.yaml new file mode 100644 index 0000000..ffdb1b7 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-snapshot-controller.yaml @@ -0,0 +1,35 @@ +{{- if .Values.enableVolumeSnapshot }} +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: ebs-snapshot-controller-role + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} +rules: + - apiGroups: [""] + resources: ["persistentvolumes"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["persistentvolumeclaims"] + verbs: ["get", "list", "watch", "update"] + - apiGroups: ["storage.k8s.io"] + resources: ["storageclasses"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["events"] + verbs: ["list", "watch", "create", "update", "patch"] + - apiGroups: ["snapshot.storage.k8s.io"] + resources: ["volumesnapshotclasses"] + verbs: ["get", "list", "watch"] + - apiGroups: ["snapshot.storage.k8s.io"] + resources: ["volumesnapshotcontents"] + verbs: ["create", "get", "list", "watch", "update", "delete"] + - apiGroups: ["snapshot.storage.k8s.io"] + resources: ["volumesnapshots"] + verbs: ["get", "list", "watch", "update"] + - apiGroups: ["snapshot.storage.k8s.io"] + resources: ["volumesnapshots/status"] + verbs: ["update"] + +{{- end }} diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-snapshotter.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-snapshotter.yaml new file mode 100644 index 0000000..061b565 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrole-snapshotter.yaml @@ -0,0 +1,25 @@ +{{- if .Values.enableVolumeSnapshot }} +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: ebs-external-snapshotter-role + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} +rules: + - apiGroups: [""] + resources: ["events"] + verbs: ["list", "watch", "create", "update", "patch"] + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "list"] + - apiGroups: ["snapshot.storage.k8s.io"] + resources: ["volumesnapshotclasses"] + verbs: ["get", "list", "watch"] + - apiGroups: ["snapshot.storage.k8s.io"] + resources: ["volumesnapshotcontents"] + verbs: ["create", "get", "list", "watch", "update", "delete"] + - apiGroups: ["snapshot.storage.k8s.io"] + resources: ["volumesnapshotcontents/status"] + verbs: ["update"] +{{- end }} diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-attacher.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-attacher.yaml new file mode 100644 index 0000000..c75cb9b --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-attacher.yaml @@ -0,0 +1,15 @@ +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: ebs-csi-attacher-binding + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} +subjects: + - kind: ServiceAccount + name: {{ .Values.serviceAccount.controller.name }} + namespace: {{ .Release.Namespace }} +roleRef: + kind: ClusterRole + name: ebs-external-attacher-role + apiGroup: rbac.authorization.k8s.io diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-provisioner.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-provisioner.yaml new file mode 100644 index 0000000..4a9174b --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-provisioner.yaml @@ -0,0 +1,15 @@ +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: ebs-csi-provisioner-binding + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} +subjects: + - kind: ServiceAccount + name: {{ .Values.serviceAccount.controller.name }} + namespace: {{ .Release.Namespace }} +roleRef: + kind: ClusterRole + name: ebs-external-provisioner-role + apiGroup: rbac.authorization.k8s.io diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-resizer.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-resizer.yaml new file mode 100644 index 0000000..6fe42d1 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-resizer.yaml @@ -0,0 +1,18 @@ +{{- if .Values.enableVolumeResizing }} +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: ebs-csi-resizer-binding + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} +subjects: + - kind: ServiceAccount + name: {{ .Values.serviceAccount.controller.name }} + namespace: {{ .Release.Namespace }} +roleRef: + kind: ClusterRole + name: ebs-external-resizer-role + apiGroup: rbac.authorization.k8s.io + +{{- end}} diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-snapshot-controller.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-snapshot-controller.yaml new file mode 100644 index 0000000..b74484f --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-snapshot-controller.yaml @@ -0,0 +1,18 @@ +{{- if .Values.enableVolumeSnapshot }} +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: ebs-csi-snapshot-controller-binding + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} +subjects: + - kind: ServiceAccount + name: {{ .Values.serviceAccount.snapshot.name }} + namespace: {{ .Release.Namespace }} +roleRef: + kind: ClusterRole + name: ebs-snapshot-controller-role + apiGroup: rbac.authorization.k8s.io + +{{- end }} diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-snapshotter.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-snapshotter.yaml new file mode 100644 index 0000000..cbc1169 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/clusterrolebinding-snapshotter.yaml @@ -0,0 +1,18 @@ +{{- if .Values.enableVolumeSnapshot }} +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: ebs-csi-snapshotter-binding + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} +subjects: + - kind: ServiceAccount + name: {{ .Values.serviceAccount.controller.name }} + namespace: {{ .Release.Namespace }} +roleRef: + kind: ClusterRole + name: ebs-external-snapshotter-role + apiGroup: rbac.authorization.k8s.io + +{{- end }} diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/controller.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/controller.yaml new file mode 100644 index 0000000..ccd7489 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/controller.yaml @@ -0,0 +1,245 @@ +# Controller Service +kind: Deployment +apiVersion: apps/v1 +metadata: + name: ebs-csi-controller + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: + app: ebs-csi-controller + {{- include "aws-ebs-csi-driver.selectorLabels" . | nindent 6 }} + template: + metadata: + labels: + app: ebs-csi-controller + {{- include "aws-ebs-csi-driver.labels" . | nindent 8 }} + {{- if .Values.podAnnotations }} + annotations: {{ toYaml .Values.podAnnotations | nindent 8 }} + {{- end }} + spec: + nodeSelector: + kubernetes.io/os: linux + {{- with .Values.nodeSelector }} +{{ toYaml . | indent 8 }} + {{- end }} + serviceAccountName: {{ .Values.serviceAccount.controller.name }} + priorityClassName: {{ .Values.priorityClassName | default "system-cluster-critical" }} + {{- with .Values.affinity }} + affinity: {{ toYaml . | nindent 8 }} + {{- end }} + tolerations: + {{- if .Values.tolerateAllTaints }} + - operator: Exists + {{- else }} + - key: CriticalAddonsOnly + operator: Exists + - operator: Exists + effect: NoExecute + tolerationSeconds: 300 + {{- end }} + {{- with .Values.tolerations }} +{{ toYaml . | indent 8 }} + {{- end }} +{{- if .Values.topologySpreadConstraints }} +{{- $tscLabelSelector := dict "labelSelector" ( dict "matchLabels" ( dict "app" "ebs-csi-controller" ) ) }} + topologySpreadConstraints: + {{- range .Values.topologySpreadConstraints }} + - {{ mergeOverwrite . $tscLabelSelector | toJson }} + {{- end }} +{{- end }} + containers: + - name: ebs-plugin + image: {{ .Values.image.repository }}:{{ .Values.image.tag }} + imagePullPolicy: {{ .Values.image.pullPolicy }} + args: + {{- if ne .Release.Name "kustomize" }} + - controller + {{- else }} + # - {all,controller,node} # specify the driver mode + {{- end }} + - --endpoint=$(CSI_ENDPOINT) + {{- if .Values.extraVolumeTags }} + {{- include "aws-ebs-csi-driver.extra-volume-tags" . | nindent 12 }} + {{- end }} + {{- if .Values.k8sTagClusterId }} + - --k8s-tag-cluster-id={{ .Values.k8sTagClusterId }} + {{- end }} + - --logtostderr + - --v={{ .Values.logLevel }} + env: + - name: CSI_ENDPOINT + value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock + - name: AWS_ACCESS_KEY_ID + valueFrom: + secretKeyRef: + name: aws-secret + key: key_id + optional: true + - name: AWS_SECRET_ACCESS_KEY + valueFrom: + secretKeyRef: + name: aws-secret + key: access_key + optional: true + {{- if .Values.region }} + - name: AWS_REGION + value: {{ .Values.region }} + {{- end }} +{{- if .Values.proxy.http_proxy }} + - name: HTTP_PROXY + value: {{ .Values.proxy.http_proxy | quote }} + - name: HTTPS_PROXY + value: {{ .Values.proxy.http_proxy | quote }} + - name: NO_PROXY + value: {{ .Values.proxy.no_proxy | quote }} +{{- end }} + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/sockets/pluginproxy/ + ports: + - name: healthz + containerPort: 9808 + protocol: TCP + livenessProbe: + httpGet: + path: /healthz + port: healthz + initialDelaySeconds: 10 + timeoutSeconds: 3 + periodSeconds: 10 + failureThreshold: 5 + readinessProbe: + httpGet: + path: /healthz + port: healthz + initialDelaySeconds: 10 + timeoutSeconds: 3 + periodSeconds: 10 + failureThreshold: 5 + {{- with .Values.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + - name: csi-provisioner + image: {{ printf "%s:%s" .Values.sidecars.provisionerImage.repository .Values.sidecars.provisionerImage.tag }} + args: + - --csi-address=$(ADDRESS) + - --v={{ .Values.logLevel }} + {{- if .Values.enableVolumeScheduling }} + - --feature-gates=Topology=true + {{- end}} + {{- if .Values.extraCreateMetadata }} + - --extra-create-metadata + {{- end}} + - --leader-election={{ ternary "true" "false" ( gt (.Values.replicaCount|int) 1 ) }} + - --default-fstype=ext4 + env: + - name: ADDRESS + value: /var/lib/csi/sockets/pluginproxy/csi.sock +{{- if .Values.proxy.http_proxy }} + - name: HTTP_PROXY + value: {{ .Values.proxy.http_proxy | quote }} + - name: HTTPS_PROXY + value: {{ .Values.proxy.http_proxy | quote }} + - name: NO_PROXY + value: {{ .Values.proxy.no_proxy | quote }} +{{- end }} + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/sockets/pluginproxy/ + {{- with .Values.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + - name: csi-attacher + image: {{ printf "%s:%s" .Values.sidecars.attacherImage.repository .Values.sidecars.attacherImage.tag }} + args: + - --csi-address=$(ADDRESS) + - --v={{ .Values.logLevel }} + - --leader-election={{ ternary "true" "false" ( gt (.Values.replicaCount|int) 1 ) }} + env: + - name: ADDRESS + value: /var/lib/csi/sockets/pluginproxy/csi.sock +{{- if .Values.proxy.http_proxy }} + - name: HTTP_PROXY + value: {{ .Values.proxy.http_proxy | quote }} + - name: HTTPS_PROXY + value: {{ .Values.proxy.http_proxy | quote }} + - name: NO_PROXY + value: {{ .Values.proxy.no_proxy | quote }} +{{- end }} + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/sockets/pluginproxy/ + {{- with .Values.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + {{- if .Values.enableVolumeSnapshot }} + - name: csi-snapshotter + image: {{ printf "%s:%s" .Values.sidecars.snapshotterImage.repository .Values.sidecars.snapshotterImage.tag }} + args: + - --csi-address=$(ADDRESS) + - --leader-election={{ ternary "true" "false" ( gt (.Values.replicaCount|int) 1 ) }} + env: + - name: ADDRESS + value: /var/lib/csi/sockets/pluginproxy/csi.sock +{{- if .Values.proxy.http_proxy }} + - name: HTTP_PROXY + value: {{ .Values.proxy.http_proxy | quote }} + - name: HTTPS_PROXY + value: {{ .Values.proxy.http_proxy | quote }} + - name: NO_PROXY + value: {{ .Values.proxy.no_proxy | quote }} +{{- end }} + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/sockets/pluginproxy/ + {{- with .Values.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + {{- end }} + {{- if .Values.enableVolumeResizing }} + - name: csi-resizer + image: {{ printf "%s:%s" .Values.sidecars.resizerImage.repository .Values.sidecars.resizerImage.tag }} + imagePullPolicy: Always + args: + - --csi-address=$(ADDRESS) + - --v={{ .Values.logLevel }} + env: + - name: ADDRESS + value: /var/lib/csi/sockets/pluginproxy/csi.sock +{{- if .Values.proxy.http_proxy }} + - name: HTTP_PROXY + value: {{ .Values.proxy.http_proxy | quote }} + - name: HTTPS_PROXY + value: {{ .Values.proxy.http_proxy | quote }} + - name: NO_PROXY + value: {{ .Values.proxy.no_proxy | quote }} +{{- end }} + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/sockets/pluginproxy/ + {{- with .Values.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + {{- end }} + - name: liveness-probe + image: {{ printf "%s:%s" .Values.sidecars.livenessProbeImage.repository .Values.sidecars.livenessProbeImage.tag }} + args: + - --csi-address=/csi/csi.sock + volumeMounts: + - name: socket-dir + mountPath: /csi + {{- with .Values.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + {{- if .Values.imagePullSecrets }} + imagePullSecrets: + {{- range .Values.imagePullSecrets }} + - name: {{ . }} + {{- end }} + {{- end }} + volumes: + - name: socket-dir + emptyDir: {} diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/csidriver.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/csidriver.yaml new file mode 100644 index 0000000..1858e39 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/csidriver.yaml @@ -0,0 +1,9 @@ +apiVersion: storage.k8s.io/v1beta1 +kind: CSIDriver +metadata: + name: ebs.csi.aws.com + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} +spec: + attachRequired: true + podInfoOnMount: false diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/node.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/node.yaml new file mode 100644 index 0000000..5caeb33 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/node.yaml @@ -0,0 +1,176 @@ +# Node Service +kind: DaemonSet +apiVersion: apps/v1 +metadata: + name: ebs-csi-node + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} +spec: + selector: + matchLabels: + app: ebs-csi-node + {{- include "aws-ebs-csi-driver.selectorLabels" . | nindent 6 }} + template: + metadata: + labels: + app: ebs-csi-node + {{- include "aws-ebs-csi-driver.labels" . | nindent 8 }} + {{- if .Values.node.podAnnotations }} + annotations: {{ toYaml .Values.node.podAnnotations | nindent 8 }} + {{- end }} + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: eks.amazonaws.com/compute-type + operator: NotIn + values: + - fargate + nodeSelector: + kubernetes.io/os: linux + {{- with .Values.node.nodeSelector }} +{{ toYaml . | indent 8 }} + {{- end }} + hostNetwork: true + serviceAccountName: {{ .Values.serviceAccount.node.name }} + priorityClassName: {{ .Values.node.priorityClassName | default "system-cluster-critical" }} + tolerations: + {{- if .Values.node.tolerateAllTaints }} + - operator: Exists + {{- else }} + - key: CriticalAddonsOnly + operator: Exists + - operator: Exists + effect: NoExecute + tolerationSeconds: 300 + {{- end }} + {{- with .Values.node.tolerations }} +{{ toYaml . | indent 8 }} + {{- end }} + containers: + - name: ebs-plugin + securityContext: + privileged: true + image: {{ .Values.image.repository }}:{{ .Values.image.tag }} + args: + - node + - --endpoint=$(CSI_ENDPOINT) + {{- if .Values.volumeAttachLimit }} + - --volume-attach-limit={{ .Values.volumeAttachLimit }} + {{- end }} + - --logtostderr + - --v={{ .Values.logLevel }} + env: + - name: CSI_ENDPOINT + value: unix:/csi/csi.sock +{{- if .Values.proxy.http_proxy }} + - name: HTTP_PROXY + value: {{ .Values.proxy.http_proxy | quote }} + - name: HTTPS_PROXY + value: {{ .Values.proxy.http_proxy | quote }} + - name: NO_PROXY + value: {{ .Values.proxy.no_proxy | quote }} +{{- end }} + volumeMounts: + - name: kubelet-dir + mountPath: /var/lib/kubelet + mountPropagation: "Bidirectional" + - name: plugin-dir + mountPath: /csi + - name: device-dir + mountPath: /dev + ports: + - name: healthz + containerPort: 9808 + protocol: TCP + livenessProbe: + httpGet: + path: /healthz + port: healthz + initialDelaySeconds: 10 + timeoutSeconds: 3 + periodSeconds: 10 + failureThreshold: 5 + {{- if .Values.node.resources }} + {{- with .Values.node.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + {{- else }} + {{- with .Values.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + {{- end }} + - name: node-driver-registrar + image: {{ printf "%s:%s" .Values.sidecars.nodeDriverRegistrarImage.repository .Values.sidecars.nodeDriverRegistrarImage.tag }} + args: + - --csi-address=$(ADDRESS) + - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH) + - --v={{ .Values.logLevel }} + env: + - name: ADDRESS + value: /csi/csi.sock + - name: DRIVER_REG_SOCK_PATH + value: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock +{{- if .Values.proxy.http_proxy }} + - name: HTTP_PROXY + value: {{ .Values.proxy.http_proxy | quote }} + - name: HTTPS_PROXY + value: {{ .Values.proxy.http_proxy | quote }} + - name: NO_PROXY + value: {{ .Values.proxy.no_proxy | quote }} +{{- end }} + volumeMounts: + - name: plugin-dir + mountPath: /csi + - name: registration-dir + mountPath: /registration + {{- if .Values.node.resources }} + {{- with .Values.node.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + {{- else }} + {{- with .Values.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + {{- end }} + - name: liveness-probe + image: {{ printf "%s:%s" .Values.sidecars.livenessProbeImage.repository .Values.sidecars.livenessProbeImage.tag }} + args: + - --csi-address=/csi/csi.sock + volumeMounts: + - name: plugin-dir + mountPath: /csi + {{- if .Values.node.resources }} + {{- with .Values.node.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + {{- else }} + {{- with .Values.resources }} + resources: {{ toYaml . | nindent 12 }} + {{- end }} + {{- end }} + {{- if .Values.imagePullSecrets }} + imagePullSecrets: + {{- range .Values.imagePullSecrets }} + - name: {{ . }} + {{- end }} + {{- end }} + volumes: + - name: kubelet-dir + hostPath: + path: /var/lib/kubelet + type: Directory + - name: plugin-dir + hostPath: + path: /var/lib/kubelet/plugins/ebs.csi.aws.com/ + type: DirectoryOrCreate + - name: registration-dir + hostPath: + path: /var/lib/kubelet/plugins_registry/ + type: Directory + - name: device-dir + hostPath: + path: /dev + type: Directory diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/role-snapshot-controller-leaderelection.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/role-snapshot-controller-leaderelection.yaml new file mode 100644 index 0000000..4d09e4c --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/role-snapshot-controller-leaderelection.yaml @@ -0,0 +1,14 @@ +{{- if .Values.enableVolumeSnapshot }} +--- +kind: Role +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: ebs-snapshot-controller-leaderelection + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} +rules: + - apiGroups: ["coordination.k8s.io"] + resources: ["leases"] + verbs: ["get", "watch", "list", "delete", "update", "create"] + +{{- end }} diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/rolebinding-snapshot-controller-leaderelection.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/rolebinding-snapshot-controller-leaderelection.yaml new file mode 100644 index 0000000..e8248bd --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/rolebinding-snapshot-controller-leaderelection.yaml @@ -0,0 +1,18 @@ +{{- if .Values.enableVolumeSnapshot }} +--- +kind: RoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: ebs-snapshot-controller-leaderelection + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} +subjects: + - kind: ServiceAccount + name: {{ .Values.serviceAccount.snapshot.name }} + namespace: {{ .Release.Namespace }} +roleRef: + kind: Role + name: ebs-snapshot-controller-leaderelection + apiGroup: rbac.authorization.k8s.io + +{{- end }} diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/serviceaccount-csi-controller.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/serviceaccount-csi-controller.yaml new file mode 100644 index 0000000..0490c32 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/serviceaccount-csi-controller.yaml @@ -0,0 +1,16 @@ +{{- if .Values.serviceAccount.controller.create -}} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ .Values.serviceAccount.controller.name }} + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.controller.annotations }} + annotations: {{ toYaml . | nindent 4 }} + {{- end }} + {{- if eq .Release.Name "kustomize" }} + #Enable if EKS IAM for SA is used + #annotations: + # eks.amazonaws.com/role-arn: arn:aws:iam::586565787010:role/ebs-csi-role + {{- end }} +{{- end -}} diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/serviceaccount-csi-node.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/serviceaccount-csi-node.yaml new file mode 100644 index 0000000..2e93f72 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/serviceaccount-csi-node.yaml @@ -0,0 +1,11 @@ +{{- if .Values.serviceAccount.node.create -}} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ .Values.serviceAccount.node.name }} + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.node.annotations }} + annotations: {{ toYaml . | nindent 4 }} + {{- end }} +{{- end -}} diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/serviceaccount-snapshot-controller.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/serviceaccount-snapshot-controller.yaml new file mode 100644 index 0000000..19d27cb --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/serviceaccount-snapshot-controller.yaml @@ -0,0 +1,14 @@ +{{- if .Values.enableVolumeSnapshot }} +{{- if .Values.serviceAccount.snapshot.create }} +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ .Values.serviceAccount.snapshot.name }} + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.snapshot.annotations }} + annotations: {{ toYaml . | nindent 4 }} + {{- end }} +{{- end }} +{{- end }} diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/statefulset.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/statefulset.yaml new file mode 100644 index 0000000..ffde0ba --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/statefulset.yaml @@ -0,0 +1,60 @@ +{{- if .Values.enableVolumeSnapshot }} +#Snapshot controller +kind: StatefulSet +apiVersion: apps/v1 +metadata: + name: ebs-snapshot-controller + labels: + {{- include "aws-ebs-csi-driver.labels" . | nindent 4 }} +spec: + serviceName: ebs-snapshot-controller + replicas: 1 + selector: + matchLabels: + app: ebs-snapshot-controller + {{- include "aws-ebs-csi-driver.selectorLabels" . | nindent 6 }} + template: + metadata: + labels: + app: ebs-snapshot-controller + {{- include "aws-ebs-csi-driver.labels" . | nindent 8 }} + spec: + serviceAccountName: {{ .Values.serviceAccount.snapshot.name }} + nodeSelector: + kubernetes.io/os: linux + {{- with .Values.nodeSelector }} +{{ toYaml . | indent 8 }} + {{- end }} + priorityClassName: {{ .Values.priorityClassName | default "system-cluster-critical" }} + {{- with .Values.affinity }} + affinity: {{ toYaml . | nindent 8 }} + {{- end }} + tolerations: + {{- if .Values.tolerateAllTaints }} + - operator: Exists + {{- end }} + {{- with .Values.tolerations }} +{{ toYaml . | indent 8 }} + {{- end }} + containers: + - name: snapshot-controller + image: {{ printf "%s:%s" .Values.snapshotController.repository .Values.snapshotController.tag }} + env: +{{- if .Values.proxy.http_proxy }} + - name: HTTP_PROXY + value: {{ .Values.proxy.http_proxy | quote }} + - name: HTTPS_PROXY + value: {{ .Values.proxy.http_proxy | quote }} + - name: NO_PROXY + value: {{ .Values.proxy.no_proxy | quote }} +{{- end }} + args: + - --v={{ .Values.logLevel }} + - --leader-election=false + {{- if .Values.imagePullSecrets }} + imagePullSecrets: + {{- range .Values.imagePullSecrets }} + - name: {{ . }} + {{- end }} + {{- end }} +{{- end }} diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/storageclass.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/storageclass.yaml new file mode 100644 index 0000000..3da90e3 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/templates/storageclass.yaml @@ -0,0 +1,15 @@ +{{- range .Values.storageClasses }} +--- +kind: StorageClass +apiVersion: storage.k8s.io/v1 +metadata: + name: {{ .name }} + {{- if .annotations }} + annotations: {{- .annotations | toYaml | trim | nindent 4 }} + {{- end }} + {{- if .labels }} + labels: {{- .labels | toYaml | trim | nindent 4 }} + {{- end }} +provisioner: ebs.csi.aws.com +{{ omit (dict "volumeBindingMode" "WaitForFirstConsumer" | merge .) "name" "annotations" "labels" | toYaml }} +{{- end }} diff --git a/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/values.yaml b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/values.yaml new file mode 100644 index 0000000..624d0e9 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/charts/aws-ebs-csi-driver/values.yaml @@ -0,0 +1,148 @@ +# Default values for aws-ebs-csi-driver. +# This is a YAML-formatted file. +# Declare variables to be passed into your templates. + +replicaCount: 2 + +image: + repository: k8s.gcr.io/provider-aws/aws-ebs-csi-driver + tag: "v0.9.0" + pullPolicy: IfNotPresent + +logLevel: 5 + +sidecars: + provisionerImage: + repository: k8s.gcr.io/sig-storage/csi-provisioner + tag: "v2.0.2" + attacherImage: + repository: k8s.gcr.io/sig-storage/csi-attacher + tag: "v3.0.0" + snapshotterImage: + repository: k8s.gcr.io/sig-storage/csi-snapshotter + tag: "v3.0.3" + livenessProbeImage: + repository: k8s.gcr.io/sig-storage/livenessprobe + tag: "v2.2.0" + resizerImage: + repository: k8s.gcr.io/sig-storage/csi-resizer + tag: "v1.0.0" + nodeDriverRegistrarImage: + repository: k8s.gcr.io/sig-storage/csi-node-driver-registrar + tag: "v2.0.1" + +snapshotController: + repository: k8s.gcr.io/sig-storage/snapshot-controller + tag: "v3.0.3" + +proxy: {} +# http_proxy: +# no_proxy: + +imagePullSecrets: [] +nameOverride: "" +fullnameOverride: "" + +podAnnotations: {} + +# True if enable volume scheduling for dynamic volume provisioning +enableVolumeScheduling: true + +# True if enable volume resizing +enableVolumeResizing: false + +# True if enable volume snapshot +enableVolumeSnapshot: false + +# The "maximum number of attachable volumes" per node +volumeAttachLimit: "" + +resources: + {} + # We usually recommend not to specify default resources and to leave this as a conscious + # choice for the user. This also increases chances charts run on environments with little + # resources, such as Minikube. If you do want to specify resources, uncomment the following + # lines, adjust them as necessary, and remove the curly braces after 'resources:'. + # limits: + # cpu: 100m + # memory: 128Mi + # requests: + # cpu: 100m + # memory: 128Mi + +priorityClassName: "" +nodeSelector: {} +tolerateAllTaints: true +tolerations: [] +affinity: {} + +# TSCs without the label selector stanza +# +# Example: +# +# topologySpreadConstraints: +# - maxSkew: 1 +# topologyKey: topology.kubernetes.io/zone +# whenUnsatisfiable: ScheduleAnyway +# - maxSkew: 1 +# topologyKey: kubernetes.io/hostname +# whenUnsatisfiable: ScheduleAnyway + +topologySpreadConstraints: [] + +# Extra volume tags to attach to each dynamically provisioned volume. +# --- +# extraVolumeTags: +# key1: value1 +# key2: value2 +extraVolumeTags: {} + +# If set, add pv/pvc metadata to plugin create requests as parameters. +extraCreateMetadata: false + +# ID of the Kubernetes cluster used for tagging provisioned EBS volumes (optional). +k8sTagClusterId: "" + +# AWS region to use. If not specified then the region will be looked up via the AWS EC2 metadata +# service. +# --- +# region: us-east-1 +region: "" + +node: + priorityClassName: "" + nodeSelector: {} + podAnnotations: {} + tolerateAllTaints: true + tolerations: [] + resources: {} + +serviceAccount: + controller: + create: true # A service account will be created for you if set to true. Set to false if you want to use your own. + name: ebs-csi-controller-sa # Name of the service-account to be used/created. + annotations: {} + snapshot: + create: true + name: ebs-snapshot-controller + annotations: {} + node: + create: true + name: ebs-csi-node-sa + annotations: {} + +storageClasses: [] +# Add StorageClass resources like: +# - name: ebs-sc +# # annotation metadata +# annotations: +# storageclass.kubernetes.io/is-default-class: "true" +# # label metadata +# labels: +# my-label-is: supercool +# # defaults to WaitForFirstConsumer +# volumeBindingMode: WaitForFirstConsumer +# # defaults to Delete +# reclaimPolicy: Retain +# parameters: +# encrypted: "true" diff --git a/charts/kubezero-aws-ebs-csi-driver/crds/crd_snapshotter.yaml b/charts/kubezero-aws-ebs-csi-driver/crds/crd_snapshotter.yaml new file mode 100644 index 0000000..aff837a --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/crds/crd_snapshotter.yaml @@ -0,0 +1,420 @@ +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: (devel) + api-approved.kubernetes.io: "https://github.com/kubernetes-csi/external-snapshotter/pull/139" + creationTimestamp: null + name: volumesnapshotclasses.snapshot.storage.k8s.io +spec: + group: snapshot.storage.k8s.io + names: + kind: VolumeSnapshotClass + listKind: VolumeSnapshotClassList + plural: volumesnapshotclasses + singular: volumesnapshotclass + scope: Cluster + preserveUnknownFields: false + validation: + openAPIV3Schema: + description: VolumeSnapshotClass specifies parameters that a underlying storage + system uses when creating a volume snapshot. A specific VolumeSnapshotClass + is used by specifying its name in a VolumeSnapshot object. VolumeSnapshotClasses + are non-namespaced + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources' + type: string + deletionPolicy: + description: deletionPolicy determines whether a VolumeSnapshotContent created + through the VolumeSnapshotClass should be deleted when its bound VolumeSnapshot + is deleted. Supported values are "Retain" and "Delete". "Retain" means + that the VolumeSnapshotContent and its physical snapshot on underlying + storage system are kept. "Delete" means that the VolumeSnapshotContent + and its physical snapshot on underlying storage system are deleted. Required. + enum: + - Delete + - Retain + type: string + driver: + description: driver is the name of the storage driver that handles this + VolumeSnapshotClass. Required. + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' + type: string + parameters: + additionalProperties: + type: string + description: parameters is a key-value map with storage driver specific + parameters for creating snapshots. These values are opaque to Kubernetes. + type: object + required: + - deletionPolicy + - driver + type: object + version: v1beta1 + versions: + - name: v1beta1 + served: true + storage: true +status: + acceptedNames: + kind: "" + plural: "" + conditions: [] + storedVersions: [] + +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: (devel) + api-approved.kubernetes.io: "https://github.com/kubernetes-csi/external-snapshotter/pull/139" + creationTimestamp: null + name: volumesnapshotcontents.snapshot.storage.k8s.io +spec: + group: snapshot.storage.k8s.io + names: + kind: VolumeSnapshotContent + listKind: VolumeSnapshotContentList + plural: volumesnapshotcontents + singular: volumesnapshotcontent + scope: Cluster + subresources: + status: {} + preserveUnknownFields: false + validation: + openAPIV3Schema: + description: VolumeSnapshotContent represents the actual "on-disk" snapshot + object in the underlying storage system + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' + type: string + spec: + description: spec defines properties of a VolumeSnapshotContent created + by the underlying storage system. Required. + properties: + deletionPolicy: + description: deletionPolicy determines whether this VolumeSnapshotContent + and its physical snapshot on the underlying storage system should + be deleted when its bound VolumeSnapshot is deleted. Supported values + are "Retain" and "Delete". "Retain" means that the VolumeSnapshotContent + and its physical snapshot on underlying storage system are kept. "Delete" + means that the VolumeSnapshotContent and its physical snapshot on + underlying storage system are deleted. In dynamic snapshot creation + case, this field will be filled in with the "DeletionPolicy" field + defined in the VolumeSnapshotClass the VolumeSnapshot refers to. For + pre-existing snapshots, users MUST specify this field when creating + the VolumeSnapshotContent object. Required. + enum: + - Delete + - Retain + type: string + driver: + description: driver is the name of the CSI driver used to create the + physical snapshot on the underlying storage system. This MUST be the + same as the name returned by the CSI GetPluginName() call for that + driver. Required. + type: string + source: + description: source specifies from where a snapshot will be created. + This field is immutable after creation. Required. + properties: + snapshotHandle: + description: snapshotHandle specifies the CSI "snapshot_id" of a + pre-existing snapshot on the underlying storage system. This field + is immutable. + type: string + volumeHandle: + description: volumeHandle specifies the CSI "volume_id" of the volume + from which a snapshot should be dynamically taken from. This field + is immutable. + type: string + type: object + volumeSnapshotClassName: + description: name of the VolumeSnapshotClass to which this snapshot + belongs. + type: string + volumeSnapshotRef: + description: volumeSnapshotRef specifies the VolumeSnapshot object to + which this VolumeSnapshotContent object is bound. VolumeSnapshot.Spec.VolumeSnapshotContentName + field must reference to this VolumeSnapshotContent's name for the + bidirectional binding to be valid. For a pre-existing VolumeSnapshotContent + object, name and namespace of the VolumeSnapshot object MUST be provided + for binding to happen. This field is immutable after creation. Required. + properties: + apiVersion: + description: API version of the referent. + type: string + fieldPath: + description: 'If referring to a piece of an object instead of an + entire object, this string should contain a valid JSON/Go field + access statement, such as desiredState.manifest.containers[2]. + For example, if the object reference is to a container within + a pod, this would take on a value like: "spec.containers{name}" + (where "name" refers to the name of the container that triggered + the event) or if no container name is specified "spec.containers[2]" + (container with index 2 in this pod). This syntax is chosen only + to have some well-defined way of referencing a part of an object. + TODO: this design is not final and this field is subject to change + in the future.' + type: string + kind: + description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' + type: string + namespace: + description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/' + type: string + resourceVersion: + description: 'Specific resourceVersion to which this reference is + made, if any. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#concurrency-control-and-consistency' + type: string + uid: + description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids' + type: string + type: object + required: + - deletionPolicy + - driver + - source + - volumeSnapshotRef + type: object + status: + description: status represents the current information of a snapshot. + properties: + creationTime: + description: creationTime is the timestamp when the point-in-time snapshot + is taken by the underlying storage system. In dynamic snapshot creation + case, this field will be filled in with the "creation_time" value + returned from CSI "CreateSnapshotRequest" gRPC call. For a pre-existing + snapshot, this field will be filled with the "creation_time" value + returned from the CSI "ListSnapshots" gRPC call if the driver supports + it. If not specified, it indicates the creation time is unknown. The + format of this field is a Unix nanoseconds time encoded as an int64. + On Unix, the command `date +%s%N` returns the current time in nanoseconds + since 1970-01-01 00:00:00 UTC. + format: int64 + type: integer + error: + description: error is the latest observed error during snapshot creation, + if any. + properties: + message: + description: 'message is a string detailing the encountered error + during snapshot creation if specified. NOTE: message may be logged, + and it should not contain sensitive information.' + type: string + time: + description: time is the timestamp when the error was encountered. + format: date-time + type: string + type: object + readyToUse: + description: readyToUse indicates if a snapshot is ready to be used + to restore a volume. In dynamic snapshot creation case, this field + will be filled in with the "ready_to_use" value returned from CSI + "CreateSnapshotRequest" gRPC call. For a pre-existing snapshot, this + field will be filled with the "ready_to_use" value returned from the + CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, + this field will be set to "True". If not specified, it means the readiness + of a snapshot is unknown. + type: boolean + restoreSize: + description: restoreSize represents the complete size of the snapshot + in bytes. In dynamic snapshot creation case, this field will be filled + in with the "size_bytes" value returned from CSI "CreateSnapshotRequest" + gRPC call. For a pre-existing snapshot, this field will be filled + with the "size_bytes" value returned from the CSI "ListSnapshots" + gRPC call if the driver supports it. When restoring a volume from + this snapshot, the size of the volume MUST NOT be smaller than the + restoreSize if it is specified, otherwise the restoration will fail. + If not specified, it indicates that the size is unknown. + format: int64 + minimum: 0 + type: integer + snapshotHandle: + description: snapshotHandle is the CSI "snapshot_id" of a snapshot on + the underlying storage system. If not specified, it indicates that + dynamic snapshot creation has either failed or it is still in progress. + type: string + type: object + required: + - spec + type: object + version: v1beta1 + versions: + - name: v1beta1 + served: true + storage: true +status: + acceptedNames: + kind: "" + plural: "" + conditions: [] + storedVersions: [] + +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: (devel) + api-approved.kubernetes.io: "https://github.com/kubernetes-csi/external-snapshotter/pull/139" + creationTimestamp: null + name: volumesnapshots.snapshot.storage.k8s.io +spec: + group: snapshot.storage.k8s.io + names: + kind: VolumeSnapshot + listKind: VolumeSnapshotList + plural: volumesnapshots + singular: volumesnapshot + scope: Namespaced + subresources: + status: {} + preserveUnknownFields: false + validation: + openAPIV3Schema: + description: VolumeSnapshot is a user's request for either creating a point-in-time + snapshot of a persistent volume, or binding to a pre-existing snapshot. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' + type: string + spec: + description: 'spec defines the desired characteristics of a snapshot requested + by a user. More info: https://kubernetes.io/docs/concepts/storage/volume-snapshots#volumesnapshots + Required.' + properties: + source: + description: source specifies where a snapshot will be created from. + This field is immutable after creation. Required. + properties: + persistentVolumeClaimName: + description: persistentVolumeClaimName specifies the name of the + PersistentVolumeClaim object in the same namespace as the VolumeSnapshot + object where the snapshot should be dynamically taken from. This + field is immutable. + type: string + volumeSnapshotContentName: + description: volumeSnapshotContentName specifies the name of a pre-existing + VolumeSnapshotContent object. This field is immutable. + type: string + type: object + volumeSnapshotClassName: + description: 'volumeSnapshotClassName is the name of the VolumeSnapshotClass + requested by the VolumeSnapshot. If not specified, the default snapshot + class will be used if one exists. If not specified, and there is no + default snapshot class, dynamic snapshot creation will fail. Empty + string is not allowed for this field. TODO(xiangqian): a webhook validation + on empty string. More info: https://kubernetes.io/docs/concepts/storage/volume-snapshot-classes' + type: string + required: + - source + type: object + status: + description: 'status represents the current information of a snapshot. NOTE: + status can be modified by sources other than system controllers, and must + not be depended upon for accuracy. Controllers should only use information + from the VolumeSnapshotContent object after verifying that the binding + is accurate and complete.' + properties: + boundVolumeSnapshotContentName: + description: 'boundVolumeSnapshotContentName represents the name of + the VolumeSnapshotContent object to which the VolumeSnapshot object + is bound. If not specified, it indicates that the VolumeSnapshot object + has not been successfully bound to a VolumeSnapshotContent object + yet. NOTE: Specified boundVolumeSnapshotContentName alone does not + mean binding is valid. Controllers MUST always verify bidirectional + binding between VolumeSnapshot and VolumeSnapshotContent to + avoid possible security issues.' + type: string + creationTime: + description: creationTime is the timestamp when the point-in-time snapshot + is taken by the underlying storage system. In dynamic snapshot creation + case, this field will be filled in with the "creation_time" value + returned from CSI "CreateSnapshotRequest" gRPC call. For a pre-existing + snapshot, this field will be filled with the "creation_time" value + returned from the CSI "ListSnapshots" gRPC call if the driver supports + it. If not specified, it indicates that the creation time of the snapshot + is unknown. + format: date-time + type: string + error: + description: error is the last observed error during snapshot creation, + if any. This field could be helpful to upper level controllers(i.e., + application controller) to decide whether they should continue on + waiting for the snapshot to be created based on the type of error + reported. + properties: + message: + description: 'message is a string detailing the encountered error + during snapshot creation if specified. NOTE: message may be logged, + and it should not contain sensitive information.' + type: string + time: + description: time is the timestamp when the error was encountered. + format: date-time + type: string + type: object + readyToUse: + description: readyToUse indicates if a snapshot is ready to be used + to restore a volume. In dynamic snapshot creation case, this field + will be filled in with the "ready_to_use" value returned from CSI + "CreateSnapshotRequest" gRPC call. For a pre-existing snapshot, this + field will be filled with the "ready_to_use" value returned from the + CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, + this field will be set to "True". If not specified, it means the readiness + of a snapshot is unknown. + type: boolean + restoreSize: + description: restoreSize represents the complete size of the snapshot + in bytes. In dynamic snapshot creation case, this field will be filled + in with the "size_bytes" value returned from CSI "CreateSnapshotRequest" + gRPC call. For a pre-existing snapshot, this field will be filled + with the "size_bytes" value returned from the CSI "ListSnapshots" + gRPC call if the driver supports it. When restoring a volume from + this snapshot, the size of the volume MUST NOT be smaller than the + restoreSize if it is specified, otherwise the restoration will fail. + If not specified, it indicates that the size is unknown. + type: string + type: object + required: + - spec + type: object + version: v1beta1 + versions: + - name: v1beta1 + served: true + storage: true +status: + acceptedNames: + kind: "" + plural: "" + conditions: [] + storedVersions: [] diff --git a/charts/kubezero-aws-ebs-csi-driver/loglevel_leader.patch b/charts/kubezero-aws-ebs-csi-driver/loglevel_leader.patch new file mode 100644 index 0000000..349eea0 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/loglevel_leader.patch @@ -0,0 +1,103 @@ +diff -rtubN aws-ebs-csi-driver.orig/templates/controller.yaml aws-ebs-csi-driver/templates/controller.yaml +--- aws-ebs-csi-driver.orig/templates/controller.yaml 2021-03-05 03:10:41.000000000 +0100 ++++ aws-ebs-csi-driver/templates/controller.yaml 2021-03-05 10:29:31.878615411 +0100 +@@ -68,7 +68,7 @@ + - --k8s-tag-cluster-id={{ .Values.k8sTagClusterId }} + {{- end }} + - --logtostderr +- - --v=5 ++ - --v={{ .Values.logLevel }} + env: + - name: CSI_ENDPOINT + value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock +@@ -126,14 +126,14 @@ + image: {{ printf "%s:%s" .Values.sidecars.provisionerImage.repository .Values.sidecars.provisionerImage.tag }} + args: + - --csi-address=$(ADDRESS) +- - --v=5 ++ - --v={{ .Values.logLevel }} + {{- if .Values.enableVolumeScheduling }} + - --feature-gates=Topology=true + {{- end}} + {{- if .Values.extraCreateMetadata }} + - --extra-create-metadata + {{- end}} +- - --leader-election=true ++ - --leader-election={{ ternary "true" "false" ( gt (.Values.replicaCount|int) 1 ) }} + - --default-fstype=ext4 + env: + - name: ADDRESS +@@ -156,8 +156,8 @@ + image: {{ printf "%s:%s" .Values.sidecars.attacherImage.repository .Values.sidecars.attacherImage.tag }} + args: + - --csi-address=$(ADDRESS) +- - --v=5 +- - --leader-election=true ++ - --v={{ .Values.logLevel }} ++ - --leader-election={{ ternary "true" "false" ( gt (.Values.replicaCount|int) 1 ) }} + env: + - name: ADDRESS + value: /var/lib/csi/sockets/pluginproxy/csi.sock +@@ -180,7 +180,7 @@ + image: {{ printf "%s:%s" .Values.sidecars.snapshotterImage.repository .Values.sidecars.snapshotterImage.tag }} + args: + - --csi-address=$(ADDRESS) +- - --leader-election=true ++ - --leader-election={{ ternary "true" "false" ( gt (.Values.replicaCount|int) 1 ) }} + env: + - name: ADDRESS + value: /var/lib/csi/sockets/pluginproxy/csi.sock +@@ -205,7 +205,7 @@ + imagePullPolicy: Always + args: + - --csi-address=$(ADDRESS) +- - --v=5 ++ - --v={{ .Values.logLevel }} + env: + - name: ADDRESS + value: /var/lib/csi/sockets/pluginproxy/csi.sock +diff -rtubN aws-ebs-csi-driver.orig/templates/node.yaml aws-ebs-csi-driver/templates/node.yaml +--- aws-ebs-csi-driver.orig/templates/node.yaml 2021-03-05 03:10:41.000000000 +0100 ++++ aws-ebs-csi-driver/templates/node.yaml 2021-03-05 10:30:07.391950366 +0100 +@@ -61,7 +61,7 @@ + - --volume-attach-limit={{ .Values.volumeAttachLimit }} + {{- end }} + - --logtostderr +- - --v=5 ++ - --v={{ .Values.logLevel }} + env: + - name: CSI_ENDPOINT + value: unix:/csi/csi.sock +@@ -107,7 +107,7 @@ + args: + - --csi-address=$(ADDRESS) + - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH) +- - --v=5 ++ - --v={{ .Values.logLevel }} + env: + - name: ADDRESS + value: /csi/csi.sock +diff -rtubN aws-ebs-csi-driver.orig/templates/statefulset.yaml aws-ebs-csi-driver/templates/statefulset.yaml +--- aws-ebs-csi-driver.orig/templates/statefulset.yaml 2021-03-05 03:10:41.000000000 +0100 ++++ aws-ebs-csi-driver/templates/statefulset.yaml 2021-03-05 10:29:31.881948744 +0100 +@@ -49,7 +49,7 @@ + value: {{ .Values.proxy.no_proxy | quote }} + {{- end }} + args: +- - --v=5 ++ - --v={{ .Values.logLevel }} + - --leader-election=false + {{- if .Values.imagePullSecrets }} + imagePullSecrets: +diff -rtubN aws-ebs-csi-driver.orig/values.yaml aws-ebs-csi-driver/values.yaml +--- aws-ebs-csi-driver.orig/values.yaml 2021-03-05 03:10:41.000000000 +0100 ++++ aws-ebs-csi-driver/values.yaml 2021-03-05 10:29:31.881948744 +0100 +@@ -9,6 +9,8 @@ + tag: "v0.9.0" + pullPolicy: IfNotPresent + ++logLevel: 5 ++ + sidecars: + provisionerImage: + repository: k8s.gcr.io/sig-storage/csi-provisioner diff --git a/charts/kubezero-aws-ebs-csi-driver/templates/storage-class.yaml b/charts/kubezero-aws-ebs-csi-driver/templates/storage-class.yaml deleted file mode 100644 index 094d43a..0000000 --- a/charts/kubezero-aws-ebs-csi-driver/templates/storage-class.yaml +++ /dev/null @@ -1,61 +0,0 @@ -{{- if .Values.StorageClass.create }} -kind: StorageClass -apiVersion: storage.k8s.io/v1 -metadata: - name: ebs-sc-gp2-xfs - labels: -{{ include "kubezero-lib.labels" . | indent 4 }} -provisioner: ebs.csi.aws.com -volumeBindingMode: WaitForFirstConsumer -parameters: - csi.storage.k8s.io/fstype: xfs - type: gp2 - encrypted: "true" -{{- if index .Values "aws-ebs-csi-driver" "enableVolumeResizing" }} -allowVolumeExpansion: true -{{- end }} ---- -kind: StorageClass -apiVersion: storage.k8s.io/v1 -metadata: - name: ebs-sc-gp3-xfs - labels: -{{ include "kubezero-lib.labels" . | indent 4 }} - {{- if .Values.StorageClass.default }} - annotations: - storageclass.kubernetes.io/is-default-class: "true" - {{- end }} -provisioner: ebs.csi.aws.com -volumeBindingMode: WaitForFirstConsumer -parameters: - csi.storage.k8s.io/fstype: xfs - type: gp3 - encrypted: "true" -{{- if index .Values "aws-ebs-csi-driver" "enableVolumeResizing" }} -allowVolumeExpansion: true -{{- end }} - -{{- range .Values.StorageClass.Zones }} ---- -kind: StorageClass -apiVersion: storage.k8s.io/v1 -metadata: - name: ebs-sc-gp2-xfs-{{ . }} - labels: -{{ include "kubezero-lib.labels" . | indent 4 }} -provisioner: ebs.csi.aws.com -volumeBindingMode: WaitForFirstConsumer -parameters: - csi.storage.k8s.io/fstype: xfs - type: gp2 - encrypted: "true" -{{- if index .Values "aws-ebs-csi-driver" "enableVolumeResizing" }} -allowVolumeExpansion: true -{{- end }} -allowedTopologies: -- matchLabelExpressions: - - key: failure-domain.beta.kubernetes.io/zone - values: - - {{ . }} -{{- end }} -{{- end }} diff --git a/charts/kubezero-aws-ebs-csi-driver/update.sh b/charts/kubezero-aws-ebs-csi-driver/update.sh new file mode 100755 index 0000000..e748fa3 --- /dev/null +++ b/charts/kubezero-aws-ebs-csi-driver/update.sh @@ -0,0 +1,8 @@ +#!/bin/bash + +VERSION=0.9.14 + +rm -rf charts/aws-ebs-csi-driver +curl -L -s -o - https://github.com/kubernetes-sigs/aws-ebs-csi-driver/releases/download/helm-chart-aws-ebs-csi-driver-${VERSION}/aws-ebs-csi-driver-${VERSION}.tgz | tar xfz - -C charts + +patch -d charts -i ../loglevel_leader.patch -p0 --no-backup-if-mismatch diff --git a/charts/kubezero-aws-ebs-csi-driver/values.yaml b/charts/kubezero-aws-ebs-csi-driver/values.yaml index 9f8bb5b..8519805 100644 --- a/charts/kubezero-aws-ebs-csi-driver/values.yaml +++ b/charts/kubezero-aws-ebs-csi-driver/values.yaml @@ -1,9 +1,10 @@ aws-ebs-csi-driver: replicaCount: 1 + logLevel: 1 enableVolumeScheduling: true - enableVolumeResizing: false - enableVolumeSnapshot: false + enableVolumeResizing: true + enableVolumeSnapshot: true nodeSelector: node-role.kubernetes.io/master: "" @@ -20,6 +21,28 @@ aws-ebs-csi-driver: extraVolumeTags: {} # Name: KubeZero-Cluster -StorageClass: - create: true - default: true + resources: + requests: + cpu: 10m + memory: 24Mi + limits: + # cpu: 50m + memory: 40Mi + + storageClasses: + - name: ebs-sc-gp2-xfs + volumeBindingMode: WaitForFirstConsumer + allowVolumeExpansion: true + parameters: + csi.storage.k8s.io/fstype: xfs + type: gp2 + encrypted: "true" + - name: ebs-sc-gp3-xfs + annotations: + storageclass.kubernetes.io/is-default-class: "true" + volumeBindingMode: WaitForFirstConsumer + allowVolumeExpansion: true + parameters: + csi.storage.k8s.io/fstype: xfs + type: gp3 + encrypted: "true" diff --git a/charts/kubezero-aws-efs-csi-driver/Chart.yaml b/charts/kubezero-aws-efs-csi-driver/Chart.yaml index 77435c9..d698900 100644 --- a/charts/kubezero-aws-efs-csi-driver/Chart.yaml +++ b/charts/kubezero-aws-efs-csi-driver/Chart.yaml @@ -1,9 +1,9 @@ apiVersion: v2 name: kubezero-aws-efs-csi-driver description: KubeZero Umbrella Chart for aws-efs-csi-driver -version: 0.2.0 -appVersion: 1.0.0 -kubeVersion: ">=1.16.0-0" +version: 0.3.2 +appVersion: 1.2.0 +kubeVersion: ">=1.18.0-0" home: https://kubezero.com icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png sources: @@ -20,7 +20,6 @@ dependencies: - name: kubezero-lib version: ">= 0.1.3" repository: https://zero-down-time.github.io/kubezero/ -# Once they properly update upstream -# - name: aws-ebs-csi-driver -# version: 1.0.0 -# repository: https://kubernetes-sigs.github.io/aws-efs-csi-driver + - name: aws-efs-csi-driver + version: 1.2.1 + # repository: https://kubernetes-sigs.github.io/aws-efs-csi-driver/ diff --git a/charts/kubezero-aws-efs-csi-driver/README.md b/charts/kubezero-aws-efs-csi-driver/README.md index ab875ff..0f393ce 100644 --- a/charts/kubezero-aws-efs-csi-driver/README.md +++ b/charts/kubezero-aws-efs-csi-driver/README.md @@ -1,6 +1,6 @@ # kubezero-aws-efs-csi-driver -![Version: 0.2.0](https://img.shields.io/badge/Version-0.2.0-informational?style=flat-square) ![AppVersion: 1.0.0](https://img.shields.io/badge/AppVersion-1.0.0-informational?style=flat-square) +![Version: 0.3.2](https://img.shields.io/badge/Version-0.3.2-informational?style=flat-square) ![AppVersion: 1.2.0](https://img.shields.io/badge/AppVersion-1.2.0-informational?style=flat-square) KubeZero Umbrella Chart for aws-efs-csi-driver @@ -19,10 +19,11 @@ KubeZero Umbrella Chart for aws-efs-csi-driver ## Requirements -Kubernetes: `>=1.16.0-0` +Kubernetes: `>=1.18.0-0` | Repository | Name | Version | |------------|------|---------| +| | aws-efs-csi-driver | 1.2.1 | | https://zero-down-time.github.io/kubezero/ | kubezero-lib | >= 0.1.3 | ## Storage Class @@ -38,4 +39,9 @@ Details also see: [Reserve PV](https://kubernetes.io/docs/concepts/storage/persi |-----|------|---------|-------------| | StorageClass.create | bool | `true` | | | StorageClass.default | bool | `false` | | -| aws-efs-csi-driver.nodeSelector | object | `{}` | | +| aws-efs-csi-driver.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key | string | `"node.kubernetes.io/csi.efs.fs"` | | +| aws-efs-csi-driver.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].operator | string | `"Exists"` | | +| aws-efs-csi-driver.logLevel | int | `1` | | +| aws-efs-csi-driver.resources.limits.memory | string | `"128Mi"` | | +| aws-efs-csi-driver.resources.requests.cpu | string | `"20m"` | | +| aws-efs-csi-driver.resources.requests.memory | string | `"64Mi"` | | diff --git a/charts/kubezero-aws-efs-csi-driver/affinity_resources.patch b/charts/kubezero-aws-efs-csi-driver/affinity_resources.patch new file mode 100644 index 0000000..7e9bd7e --- /dev/null +++ b/charts/kubezero-aws-efs-csi-driver/affinity_resources.patch @@ -0,0 +1,53 @@ +diff -tubNr charts/aws-efs-csi-driver/templates/node-daemonset.yaml charts/aws-efs-csi-driver.zdt/templates/node-daemonset.yaml +--- charts/aws-efs-csi-driver/templates/node-daemonset.yaml 2021-03-19 00:22:54.000000000 +0100 ++++ charts/aws-efs-csi-driver.zdt/templates/node-daemonset.yaml 2021-03-24 13:06:56.105762451 +0100 +@@ -40,15 +40,10 @@ + {{- with .Values.nodeSelector }} + {{- . | toYaml | nindent 8 }} + {{- end }} ++ {{- with .Values.affinity }} + affinity: +- nodeAffinity: +- requiredDuringSchedulingIgnoredDuringExecution: +- nodeSelectorTerms: +- - matchExpressions: +- - key: eks.amazonaws.com/compute-type +- operator: NotIn +- values: +- - fargate ++ {{- . | toYaml | nindent 8 }} ++ {{- end }} + hostNetwork: true + {{- if .Values.dnsPolicy }} + dnsPolicy: "{{ .Values.dnsPolicy }}" +@@ -99,6 +94,10 @@ + timeoutSeconds: 3 + periodSeconds: 2 + failureThreshold: 5 ++ {{- with .Values.resources }} ++ resources: ++ {{- . | toYaml | nindent 12 }} ++ {{- end }} + - name: csi-driver-registrar + image: {{ printf "%s:%s" .Values.sidecars.nodeDriverRegistrarImage.repository .Values.sidecars.nodeDriverRegistrarImage.tag }} + args: +diff -tubNr charts/aws-efs-csi-driver/values.yaml charts/aws-efs-csi-driver.zdt/values.yaml +--- charts/aws-efs-csi-driver/values.yaml 2021-03-19 00:22:54.000000000 +0100 ++++ charts/aws-efs-csi-driver.zdt/values.yaml 2021-03-24 13:06:56.105762451 +0100 +@@ -43,7 +43,15 @@ + + tolerations: [] + +-affinity: {} ++affinity: ++ nodeAffinity: ++ requiredDuringSchedulingIgnoredDuringExecution: ++ nodeSelectorTerms: ++ - matchExpressions: ++ - key: eks.amazonaws.com/compute-type ++ operator: NotIn ++ values: ++ - fargate + + node: + podAnnotations: {} diff --git a/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/.helmignore b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/.helmignore new file mode 100644 index 0000000..50af031 --- /dev/null +++ b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/.helmignore @@ -0,0 +1,22 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*~ +# Various IDEs +.project +.idea/ +*.tmproj +.vscode/ diff --git a/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/Chart.yaml b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/Chart.yaml index 1a6eaa5..b7d32af 100644 --- a/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/Chart.yaml +++ b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/Chart.yaml @@ -1,15 +1,18 @@ apiVersion: v1 -appVersion: "1.0.0" -name: aws-efs-csi-driver +appVersion: 1.2.0 description: A Helm chart for AWS EFS CSI Driver -version: 0.2.0 -kubeVersion: ">=1.14.0-0" home: https://github.com/kubernetes-sigs/aws-efs-csi-driver -sources: - - https://github.com/kubernetes-sigs/aws-efs-csi-driver keywords: - - aws - - efs - - csi +- aws +- efs +- csi +kubeVersion: '>=1.17.0-0' maintainers: - - name: leakingtapan +- name: leakingtapan + url: https://github.com/leakingtapan +- name: krmichel + url: https://github.com/krmichel +name: aws-efs-csi-driver +sources: +- https://github.com/kubernetes-sigs/aws-efs-csi-driver +version: 1.2.1 diff --git a/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/NOTES.txt b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/NOTES.txt index a0b1e08..66f4d12 100644 --- a/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/NOTES.txt +++ b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/NOTES.txt @@ -1,3 +1,3 @@ To verify that aws-efs-csi-driver has started, run: - kubectl get pod -n kube-system -l "app.kubernetes.io/name={{ include "aws-efs-csi-driver.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" + kubectl get pod -n {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "aws-efs-csi-driver.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" diff --git a/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/_helpers.tpl b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/_helpers.tpl index 6e661c3..f0a743d 100644 --- a/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/_helpers.tpl +++ b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/_helpers.tpl @@ -43,3 +43,14 @@ app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} {{- end }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end -}} + +{{/* +Create the name of the service account to use +*/}} +{{- define "aws-efs-csi-driver.serviceAccountName" -}} +{{- if .Values.serviceAccount.controller.create -}} + {{ default (include "aws-efs-csi-driver.fullname" .) .Values.serviceAccount.controller.name }} +{{- else -}} + {{ default "default" .Values.serviceAccount.controller.name }} +{{- end -}} +{{- end -}} diff --git a/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/controller-deployment.yaml b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/controller-deployment.yaml new file mode 100644 index 0000000..d1c9901 --- /dev/null +++ b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/controller-deployment.yaml @@ -0,0 +1,100 @@ +{{- if .Values.controller.create }} +# Controller Service +kind: Deployment +apiVersion: apps/v1 +metadata: + name: efs-csi-controller + labels: + app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: + app: efs-csi-controller + app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} + app.kubernetes.io/instance: {{ .Release.Name }} + template: + metadata: + labels: + app: efs-csi-controller + app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} + app.kubernetes.io/instance: {{ .Release.Name }} + {{- if .Values.node.podAnnotations }} + annotations: {{ toYaml .Values.node.podAnnotations | nindent 8 }} + {{- end }} + spec: + {{- if .Values.imagePullSecrets }} + imagePullSecrets: + {{- range .Values.imagePullSecrets }} + - name: {{ . }} + {{- end }} + {{- end }} + nodeSelector: + kubernetes.io/os: linux + {{- with .Values.nodeSelector }} + {{- . | toYaml | nindent 8 }} + {{- end }} + {{- if .Values.serviceAccount.controller.create }} + serviceAccountName: {{ include "aws-efs-csi-driver.serviceAccountName" . }} + {{- end }} + priorityClassName: system-cluster-critical + tolerations: + - operator: Exists + {{- with .Values.node.tolerations }} + {{- . | toYaml | nindent 8 }} + {{- end }} + containers: + - name: efs-plugin + securityContext: + privileged: true + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + imagePullPolicy: IfNotPresent + args: + - --endpoint=$(CSI_ENDPOINT) + - --logtostderr + - --v=5 + # Uncomment below line to allow access point root directory to be deleted by controller. + #- --delete-access-point-root-dir + env: + - name: CSI_ENDPOINT + value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/sockets/pluginproxy/ + ports: + - name: healthz + containerPort: 9808 + protocol: TCP + livenessProbe: + httpGet: + path: /healthz + port: healthz + initialDelaySeconds: 10 + timeoutSeconds: 3 + periodSeconds: 10 + failureThreshold: 5 + - name: csi-provisioner + image: {{ printf "%s:%s" .Values.sidecars.csiProvisionerImage.repository .Values.sidecars.csiProvisionerImage.tag }} + args: + - --csi-address=$(ADDRESS) + - --v=5 + - --feature-gates=Topology=true + - --leader-election + env: + - name: ADDRESS + value: /var/lib/csi/sockets/pluginproxy/csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/sockets/pluginproxy/ + - name: liveness-probe + image: {{ printf "%s:%s" .Values.sidecars.livenessProbeImage.repository .Values.sidecars.livenessProbeImage.tag }} + args: + - --csi-address=/csi/csi.sock + - --health-port=9808 + volumeMounts: + - name: socket-dir + mountPath: /csi + volumes: + - name: socket-dir + emptyDir: {} +{{- end }} diff --git a/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/controller-serviceaccount.yaml b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/controller-serviceaccount.yaml new file mode 100644 index 0000000..612713b --- /dev/null +++ b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/controller-serviceaccount.yaml @@ -0,0 +1,59 @@ +{{- if .Values.serviceAccount.controller.create -}} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "aws-efs-csi-driver.serviceAccountName" . }} + labels: + app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} + {{- with .Values.serviceAccount.controller.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +--- + +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: efs-csi-external-provisioner-role + labels: + app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} +rules: + - apiGroups: [""] + resources: ["persistentvolumes"] + verbs: ["get", "list", "watch", "create", "delete"] + - apiGroups: [""] + resources: ["persistentvolumeclaims"] + verbs: ["get", "list", "watch", "update"] + - apiGroups: ["storage.k8s.io"] + resources: ["storageclasses"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["events"] + verbs: ["list", "watch", "create"] + - apiGroups: ["storage.k8s.io"] + resources: ["csinodes"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["nodes"] + verbs: ["get", "list", "watch"] + - apiGroups: ["coordination.k8s.io"] + resources: ["leases"] + verbs: ["get", "watch", "list", "delete", "update", "create"] + +--- + +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: efs-csi-provisioner-binding + labels: + app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} +subjects: + - kind: ServiceAccount + name: {{ include "aws-efs-csi-driver.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +roleRef: + kind: ClusterRole + name: efs-csi-external-provisioner-role + apiGroup: rbac.authorization.k8s.io +{{- end }} diff --git a/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/csidriver.yaml b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/csidriver.yaml index 4dbf6f1..33247d5 100644 --- a/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/csidriver.yaml +++ b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/csidriver.yaml @@ -2,5 +2,9 @@ apiVersion: storage.k8s.io/v1beta1 kind: CSIDriver metadata: name: efs.csi.aws.com + annotations: + "helm.sh/hook": pre-install, pre-upgrade + "helm.sh/hook-delete-policy": before-hook-creation + "helm.sh/resource-policy": keep spec: attachRequired: false diff --git a/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/daemonset.yaml b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/node-daemonset.yaml similarity index 73% rename from charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/daemonset.yaml rename to charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/node-daemonset.yaml index bfe3496..a8cd9a7 100644 --- a/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/daemonset.yaml +++ b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/node-daemonset.yaml @@ -3,7 +3,8 @@ kind: DaemonSet apiVersion: apps/v1 metadata: name: efs-csi-node - namespace: kube-system + labels: + app.kubernetes.io/name: {{ include "aws-efs-csi-driver.name" . }} spec: selector: matchLabels: @@ -20,12 +21,37 @@ spec: annotations: {{ toYaml .Values.node.podAnnotations | nindent 8 }} {{- end }} spec: + {{- if .Values.hostAliases }} + hostAliases: + {{- range $k, $v := .Values.hostAliases }} + - ip: {{ $v.ip }} + hostnames: + - {{ $k }}.efs.{{ $v.region }}.amazonaws.com + {{- end }} + {{- end }} + {{- if .Values.imagePullSecrets }} + imagePullSecrets: + {{- range .Values.imagePullSecrets }} + - name: {{ . }} + {{- end }} + {{- end }} nodeSelector: beta.kubernetes.io/os: linux {{- with .Values.nodeSelector }} {{- . | toYaml | nindent 8 }} {{- end }} + {{- with .Values.affinity }} + affinity: + {{- . | toYaml | nindent 8 }} + {{- end }} hostNetwork: true + {{- if .Values.dnsPolicy }} + dnsPolicy: "{{ .Values.dnsPolicy }}" + {{- end }} + {{- with .Values.dnsConfig }} + dnsConfig: + {{- toYaml . | nindent 8 }} + {{- end }} priorityClassName: system-node-critical tolerations: - operator: Exists @@ -40,7 +66,7 @@ spec: args: - --endpoint=$(CSI_ENDPOINT) - --logtostderr - - --v=5 + - --v={{ .Values.logLevel }} env: - name: CSI_ENDPOINT value: unix:/csi/csi.sock @@ -53,7 +79,9 @@ spec: - name: efs-state-dir mountPath: /var/run/efs - name: efs-utils-config - mountPath: /etc/amazon/efs + mountPath: /var/amazon/efs + - name: efs-utils-config-legacy + mountPath: /etc/amazon/efs-legacy ports: - name: healthz containerPort: 9809 @@ -66,12 +94,16 @@ spec: timeoutSeconds: 3 periodSeconds: 2 failureThreshold: 5 - - name: cs-driver-registrar + {{- with .Values.resources }} + resources: + {{- . | toYaml | nindent 12 }} + {{- end }} + - name: csi-driver-registrar image: {{ printf "%s:%s" .Values.sidecars.nodeDriverRegistrarImage.repository .Values.sidecars.nodeDriverRegistrarImage.tag }} args: - --csi-address=$(ADDRESS) - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH) - - --v=5 + - --v={{ .Values.logLevel }} env: - name: ADDRESS value: /csi/csi.sock @@ -91,6 +123,7 @@ spec: args: - --csi-address=/csi/csi.sock - --health-port=9809 + - --v={{ .Values.logLevel }} volumeMounts: - name: plugin-dir mountPath: /csi @@ -112,6 +145,10 @@ spec: path: /var/run/efs type: DirectoryOrCreate - name: efs-utils-config + hostPath: + path: /var/amazon/efs + type: DirectoryOrCreate + - name: efs-utils-config-legacy hostPath: path: /etc/amazon/efs type: DirectoryOrCreate diff --git a/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/storageclass.yaml b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/storageclass.yaml new file mode 100644 index 0000000..39f1fae --- /dev/null +++ b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/templates/storageclass.yaml @@ -0,0 +1,15 @@ +{{- range .Values.storageClasses }} +kind: StorageClass +apiVersion: storage.k8s.io/v1 +metadata: + name: {{ .name }} +provisioner: efs.csi.aws.com +{{- with .mountOptions }} +mountOptions: +{{ toYaml . }} +{{- end }} +{{- with .parameters }} +parameters: +{{ toYaml . | indent 2 }} +{{- end }} +{{- end }} diff --git a/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/values.yaml b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/values.yaml index e583428..e1377c1 100644 --- a/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/values.yaml +++ b/charts/kubezero-aws-efs-csi-driver/charts/aws-efs-csi-driver/values.yaml @@ -6,16 +6,19 @@ replicaCount: 2 image: repository: amazon/aws-efs-csi-driver - tag: "v1.0.0" + tag: "v1.2.0" pullPolicy: IfNotPresent sidecars: livenessProbeImage: - repository: quay.io/k8scsi/livenessprobe - tag: "v2.0.0" + repository: public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe + tag: v2.1.0-eks-1-18-1 nodeDriverRegistrarImage: - repository: quay.io/k8scsi/csi-node-driver-registrar - tag: "v1.3.0" + repository: public.ecr.aws/eks-distro/kubernetes-csi/node-driver-registrar + tag: v2.0.1-eks-1-18-1 + csiProvisionerImage: + repository: public.ecr.aws/eks-distro/kubernetes-csi/external-provisioner + tag: v2.0.3-eks-1-18-1 imagePullSecrets: [] nameOverride: "" @@ -23,7 +26,8 @@ fullnameOverride: "" podAnnotations: {} -resources: {} +resources: + {} # We usually recommend not to specify default resources and to leave this as a conscious # choice for the user. This also increases chances charts run on environments with little # resources, such as Minikube. If you do want to specify resources, uncomment the following @@ -39,8 +43,63 @@ nodeSelector: {} tolerations: [] -affinity: {} +affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: eks.amazonaws.com/compute-type + operator: NotIn + values: + - fargate node: podAnnotations: {} tolerations: [] + +logLevel: 5 + +hostAliases: + {} + # for cross VPC EFS, you need to poison or overwrite the DNS for the efs volume as per + # https://docs.aws.amazon.com/efs/latest/ug/efs-different-vpc.html#wt6-efs-utils-step3 + # implementing the suggested solution found here: + # https://github.com/kubernetes-sigs/aws-efs-csi-driver/issues/240#issuecomment-676849346 + # EFS Vol ID, IP, Region + # "fs-01234567": + # ip: 10.10.2.2 + # region: us-east-2 + +dnsPolicy: "" +dnsConfig: + {} + # Example config which uses the AWS nameservers + # dnsPolicy: "None" + # dnsConfig: + # nameservers: + # - 169.254.169.253 + +serviceAccount: + controller: + # Specifies whether a service account should be created + create: true + annotations: {} + ## Enable if EKS IAM for SA is used + # eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/efs-csi-role + name: efs-csi-controller-sa + +controller: + create: true + +storageClasses: [] +# Add StorageClass resources like: +# - name: efs-sc +# mountOptions: +# - tls +# parameters: +# provisioningMode: efs-ap +# fileSystemId: fs-92107410 +# directoryPerms: "700" +# gidRangeStart: "1000" +# gidRangeEnd: "2000" +# basePath: "/dynamic_provisioning" diff --git a/charts/kubezero-aws-efs-csi-driver/update.sh b/charts/kubezero-aws-efs-csi-driver/update.sh index c35e274..bc60b1b 100755 --- a/charts/kubezero-aws-efs-csi-driver/update.sh +++ b/charts/kubezero-aws-efs-csi-driver/update.sh @@ -1,14 +1,8 @@ #!/bin/bash -set -ex -# Upstream doesnt have proper Helm repo yet so we just download latest release and stuff it into charts +VERSION=1.2.1 -REPO="kubernetes-sigs/aws-efs-csi-driver" -[ -z "$RELEASE" ] && RELEASE=$(curl -sL -s https://api.github.com/repos/${REPO}/releases | grep '"tag_name":' | cut -d'"' -f4 | grep -v -E "(alpha|beta|rc)" | sort -t"." -k 1,1 -k 2,2 -k 3,3 -k 4,4 | tail -n 1) +rm -rf charts/aws-efs-csi-driver +curl -L -s -o - https://github.com/kubernetes-sigs/aws-efs-csi-driver/releases/download/helm-chart-aws-efs-csi-driver-${VERSION}/aws-efs-csi-driver-${VERSION}.tgz | tar xfz - -C charts -rm -rf git -git clone https://github.com/$REPO.git git -cd git && git checkout $RELEASE && cd - - -rm -rf charts/aws-efs-csi-driver && mkdir -p charts/aws-efs-csi-driver -mv git/helm/* charts/aws-efs-csi-driver +patch -i affinity_resources.patch -p0 --no-backup-if-mismatch diff --git a/charts/kubezero-aws-efs-csi-driver/values.yaml b/charts/kubezero-aws-efs-csi-driver/values.yaml index e02877c..32342bf 100644 --- a/charts/kubezero-aws-efs-csi-driver/values.yaml +++ b/charts/kubezero-aws-efs-csi-driver/values.yaml @@ -1,6 +1,20 @@ aws-efs-csi-driver: - nodeSelector: {} - # node.kubernetes.io/csi.efs.fs: EFS-FS_ID + logLevel: 1 + + resources: + requests: + cpu: 20m + memory: 64Mi + limits: + memory: 128Mi + + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node.kubernetes.io/csi.efs.fs + operator: Exists StorageClass: create: true diff --git a/charts/kubezero-aws-node-termination-handler/Chart.yaml b/charts/kubezero-aws-node-termination-handler/Chart.yaml new file mode 100644 index 0000000..4dbf3b0 --- /dev/null +++ b/charts/kubezero-aws-node-termination-handler/Chart.yaml @@ -0,0 +1,20 @@ +apiVersion: v2 +name: kubezero-aws-node-termination-handler +description: Umbrella chart for all aws-node-termination-handler +type: application +version: 0.1.2 +home: https://kubezero.com +icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png +keywords: + - kubezero + - aws-node-termination-handler +maintainers: + - name: Quarky9 +dependencies: + - name: kubezero-lib + version: ">= 0.1.3" + repository: https://zero-down-time.github.io/kubezero/ + - name: aws-node-termination-handler + version: ">= 0.14.1" + repository: https://aws.github.io/eks-charts +kubeVersion: ">= 1.18.0" diff --git a/charts/kubezero-aws-node-termination-handler/README.md b/charts/kubezero-aws-node-termination-handler/README.md new file mode 100644 index 0000000..866a7be --- /dev/null +++ b/charts/kubezero-aws-node-termination-handler/README.md @@ -0,0 +1,48 @@ +# kubezero-aws-node-termination-handler + +![Version: 0.1.2](https://img.shields.io/badge/Version-0.1.2-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) + +Umbrella chart for all aws-node-termination-handler + +**Homepage:** + +## Maintainers + +| Name | Email | Url | +| ---- | ------ | --- | +| Quarky9 | | | + +## Requirements + +Kubernetes: `>= 1.18.0` + +| Repository | Name | Version | +|------------|------|---------| +| https://aws.github.io/eks-charts | aws-node-termination-handler | >= 0.14.1 | +| https://zero-down-time.github.io/kubezero/ | kubezero-lib | >= 0.1.3 | + +## Values + +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| aws-node-termination-handler.deleteLocalData | bool | `true` | | +| aws-node-termination-handler.enablePrometheusServer | bool | `false` | | +| aws-node-termination-handler.enableSqsTerminationDraining | bool | `true` | | +| aws-node-termination-handler.jsonLogging | bool | `true` | | +| aws-node-termination-handler.nodeSelector."node-role.kubernetes.io/master" | string | `""` | | +| aws-node-termination-handler.podMonitor.create | bool | `false` | | +| aws-node-termination-handler.podMonitor.labels.release | string | `"metrics"` | | +| aws-node-termination-handler.taintNode | bool | `true` | | +| aws-node-termination-handler.tolerations[0].effect | string | `"NoSchedule"` | | +| aws-node-termination-handler.tolerations[0].key | string | `"node-role.kubernetes.io/master"` | | + +## KubeZero default configuration + +- enable SQS Mode +- allow draining of localdata +- enable prometheus + +## Resources + +- https://github.com/aws/aws-node-termination-handler +- https://github.com/aws/eks-charts/tree/master/stable/aws-node-termination-handler diff --git a/charts/kubezero-aws-node-termination-handler/README.md.gotmpl b/charts/kubezero-aws-node-termination-handler/README.md.gotmpl new file mode 100644 index 0000000..c236518 --- /dev/null +++ b/charts/kubezero-aws-node-termination-handler/README.md.gotmpl @@ -0,0 +1,27 @@ +{{ template "chart.header" . }} +{{ template "chart.deprecationWarning" . }} + +{{ template "chart.versionBadge" . }}{{ template "chart.typeBadge" . }}{{ template "chart.appVersionBadge" . }} + +{{ template "chart.description" . }} + +{{ template "chart.homepageLine" . }} + +{{ template "chart.maintainersSection" . }} + +{{ template "chart.sourcesSection" . }} + +{{ template "chart.requirementsSection" . }} + +{{ template "chart.valuesSection" . }} + +## KubeZero default configuration + +- enable SQS Mode +- allow draining of localdata +- enable prometheus + +## Resources + +- https://github.com/aws/aws-node-termination-handler +- https://github.com/aws/eks-charts/tree/master/stable/aws-node-termination-handler diff --git a/charts/kubezero-aws-node-termination-handler/values.yaml b/charts/kubezero-aws-node-termination-handler/values.yaml new file mode 100644 index 0000000..02e5a7d --- /dev/null +++ b/charts/kubezero-aws-node-termination-handler/values.yaml @@ -0,0 +1,19 @@ +aws-node-termination-handler: + enableSqsTerminationDraining: true + # queueURL: + deleteLocalData: true + taintNode: true + + enablePrometheusServer: false + podMonitor: + create: false + labels: + release: metrics + + jsonLogging: true + + tolerations: + - key: node-role.kubernetes.io/master + effect: NoSchedule + nodeSelector: + node-role.kubernetes.io/master: "" diff --git a/charts/kubezero-calico/Chart.yaml b/charts/kubezero-calico/Chart.yaml index f346a5f..bb95e27 100644 --- a/charts/kubezero-calico/Chart.yaml +++ b/charts/kubezero-calico/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: kubezero-calico description: KubeZero Umbrella Chart for Calico type: application -version: 0.2.1 +version: 0.2.2 appVersion: v3.16.5 home: https://kubezero.com icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png diff --git a/charts/kubezero-calico/README.md b/charts/kubezero-calico/README.md index e660c3b..061455d 100644 --- a/charts/kubezero-calico/README.md +++ b/charts/kubezero-calico/README.md @@ -1,6 +1,6 @@ # kubezero-calico -![Version: 0.2.1](https://img.shields.io/badge/Version-0.2.1-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v3.16.5](https://img.shields.io/badge/AppVersion-v3.16.5-informational?style=flat-square) +![Version: 0.2.2](https://img.shields.io/badge/Version-0.2.2-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v3.16.5](https://img.shields.io/badge/AppVersion-v3.16.5-informational?style=flat-square) KubeZero Umbrella Chart for Calico diff --git a/charts/kubezero-calico/templates/calico.yaml b/charts/kubezero-calico/templates/calico.yaml index 9946764..f61b003 100644 --- a/charts/kubezero-calico/templates/calico.yaml +++ b/charts/kubezero-calico/templates/calico.yaml @@ -482,7 +482,7 @@ spec: privileged: true resources: requests: - cpu: 250m + cpu: 50m livenessProbe: exec: command: @@ -619,6 +619,9 @@ spec: command: - /usr/bin/check-status - -r + resources: + requests: + cpu: 50m --- diff --git a/charts/kubezero-cert-manager/Chart.yaml b/charts/kubezero-cert-manager/Chart.yaml index 72e6898..bb8ab08 100644 --- a/charts/kubezero-cert-manager/Chart.yaml +++ b/charts/kubezero-cert-manager/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: kubezero-cert-manager description: KubeZero Umbrella Chart for cert-manager type: application -version: 0.4.1 +version: 0.5.0 home: https://kubezero.com icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png keywords: @@ -15,7 +15,7 @@ dependencies: version: ">= 0.1.3" repository: https://zero-down-time.github.io/kubezero/ - name: cert-manager - version: 1.1.0 + version: 1.2.0 repository: https://charts.jetstack.io condition: cert-manager.enabled -kubeVersion: ">= 1.16.0" +kubeVersion: ">= 1.18.0" diff --git a/charts/kubezero-cert-manager/README.md b/charts/kubezero-cert-manager/README.md index 9a15eae..bfab896 100644 --- a/charts/kubezero-cert-manager/README.md +++ b/charts/kubezero-cert-manager/README.md @@ -1,6 +1,6 @@ # kubezero-cert-manager -![Version: 0.4.1](https://img.shields.io/badge/Version-0.4.1-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) +![Version: 0.5.0](https://img.shields.io/badge/Version-0.5.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) KubeZero Umbrella Chart for cert-manager @@ -14,11 +14,11 @@ KubeZero Umbrella Chart for cert-manager ## Requirements -Kubernetes: `>= 1.16.0` +Kubernetes: `>= 1.18.0` | Repository | Name | Version | |------------|------|---------| -| https://charts.jetstack.io | cert-manager | 1.1.0 | +| https://charts.jetstack.io | cert-manager | 1.2.0 | | https://zero-down-time.github.io/kubezero/ | kubezero-lib | >= 0.1.3 | ## AWS - IAM Role diff --git a/charts/kubezero-istio-ingress/Chart.yaml b/charts/kubezero-istio-ingress/Chart.yaml index e66651e..f5aa064 100644 --- a/charts/kubezero-istio-ingress/Chart.yaml +++ b/charts/kubezero-istio-ingress/Chart.yaml @@ -2,8 +2,8 @@ apiVersion: v2 name: kubezero-istio-ingress description: KubeZero Umbrella Chart for Istio based Ingress type: application -version: 0.1.1 -appVersion: 1.8.1 +version: 0.5.0 +appVersion: 1.9.1 home: https://kubezero.com icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png keywords: @@ -16,9 +16,9 @@ dependencies: version: ">= 0.1.3" repository: https://zero-down-time.github.io/kubezero/ - name: istio-ingress - version: 1.1.0 + version: 1.9.1 condition: istio-ingress.enabled - name: istio-private-ingress - version: 1.1.0 + version: 1.9.1 condition: istio-private-ingress.enabled -kubeVersion: ">= 1.16.0" +kubeVersion: ">= 1.18.0" diff --git a/charts/kubezero-istio-ingress/README.md b/charts/kubezero-istio-ingress/README.md index e7557d3..8c7b0c4 100644 --- a/charts/kubezero-istio-ingress/README.md +++ b/charts/kubezero-istio-ingress/README.md @@ -1,6 +1,6 @@ # kubezero-istio-ingress -![Version: 0.1.1](https://img.shields.io/badge/Version-0.1.1-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.8.1](https://img.shields.io/badge/AppVersion-1.8.1-informational?style=flat-square) +![Version: 0.5.0](https://img.shields.io/badge/Version-0.5.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.9.1](https://img.shields.io/badge/AppVersion-1.9.1-informational?style=flat-square) KubeZero Umbrella Chart for Istio based Ingress @@ -16,12 +16,12 @@ Installs Istio Ingress Gateways, requires kubezero-istio to be installed ! ## Requirements -Kubernetes: `>= 1.16.0` +Kubernetes: `>= 1.18.0` | Repository | Name | Version | |------------|------|---------| -| | istio-ingress | 1.1.0 | -| | istio-private-ingress | 1.1.0 | +| | istio-ingress | 1.9.1 | +| | istio-private-ingress | 1.9.1 | | https://zero-down-time.github.io/kubezero/ | kubezero-lib | >= 0.1.3 | ## Values @@ -30,11 +30,9 @@ Kubernetes: `>= 1.16.0` |-----|------|---------|-------------| | global.arch.amd64 | int | `2` | | | global.defaultPodDisruptionBudget.enabled | bool | `false` | | -| global.hub | string | `"docker.io/istio"` | | | global.jwtPolicy | string | `"first-party-jwt"` | | | global.logAsJson | bool | `true` | | | global.priorityClassName | string | `"system-cluster-critical"` | | -| global.tag | string | `"1.8.1"` | | | istio-ingress.dnsNames | list | `[]` | | | istio-ingress.enabled | bool | `false` | | | istio-ingress.gateways.istio-ingressgateway.autoscaleEnabled | bool | `false` | | @@ -45,16 +43,19 @@ Kubernetes: `>= 1.16.0` | istio-ingress.gateways.istio-ingressgateway.podAntiAffinityLabelSelector[0].operator | string | `"In"` | | | istio-ingress.gateways.istio-ingressgateway.podAntiAffinityLabelSelector[0].topologyKey | string | `"kubernetes.io/hostname"` | | | istio-ingress.gateways.istio-ingressgateway.podAntiAffinityLabelSelector[0].values | string | `"istio-ingressgateway"` | | -| istio-ingress.gateways.istio-ingressgateway.ports[0].name | string | `"http-status"` | | +| istio-ingress.gateways.istio-ingressgateway.ports[0].name | string | `"status-port"` | | | istio-ingress.gateways.istio-ingressgateway.ports[0].nodePort | int | `30021` | | | istio-ingress.gateways.istio-ingressgateway.ports[0].port | int | `15021` | | +| istio-ingress.gateways.istio-ingressgateway.ports[0].protocol | string | `"TCP"` | | | istio-ingress.gateways.istio-ingressgateway.ports[1].name | string | `"http2"` | | | istio-ingress.gateways.istio-ingressgateway.ports[1].nodePort | int | `30080` | | | istio-ingress.gateways.istio-ingressgateway.ports[1].port | int | `80` | | +| istio-ingress.gateways.istio-ingressgateway.ports[1].protocol | string | `"TCP"` | | | istio-ingress.gateways.istio-ingressgateway.ports[1].targetPort | int | `8080` | | | istio-ingress.gateways.istio-ingressgateway.ports[2].name | string | `"https"` | | | istio-ingress.gateways.istio-ingressgateway.ports[2].nodePort | int | `30443` | | | istio-ingress.gateways.istio-ingressgateway.ports[2].port | int | `443` | | +| istio-ingress.gateways.istio-ingressgateway.ports[2].protocol | string | `"TCP"` | | | istio-ingress.gateways.istio-ingressgateway.ports[2].targetPort | int | `8443` | | | istio-ingress.gateways.istio-ingressgateway.replicaCount | int | `1` | | | istio-ingress.gateways.istio-ingressgateway.resources.limits.memory | string | `"256Mi"` | | @@ -73,17 +74,30 @@ Kubernetes: `>= 1.16.0` | istio-private-ingress.gateways.istio-ingressgateway.podAntiAffinityLabelSelector[0].operator | string | `"In"` | | | istio-private-ingress.gateways.istio-ingressgateway.podAntiAffinityLabelSelector[0].topologyKey | string | `"kubernetes.io/hostname"` | | | istio-private-ingress.gateways.istio-ingressgateway.podAntiAffinityLabelSelector[0].values | string | `"istio-private-ingressgateway"` | | -| istio-private-ingress.gateways.istio-ingressgateway.ports[0].name | string | `"http-status"` | | +| istio-private-ingress.gateways.istio-ingressgateway.ports[0].name | string | `"status-port"` | | | istio-private-ingress.gateways.istio-ingressgateway.ports[0].nodePort | int | `31021` | | | istio-private-ingress.gateways.istio-ingressgateway.ports[0].port | int | `15021` | | +| istio-private-ingress.gateways.istio-ingressgateway.ports[0].protocol | string | `"TCP"` | | | istio-private-ingress.gateways.istio-ingressgateway.ports[1].name | string | `"http2"` | | | istio-private-ingress.gateways.istio-ingressgateway.ports[1].nodePort | int | `31080` | | | istio-private-ingress.gateways.istio-ingressgateway.ports[1].port | int | `80` | | +| istio-private-ingress.gateways.istio-ingressgateway.ports[1].protocol | string | `"TCP"` | | | istio-private-ingress.gateways.istio-ingressgateway.ports[1].targetPort | int | `8080` | | | istio-private-ingress.gateways.istio-ingressgateway.ports[2].name | string | `"https"` | | | istio-private-ingress.gateways.istio-ingressgateway.ports[2].nodePort | int | `31443` | | | istio-private-ingress.gateways.istio-ingressgateway.ports[2].port | int | `443` | | +| istio-private-ingress.gateways.istio-ingressgateway.ports[2].protocol | string | `"TCP"` | | | istio-private-ingress.gateways.istio-ingressgateway.ports[2].targetPort | int | `8443` | | +| istio-private-ingress.gateways.istio-ingressgateway.ports[3].name | string | `"tcp-istiod"` | | +| istio-private-ingress.gateways.istio-ingressgateway.ports[3].nodePort | int | `31012` | | +| istio-private-ingress.gateways.istio-ingressgateway.ports[3].port | int | `15012` | | +| istio-private-ingress.gateways.istio-ingressgateway.ports[3].protocol | string | `"TCP"` | | +| istio-private-ingress.gateways.istio-ingressgateway.ports[3].targetPort | int | `15012` | | +| istio-private-ingress.gateways.istio-ingressgateway.ports[4].name | string | `"tls"` | | +| istio-private-ingress.gateways.istio-ingressgateway.ports[4].nodePort | int | `31044` | | +| istio-private-ingress.gateways.istio-ingressgateway.ports[4].port | int | `15443` | | +| istio-private-ingress.gateways.istio-ingressgateway.ports[4].protocol | string | `"TCP"` | | +| istio-private-ingress.gateways.istio-ingressgateway.ports[4].targetPort | int | `15443` | | | istio-private-ingress.gateways.istio-ingressgateway.replicaCount | int | `1` | | | istio-private-ingress.gateways.istio-ingressgateway.resources.limits.memory | string | `"256Mi"` | | | istio-private-ingress.gateways.istio-ingressgateway.resources.requests.cpu | string | `"100m"` | | diff --git a/charts/kubezero-istio-ingress/charts/istio-ingress/Chart.yaml b/charts/kubezero-istio-ingress/charts/istio-ingress/Chart.yaml index 108c8f2..d402a1f 100644 --- a/charts/kubezero-istio-ingress/charts/istio-ingress/Chart.yaml +++ b/charts/kubezero-istio-ingress/charts/istio-ingress/Chart.yaml @@ -1,6 +1,6 @@ apiVersion: v1 name: istio-ingress -version: 1.1.0 +version: 1.9.1 tillerVersion: ">=2.7.2" description: Helm chart for deploying Istio gateways keywords: diff --git a/charts/kubezero-istio-ingress/charts/istio-ingress/templates/_affinity.tpl b/charts/kubezero-istio-ingress/charts/istio-ingress/templates/_affinity.tpl index 400ff54..cb6a91c 100644 --- a/charts/kubezero-istio-ingress/charts/istio-ingress/templates/_affinity.tpl +++ b/charts/kubezero-istio-ingress/charts/istio-ingress/templates/_affinity.tpl @@ -1,11 +1,11 @@ {{/* affinity - https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ */}} -{{- define "nodeaffinity" }} - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - {{- include "nodeAffinityRequiredDuringScheduling" . }} - preferredDuringSchedulingIgnoredDuringExecution: - {{- include "nodeAffinityPreferredDuringScheduling" . }} +{{ define "nodeaffinity" }} +nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + {{- include "nodeAffinityRequiredDuringScheduling" . }} + preferredDuringSchedulingIgnoredDuringExecution: + {{- include "nodeAffinityPreferredDuringScheduling" . }} {{- end }} {{- define "nodeAffinityRequiredDuringScheduling" }} diff --git a/charts/kubezero-istio-ingress/charts/istio-ingress/templates/deployment.yaml b/charts/kubezero-istio-ingress/charts/istio-ingress/templates/deployment.yaml index b5137a4..0c32e2f 100644 --- a/charts/kubezero-istio-ingress/charts/istio-ingress/templates/deployment.yaml +++ b/charts/kubezero-istio-ingress/charts/istio-ingress/templates/deployment.yaml @@ -41,6 +41,7 @@ spec: istio.io/rev: {{ .Values.revision | default "default" }} install.operator.istio.io/owning-resource: {{ .Values.ownerName | default "unknown" }} operator.istio.io/component: "IngressGateways" + sidecar.istio.io/inject: "false" annotations: {{- if .Values.meshConfig.enablePrometheusMerge }} prometheus.io/port: "15020" @@ -101,12 +102,6 @@ spec: - containerPort: {{ $val.targetPort | default $val.port }} protocol: {{ $val.protocol | default "TCP" }} {{- end }} - {{- if $.Values.global.meshExpansion.enabled }} - {{- range $key, $val := $gateway.meshExpansionPorts }} - - containerPort: {{ $val.targetPort | default $val.port }} - protocol: {{ $val.protocol | default "TCP" }} - {{- end }} - {{- end }} - containerPort: 15090 protocol: TCP name: http-envoy-prom @@ -220,6 +215,10 @@ spec: - name: TRUST_DOMAIN value: "{{ .Values.global.trustDomain | default (index .Values.meshConfig "trustDomain") }}" {{- end }} + {{- if not $gateway.runAsRoot }} + - name: ISTIO_META_UNPRIVILEGED_POD + value: "true" + {{- end }} {{- range $key, $val := $gateway.env }} - name: {{ $key }} value: {{ $val }} @@ -228,10 +227,10 @@ spec: - name: {{ $key }} value: "{{ $value }}" {{- end }} - {{ $network_set := index $gateway.env "ISTIO_META_NETWORK" }} + {{- $network_set := index $gateway.env "ISTIO_META_NETWORK" }} {{- if and (not $network_set) .Values.global.network }} - name: ISTIO_META_NETWORK - value: {{ .Values.global.network }} + value: "{{ .Values.global.network }}" {{- end }} {{- if $gateway.podAnnotations }} - name: "ISTIO_METAJSON_ANNOTATIONS" @@ -254,8 +253,6 @@ spec: mountPath: /var/run/secrets/tokens readOnly: true {{- end }} - - name: gatewaysdsudspath - mountPath: /var/run/ingress_gateway {{- if .Values.global.mountMtlsCerts }} # Use the key and cert mounted to /etc/certs/ for the in-cluster mTLS communications. - name: istio-certs @@ -296,10 +293,18 @@ spec: - path: "annotations" fieldRef: fieldPath: metadata.annotations + - path: "cpu-limit" + resourceFieldRef: + containerName: istio-proxy + resource: limits.cpu + divisor: 1m + - path: "cpu-request" + resourceFieldRef: + containerName: istio-proxy + resource: requests.cpu + divisor: 1m - name: istio-envoy emptyDir: {} - - name: gatewaysdsudspath - emptyDir: {} - name: istio-data emptyDir: {} {{- if eq .Values.global.jwtPolicy "third-party-jwt" }} @@ -335,7 +340,7 @@ spec: optional: true {{- end }} affinity: - {{- include "nodeaffinity" (dict "global" .Values.global "nodeSelector" $gateway.nodeSelector) | indent 6 }} +{{ include "nodeaffinity" (dict "global" .Values.global "nodeSelector" $gateway.nodeSelector) | trim | indent 8 }} {{- include "podAntiAffinity" $gateway | indent 6 }} {{- if $gateway.tolerations }} tolerations: diff --git a/charts/kubezero-istio-ingress/charts/istio-ingress/templates/meshexpansion.yaml b/charts/kubezero-istio-ingress/charts/istio-ingress/templates/meshexpansion.yaml deleted file mode 100644 index 67c164b..0000000 --- a/charts/kubezero-istio-ingress/charts/istio-ingress/templates/meshexpansion.yaml +++ /dev/null @@ -1,79 +0,0 @@ -{{- if .Values.global.meshExpansion.enabled }} -apiVersion: networking.istio.io/v1alpha3 -kind: Gateway -metadata: - name: meshexpansion-gateway - namespace: {{ .Release.Namespace }} - labels: - release: {{ .Release.Name }} - istio.io/rev: {{ .Values.revision | default "default" }} - install.operator.istio.io/owning-resource: {{ .Values.ownerName | default "unknown" }} - operator.istio.io/component: "IngressGateways" -spec: - selector: - istio: ingressgateway - servers: - - port: - number: 15012 - protocol: TCP - name: tcp-istiod - hosts: - - "*" - - port: - number: 15017 - protocol: TCP - name: tcp-istiodwebhook - hosts: - - "*" ---- - -apiVersion: networking.istio.io/v1alpha3 -kind: VirtualService -metadata: - name: meshexpansion-vs-istiod - namespace: {{ .Release.Namespace }} - labels: - release: {{ .Release.Name }} -spec: - hosts: - - istiod.{{ .Values.global.istioNamespace }}.svc.{{ .Values.global.proxy.clusterDomain }} - gateways: - - meshexpansion-gateway - tcp: - - match: - - port: 15012 - route: - - destination: - host: istiod.{{ .Values.global.istioNamespace }}.svc.{{ .Values.global.proxy.clusterDomain }} - port: - number: 15012 - - match: - - port: 15017 - route: - - destination: - host: istiod.{{ .Release.Namespace }}.svc.{{ .Values.global.proxy.clusterDomain }} - port: - number: 443 ---- - -apiVersion: networking.istio.io/v1alpha3 -kind: DestinationRule -metadata: - name: meshexpansion-dr-istiod - namespace: {{ .Release.Namespace }} - labels: - release: {{ .Release.Name }} -spec: - host: istiod.{{ .Release.Namespace }}.svc.{{ .Values.global.proxy.clusterDomain }} - trafficPolicy: - portLevelSettings: - - port: - number: 15012 - tls: - mode: DISABLE - - port: - number: 15017 - tls: - mode: DISABLE - -{{- end }} diff --git a/charts/kubezero-istio-ingress/charts/istio-ingress/templates/service.yaml b/charts/kubezero-istio-ingress/charts/istio-ingress/templates/service.yaml index 237be95..0f9dbf2 100644 --- a/charts/kubezero-istio-ingress/charts/istio-ingress/templates/service.yaml +++ b/charts/kubezero-istio-ingress/charts/istio-ingress/templates/service.yaml @@ -38,14 +38,6 @@ spec: {{- end }} {{- end }} - {{- if $.Values.global.meshExpansion.enabled }} - {{- range $key, $val := $gateway.meshExpansionPorts }} - - - {{- range $pkey, $pval := $val }} - {{ $pkey}}: {{ $pval }} - {{- end }} - {{- end }} - {{- end }} {{ range $app := $gateway.ingressPorts }} - port: {{ $app.port }} diff --git a/charts/kubezero-istio-ingress/charts/istio-ingress/values.yaml b/charts/kubezero-istio-ingress/charts/istio-ingress/values.yaml index 9f4dee6..249b3a8 100644 --- a/charts/kubezero-istio-ingress/charts/istio-ingress/values.yaml +++ b/charts/kubezero-istio-ingress/charts/istio-ingress/values.yaml @@ -24,7 +24,11 @@ gateways: targetPort: 8443 name: https protocol: TCP - # This is the port where sni routing happens + - port: 15012 + targetPort: 15012 + name: tcp-istiod + protocol: TCP + # This is the port where sni routing happens - port: 15443 targetPort: 15443 name: tls @@ -66,18 +70,6 @@ gateways: podAnnotations: {} type: LoadBalancer #change to NodePort, ClusterIP or LoadBalancer if need be - #### MESH EXPANSION PORTS ######## - # Pilot and Citadel MTLS ports are enabled in gateway - but will only redirect - # to pilot/citadel if global.meshExpansion settings are enabled. - # Delete these ports if mesh expansion is not enabled, to avoid - # exposing unnecessary ports on the web. - # You can remove these ports if you are not using mesh expansion - meshExpansionPorts: - - port: 15012 - targetPort: 15012 - name: tcp-istiod - ####### end MESH EXPANSION PORTS ###### - ############## secretVolumes: - name: ingressgateway-certs @@ -99,7 +91,7 @@ gateways: # A gateway with this mode ensures that pilot generates an additional # set of clusters for internal services but without Istio mTLS, to # enable cross cluster routing. - ISTIO_META_ROUTER_MODE: "sni-dnat" + ISTIO_META_ROUTER_MODE: "standard" nodeSelector: {} tolerations: [] @@ -142,6 +134,12 @@ global: # . implies these objects are visible to only to sidecars in the same namespace, or if imported as a Sidecar.egress.host defaultConfigVisibilitySettings: [] + # Default node selector to be applied to all deployments so that all pods can be + # constrained to run a particular nodes. Each component can overwrite these default + # values by adding its node selector block in the relevant section below and setting + # the desired values. + defaultNodeSelector: {} + # enable pod disruption budget for the control plane, which is used to # ensure Istio control plane components are gradually upgraded or recovered. defaultPodDisruptionBudget: @@ -170,10 +168,10 @@ global: # Default hub for Istio images. # Releases are published to docker hub under 'istio' project. # Dev builds from prow are on gcr.io - hub: gcr.io/istio-testing + hub: docker.io/istio # Default tag for Istio images. - tag: latest + tag: 1.9.1 # Specify image pull policy if default behavior isn't desired. # Default behavior: latest images will be Always else IfNotPresent. @@ -206,14 +204,6 @@ global: logging: level: "default:info" - # If set to true, the pilot and citadel mtls will be exposed on the - # ingress gateway - meshExpansion: - enabled: false - # If set to true, the pilot and citadel mtls and the plain text pilot ports - # will be exposed on an internal gateway - useILB: false - # Kubernetes >=v1.11.0 will create two PriorityClass, including system-cluster-critical and # system-node-critical, it is better to configure this in order to make sure your Istio pods # will not be killed because of low priority class. @@ -318,11 +308,10 @@ global: servicePort: 0 # Deprecated, use meshConfig.trustDomain -# trustDomain: "" + trustDomain: "" meshConfig: enablePrometheusMerge: true -# trustDomain: "" defaultConfig: proxyMetadata: {} tracing: diff --git a/charts/kubezero-istio-ingress/charts/istio-private-ingress/Chart.yaml b/charts/kubezero-istio-ingress/charts/istio-private-ingress/Chart.yaml index 2cd775d..3786335 100644 --- a/charts/kubezero-istio-ingress/charts/istio-private-ingress/Chart.yaml +++ b/charts/kubezero-istio-ingress/charts/istio-private-ingress/Chart.yaml @@ -1,6 +1,6 @@ apiVersion: v1 name: istio-private-ingress -version: 1.1.0 +version: 1.9.1 tillerVersion: ">=2.7.2" description: Helm chart for deploying Istio gateways keywords: diff --git a/charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/_affinity.tpl b/charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/_affinity.tpl index 400ff54..cb6a91c 100644 --- a/charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/_affinity.tpl +++ b/charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/_affinity.tpl @@ -1,11 +1,11 @@ {{/* affinity - https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ */}} -{{- define "nodeaffinity" }} - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - {{- include "nodeAffinityRequiredDuringScheduling" . }} - preferredDuringSchedulingIgnoredDuringExecution: - {{- include "nodeAffinityPreferredDuringScheduling" . }} +{{ define "nodeaffinity" }} +nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + {{- include "nodeAffinityRequiredDuringScheduling" . }} + preferredDuringSchedulingIgnoredDuringExecution: + {{- include "nodeAffinityPreferredDuringScheduling" . }} {{- end }} {{- define "nodeAffinityRequiredDuringScheduling" }} diff --git a/charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/deployment.yaml b/charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/deployment.yaml index b5137a4..0c32e2f 100644 --- a/charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/deployment.yaml +++ b/charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/deployment.yaml @@ -41,6 +41,7 @@ spec: istio.io/rev: {{ .Values.revision | default "default" }} install.operator.istio.io/owning-resource: {{ .Values.ownerName | default "unknown" }} operator.istio.io/component: "IngressGateways" + sidecar.istio.io/inject: "false" annotations: {{- if .Values.meshConfig.enablePrometheusMerge }} prometheus.io/port: "15020" @@ -101,12 +102,6 @@ spec: - containerPort: {{ $val.targetPort | default $val.port }} protocol: {{ $val.protocol | default "TCP" }} {{- end }} - {{- if $.Values.global.meshExpansion.enabled }} - {{- range $key, $val := $gateway.meshExpansionPorts }} - - containerPort: {{ $val.targetPort | default $val.port }} - protocol: {{ $val.protocol | default "TCP" }} - {{- end }} - {{- end }} - containerPort: 15090 protocol: TCP name: http-envoy-prom @@ -220,6 +215,10 @@ spec: - name: TRUST_DOMAIN value: "{{ .Values.global.trustDomain | default (index .Values.meshConfig "trustDomain") }}" {{- end }} + {{- if not $gateway.runAsRoot }} + - name: ISTIO_META_UNPRIVILEGED_POD + value: "true" + {{- end }} {{- range $key, $val := $gateway.env }} - name: {{ $key }} value: {{ $val }} @@ -228,10 +227,10 @@ spec: - name: {{ $key }} value: "{{ $value }}" {{- end }} - {{ $network_set := index $gateway.env "ISTIO_META_NETWORK" }} + {{- $network_set := index $gateway.env "ISTIO_META_NETWORK" }} {{- if and (not $network_set) .Values.global.network }} - name: ISTIO_META_NETWORK - value: {{ .Values.global.network }} + value: "{{ .Values.global.network }}" {{- end }} {{- if $gateway.podAnnotations }} - name: "ISTIO_METAJSON_ANNOTATIONS" @@ -254,8 +253,6 @@ spec: mountPath: /var/run/secrets/tokens readOnly: true {{- end }} - - name: gatewaysdsudspath - mountPath: /var/run/ingress_gateway {{- if .Values.global.mountMtlsCerts }} # Use the key and cert mounted to /etc/certs/ for the in-cluster mTLS communications. - name: istio-certs @@ -296,10 +293,18 @@ spec: - path: "annotations" fieldRef: fieldPath: metadata.annotations + - path: "cpu-limit" + resourceFieldRef: + containerName: istio-proxy + resource: limits.cpu + divisor: 1m + - path: "cpu-request" + resourceFieldRef: + containerName: istio-proxy + resource: requests.cpu + divisor: 1m - name: istio-envoy emptyDir: {} - - name: gatewaysdsudspath - emptyDir: {} - name: istio-data emptyDir: {} {{- if eq .Values.global.jwtPolicy "third-party-jwt" }} @@ -335,7 +340,7 @@ spec: optional: true {{- end }} affinity: - {{- include "nodeaffinity" (dict "global" .Values.global "nodeSelector" $gateway.nodeSelector) | indent 6 }} +{{ include "nodeaffinity" (dict "global" .Values.global "nodeSelector" $gateway.nodeSelector) | trim | indent 8 }} {{- include "podAntiAffinity" $gateway | indent 6 }} {{- if $gateway.tolerations }} tolerations: diff --git a/charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/meshexpansion.yaml b/charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/meshexpansion.yaml deleted file mode 100644 index 67c164b..0000000 --- a/charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/meshexpansion.yaml +++ /dev/null @@ -1,79 +0,0 @@ -{{- if .Values.global.meshExpansion.enabled }} -apiVersion: networking.istio.io/v1alpha3 -kind: Gateway -metadata: - name: meshexpansion-gateway - namespace: {{ .Release.Namespace }} - labels: - release: {{ .Release.Name }} - istio.io/rev: {{ .Values.revision | default "default" }} - install.operator.istio.io/owning-resource: {{ .Values.ownerName | default "unknown" }} - operator.istio.io/component: "IngressGateways" -spec: - selector: - istio: ingressgateway - servers: - - port: - number: 15012 - protocol: TCP - name: tcp-istiod - hosts: - - "*" - - port: - number: 15017 - protocol: TCP - name: tcp-istiodwebhook - hosts: - - "*" ---- - -apiVersion: networking.istio.io/v1alpha3 -kind: VirtualService -metadata: - name: meshexpansion-vs-istiod - namespace: {{ .Release.Namespace }} - labels: - release: {{ .Release.Name }} -spec: - hosts: - - istiod.{{ .Values.global.istioNamespace }}.svc.{{ .Values.global.proxy.clusterDomain }} - gateways: - - meshexpansion-gateway - tcp: - - match: - - port: 15012 - route: - - destination: - host: istiod.{{ .Values.global.istioNamespace }}.svc.{{ .Values.global.proxy.clusterDomain }} - port: - number: 15012 - - match: - - port: 15017 - route: - - destination: - host: istiod.{{ .Release.Namespace }}.svc.{{ .Values.global.proxy.clusterDomain }} - port: - number: 443 ---- - -apiVersion: networking.istio.io/v1alpha3 -kind: DestinationRule -metadata: - name: meshexpansion-dr-istiod - namespace: {{ .Release.Namespace }} - labels: - release: {{ .Release.Name }} -spec: - host: istiod.{{ .Release.Namespace }}.svc.{{ .Values.global.proxy.clusterDomain }} - trafficPolicy: - portLevelSettings: - - port: - number: 15012 - tls: - mode: DISABLE - - port: - number: 15017 - tls: - mode: DISABLE - -{{- end }} diff --git a/charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/service.yaml b/charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/service.yaml index 237be95..0f9dbf2 100644 --- a/charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/service.yaml +++ b/charts/kubezero-istio-ingress/charts/istio-private-ingress/templates/service.yaml @@ -38,14 +38,6 @@ spec: {{- end }} {{- end }} - {{- if $.Values.global.meshExpansion.enabled }} - {{- range $key, $val := $gateway.meshExpansionPorts }} - - - {{- range $pkey, $pval := $val }} - {{ $pkey}}: {{ $pval }} - {{- end }} - {{- end }} - {{- end }} {{ range $app := $gateway.ingressPorts }} - port: {{ $app.port }} diff --git a/charts/kubezero-istio-ingress/charts/istio-private-ingress/values.yaml b/charts/kubezero-istio-ingress/charts/istio-private-ingress/values.yaml index 9f4dee6..249b3a8 100644 --- a/charts/kubezero-istio-ingress/charts/istio-private-ingress/values.yaml +++ b/charts/kubezero-istio-ingress/charts/istio-private-ingress/values.yaml @@ -24,7 +24,11 @@ gateways: targetPort: 8443 name: https protocol: TCP - # This is the port where sni routing happens + - port: 15012 + targetPort: 15012 + name: tcp-istiod + protocol: TCP + # This is the port where sni routing happens - port: 15443 targetPort: 15443 name: tls @@ -66,18 +70,6 @@ gateways: podAnnotations: {} type: LoadBalancer #change to NodePort, ClusterIP or LoadBalancer if need be - #### MESH EXPANSION PORTS ######## - # Pilot and Citadel MTLS ports are enabled in gateway - but will only redirect - # to pilot/citadel if global.meshExpansion settings are enabled. - # Delete these ports if mesh expansion is not enabled, to avoid - # exposing unnecessary ports on the web. - # You can remove these ports if you are not using mesh expansion - meshExpansionPorts: - - port: 15012 - targetPort: 15012 - name: tcp-istiod - ####### end MESH EXPANSION PORTS ###### - ############## secretVolumes: - name: ingressgateway-certs @@ -99,7 +91,7 @@ gateways: # A gateway with this mode ensures that pilot generates an additional # set of clusters for internal services but without Istio mTLS, to # enable cross cluster routing. - ISTIO_META_ROUTER_MODE: "sni-dnat" + ISTIO_META_ROUTER_MODE: "standard" nodeSelector: {} tolerations: [] @@ -142,6 +134,12 @@ global: # . implies these objects are visible to only to sidecars in the same namespace, or if imported as a Sidecar.egress.host defaultConfigVisibilitySettings: [] + # Default node selector to be applied to all deployments so that all pods can be + # constrained to run a particular nodes. Each component can overwrite these default + # values by adding its node selector block in the relevant section below and setting + # the desired values. + defaultNodeSelector: {} + # enable pod disruption budget for the control plane, which is used to # ensure Istio control plane components are gradually upgraded or recovered. defaultPodDisruptionBudget: @@ -170,10 +168,10 @@ global: # Default hub for Istio images. # Releases are published to docker hub under 'istio' project. # Dev builds from prow are on gcr.io - hub: gcr.io/istio-testing + hub: docker.io/istio # Default tag for Istio images. - tag: latest + tag: 1.9.1 # Specify image pull policy if default behavior isn't desired. # Default behavior: latest images will be Always else IfNotPresent. @@ -206,14 +204,6 @@ global: logging: level: "default:info" - # If set to true, the pilot and citadel mtls will be exposed on the - # ingress gateway - meshExpansion: - enabled: false - # If set to true, the pilot and citadel mtls and the plain text pilot ports - # will be exposed on an internal gateway - useILB: false - # Kubernetes >=v1.11.0 will create two PriorityClass, including system-cluster-critical and # system-node-critical, it is better to configure this in order to make sure your Istio pods # will not be killed because of low priority class. @@ -318,11 +308,10 @@ global: servicePort: 0 # Deprecated, use meshConfig.trustDomain -# trustDomain: "" + trustDomain: "" meshConfig: enablePrometheusMerge: true -# trustDomain: "" defaultConfig: proxyMetadata: {} tracing: diff --git a/charts/kubezero-istio-ingress/templates/ingress-gateway.yaml b/charts/kubezero-istio-ingress/templates/ingress-gateway.yaml index 4995ee0..cd35cb2 100644 --- a/charts/kubezero-istio-ingress/templates/ingress-gateway.yaml +++ b/charts/kubezero-istio-ingress/templates/ingress-gateway.yaml @@ -1,3 +1,5 @@ +# https://istio.io/latest/docs/tasks/traffic-management/ingress/secure-ingress/#configure-a-tls-ingress-gateway-for-multiple-hosts + {{- if and (index .Values "istio-ingress" "enabled") (index .Values "istio-ingress" "dnsNames") }} apiVersion: networking.istio.io/v1beta1 kind: Gateway diff --git a/charts/kubezero-istio-ingress/values.yaml b/charts/kubezero-istio-ingress/values.yaml index b989ba7..50a9aee 100644 --- a/charts/kubezero-istio-ingress/values.yaml +++ b/charts/kubezero-istio-ingress/values.yaml @@ -1,7 +1,7 @@ # Make sure these values match kuberzero-istio !!! global: - hub: docker.io/istio - tag: 1.8.1 + #hub: docker.io/istio + #tag: 1.9.0 logAsJson: true jwtPolicy: first-party-jwt @@ -42,17 +42,32 @@ istio-ingress: nodeSelector: node.kubernetes.io/ingress.public: "30080_30443" ports: - - name: http-status + - name: status-port port: 15021 nodePort: 30021 + protocol: TCP - name: http2 port: 80 targetPort: 8080 nodePort: 30080 + protocol: TCP - name: https port: 443 targetPort: 8443 nodePort: 30443 + protocol: TCP + ## multi-cluster - disabled on public LBs + #- name: tcp-istiod + # port: 15012 + # targetPort: 15012 + # nodePort: 30012 + # protocol: TCP + ## multi-cluster sni east-west + #- name: tls + # port: 15443 + # targetPort: 15443 + # nodePort: 30044 + # protocol: TCP dnsNames: [] # - '*.example.com' @@ -93,17 +108,32 @@ istio-private-ingress: #nodeSelector: "31080_31443_31671_31672_31224" ports: - - name: http-status + - name: status-port port: 15021 nodePort: 31021 + protocol: TCP - name: http2 port: 80 targetPort: 8080 nodePort: 31080 + protocol: TCP - name: https port: 443 targetPort: 8443 nodePort: 31443 + protocol: TCP + # multi-cluster + - name: tcp-istiod + port: 15012 + targetPort: 15012 + nodePort: 31012 + protocol: TCP + # multi-cluster sni east-west + - name: tls + port: 15443 + targetPort: 15443 + nodePort: 31044 + protocol: TCP #- name: fluentd-forward # port: 24224 # nodePort: 31224 diff --git a/charts/kubezero-istio/Chart.yaml b/charts/kubezero-istio/Chart.yaml index 903bd5a..bae339a 100644 --- a/charts/kubezero-istio/Chart.yaml +++ b/charts/kubezero-istio/Chart.yaml @@ -2,8 +2,8 @@ apiVersion: v2 name: kubezero-istio description: KubeZero Umbrella Chart for Istio type: application -version: 0.4.1 -appVersion: 1.8.1 +version: 0.5.0 +appVersion: 1.9.1 home: https://kubezero.com icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png keywords: @@ -16,7 +16,7 @@ dependencies: version: ">= 0.1.3" repository: https://zero-down-time.github.io/kubezero/ - name: base - version: 1.1.0 + version: 1.9.1 - name: istio-discovery - version: 1.2.0 -kubeVersion: ">= 1.16.0" + version: 1.9.1 +kubeVersion: ">= 1.18.0" diff --git a/charts/kubezero-istio/README.md b/charts/kubezero-istio/README.md index 849a5b4..3fd4c4c 100644 --- a/charts/kubezero-istio/README.md +++ b/charts/kubezero-istio/README.md @@ -1,6 +1,6 @@ # kubezero-istio -![Version: 0.4.1](https://img.shields.io/badge/Version-0.4.1-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.8.1](https://img.shields.io/badge/AppVersion-1.8.1-informational?style=flat-square) +![Version: 0.5.0](https://img.shields.io/badge/Version-0.5.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.9.1](https://img.shields.io/badge/AppVersion-1.9.1-informational?style=flat-square) KubeZero Umbrella Chart for Istio @@ -16,12 +16,12 @@ Installs the Istio control plane ## Requirements -Kubernetes: `>= 1.16.0` +Kubernetes: `>= 1.18.0` | Repository | Name | Version | |------------|------|---------| -| | base | 1.1.0 | -| | istio-discovery | 1.2.0 | +| | base | 1.9.1 | +| | istio-discovery | 1.9.1 | | https://zero-down-time.github.io/kubezero/ | kubezero-lib | >= 0.1.3 | ## Values @@ -29,11 +29,9 @@ Kubernetes: `>= 1.16.0` | Key | Type | Default | Description | |-----|------|---------|-------------| | global.defaultPodDisruptionBudget.enabled | bool | `false` | | -| global.hub | string | `"docker.io/istio"` | | | global.jwtPolicy | string | `"first-party-jwt"` | | | global.logAsJson | bool | `true` | | | global.priorityClassName | string | `"system-cluster-critical"` | | -| global.tag | string | `"1.8.0"` | | | istio-discovery.meshConfig.accessLogEncoding | string | `"JSON"` | | | istio-discovery.meshConfig.accessLogFile | string | `"/dev/stdout"` | | | istio-discovery.meshConfig.h2UpgradePolicy | string | `"DO_NOT_UPGRADE"` | | diff --git a/charts/kubezero-istio/charts/base/Chart.yaml b/charts/kubezero-istio/charts/base/Chart.yaml index e012727..e6142de 100644 --- a/charts/kubezero-istio/charts/base/Chart.yaml +++ b/charts/kubezero-istio/charts/base/Chart.yaml @@ -1,6 +1,6 @@ apiVersion: v1 name: base -version: 1.1.0 +version: 1.9.1 tillerVersion: ">=2.7.2" description: Helm chart for deploying Istio cluster resources and CRDs keywords: diff --git a/charts/kubezero-istio/charts/base/crds/crd-all.gen.yaml b/charts/kubezero-istio/charts/base/crds/crd-all.gen.yaml index deea651..faddc02 100644 --- a/charts/kubezero-istio/charts/base/crds/crd-all.gen.yaml +++ b/charts/kubezero-istio/charts/base/crds/crd-all.gen.yaml @@ -1287,6 +1287,10 @@ spec: description: Applies only to sidecars. format: string type: string + destinationPort: + description: The destination_port value used by a + filter chain's match condition. + type: integer filter: description: The name of a specific filter to apply the patch to. @@ -2817,6 +2821,11 @@ spec: apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: + labels: + app: istio-pilot + chart: istio + heritage: Tiller + release: istio name: workloadgroups.networking.istio.io spec: additionalPrinterColumns: @@ -2884,11 +2893,11 @@ spec: - exec properties: exec: - description: health is determined by how the command that is executed + description: Health is determined by how the command that is executed exited. properties: command: - description: command to run. + description: Command to run. items: format: string type: string @@ -2906,7 +2915,7 @@ spec: format: string type: string httpHeaders: - description: headers the proxy will pass on to make the request. + description: Headers the proxy will pass on to make the request. items: properties: name: @@ -2922,7 +2931,7 @@ spec: format: string type: string port: - description: port on which the endpoint lives. + description: Port on which the endpoint lives. type: integer scheme: format: string @@ -2943,7 +2952,7 @@ spec: format: int32 type: integer tcpSocket: - description: health is determined by if the proxy is able to connect. + description: Health is determined by if the proxy is able to connect. properties: host: format: string @@ -3244,6 +3253,19 @@ metadata: release: istio name: peerauthentications.security.istio.io spec: + additionalPrinterColumns: + - JSONPath: .spec.mtls.mode + description: Defines the mTLS mode used for peer authentication. + name: Mode + type: string + - JSONPath: .metadata.creationTimestamp + description: 'CreationTimestamp is a timestamp representing the server time when + this object was created. It is not guaranteed to be set in happens-before order + across separate operations. Clients may not set this value. It is represented + in RFC3339 form and is in UTC. Populated by the system. Read-only. Null for + lists. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata' + name: Age + type: date group: security.istio.io names: categories: diff --git a/charts/kubezero-istio/charts/base/files/gen-istio-cluster.yaml b/charts/kubezero-istio/charts/base/files/gen-istio-cluster.yaml index 0a30433..2fda970 100644 --- a/charts/kubezero-istio/charts/base/files/gen-istio-cluster.yaml +++ b/charts/kubezero-istio/charts/base/files/gen-istio-cluster.yaml @@ -1289,6 +1289,10 @@ spec: description: Applies only to sidecars. format: string type: string + destinationPort: + description: The destination_port value used by a + filter chain's match condition. + type: integer filter: description: The name of a specific filter to apply the patch to. @@ -2819,6 +2823,11 @@ spec: apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: + labels: + app: istio-pilot + chart: istio + heritage: Tiller + release: istio name: workloadgroups.networking.istio.io spec: additionalPrinterColumns: @@ -2886,11 +2895,11 @@ spec: - exec properties: exec: - description: health is determined by how the command that is executed + description: Health is determined by how the command that is executed exited. properties: command: - description: command to run. + description: Command to run. items: format: string type: string @@ -2908,7 +2917,7 @@ spec: format: string type: string httpHeaders: - description: headers the proxy will pass on to make the request. + description: Headers the proxy will pass on to make the request. items: properties: name: @@ -2924,7 +2933,7 @@ spec: format: string type: string port: - description: port on which the endpoint lives. + description: Port on which the endpoint lives. type: integer scheme: format: string @@ -2945,7 +2954,7 @@ spec: format: int32 type: integer tcpSocket: - description: health is determined by if the proxy is able to connect. + description: Health is determined by if the proxy is able to connect. properties: host: format: string @@ -3246,6 +3255,19 @@ metadata: release: istio name: peerauthentications.security.istio.io spec: + additionalPrinterColumns: + - JSONPath: .spec.mtls.mode + description: Defines the mTLS mode used for peer authentication. + name: Mode + type: string + - JSONPath: .metadata.creationTimestamp + description: 'CreationTimestamp is a timestamp representing the server time when + this object was created. It is not guaranteed to be set in happens-before order + across separate operations. Clients may not set this value. It is represented + in RFC3339 form and is in UTC. Populated by the system. Read-only. Null for + lists. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata' + name: Age + type: date group: security.istio.io names: categories: @@ -3529,7 +3551,7 @@ rules: # sidecar injection controller - apiGroups: ["admissionregistration.k8s.io"] resources: ["mutatingwebhookconfigurations"] - verbs: ["get", "list", "watch", "patch"] + verbs: ["get", "list", "watch", "update", "patch"] # configuration validation webhook controller - apiGroups: ["admissionregistration.k8s.io"] @@ -3537,7 +3559,9 @@ rules: verbs: ["get", "list", "watch", "update"] # istio configuration - - apiGroups: ["config.istio.io", "security.istio.io", "networking.istio.io", "authentication.istio.io"] + # removing CRD permissions can break older versions of Istio running alongside this control plane (https://github.com/istio/istio/issues/29382) + # please proceed with caution + - apiGroups: ["config.istio.io", "security.istio.io", "networking.istio.io", "authentication.istio.io", "rbac.istio.io"] verbs: ["get", "watch", "list"] resources: ["*"] - apiGroups: ["networking.istio.io"] @@ -3621,11 +3645,18 @@ rules: - "security.istio.io" - "networking.istio.io" - "authentication.istio.io" + - "rbac.istio.io" resources: ["*"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["endpoints", "pods", "services", "nodes", "replicationcontrollers", "namespaces", "secrets"] verbs: ["get", "list", "watch"] + - apiGroups: ["networking.istio.io"] + verbs: [ "get", "watch", "list" ] + resources: [ "workloadentries" ] + - apiGroups: ["apiextensions.k8s.io"] + resources: ["customresourcedefinitions"] + verbs: ["get", "list", "watch"] - apiGroups: ["discovery.k8s.io"] resources: ["endpointslices"] verbs: ["get", "list", "watch"] diff --git a/charts/kubezero-istio/charts/base/templates/clusterrole.yaml b/charts/kubezero-istio/charts/base/templates/clusterrole.yaml index 521c24b..e4176d5 100644 --- a/charts/kubezero-istio/charts/base/templates/clusterrole.yaml +++ b/charts/kubezero-istio/charts/base/templates/clusterrole.yaml @@ -9,7 +9,7 @@ rules: # sidecar injection controller - apiGroups: ["admissionregistration.k8s.io"] resources: ["mutatingwebhookconfigurations"] - verbs: ["get", "list", "watch", "patch"] + verbs: ["get", "list", "watch", "update", "patch"] # configuration validation webhook controller - apiGroups: ["admissionregistration.k8s.io"] @@ -17,11 +17,13 @@ rules: verbs: ["get", "list", "watch", "update"] # istio configuration - - apiGroups: ["config.istio.io", "security.istio.io", "networking.istio.io", "authentication.istio.io"] + # removing CRD permissions can break older versions of Istio running alongside this control plane (https://github.com/istio/istio/issues/29382) + # please proceed with caution + - apiGroups: ["config.istio.io", "security.istio.io", "networking.istio.io", "authentication.istio.io", "rbac.istio.io"] verbs: ["get", "watch", "list"] resources: ["*"] {{- if .Values.global.istiod.enableAnalysis }} - - apiGroups: ["config.istio.io", "security.istio.io", "networking.istio.io", "authentication.istio.io"] + - apiGroups: ["config.istio.io", "security.istio.io", "networking.istio.io", "authentication.istio.io", "rbac.istio.io"] verbs: ["update"] # TODO: should be on just */status but wildcard is not supported resources: ["*"] @@ -115,11 +117,18 @@ rules: - "security.istio.io" - "networking.istio.io" - "authentication.istio.io" + - "rbac.istio.io" resources: ["*"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["endpoints", "pods", "services", "nodes", "replicationcontrollers", "namespaces", "secrets"] verbs: ["get", "list", "watch"] + - apiGroups: ["networking.istio.io"] + verbs: [ "get", "watch", "list" ] + resources: [ "workloadentries" ] + - apiGroups: ["apiextensions.k8s.io"] + resources: ["customresourcedefinitions"] + verbs: ["get", "list", "watch"] - apiGroups: ["discovery.k8s.io"] resources: ["endpointslices"] verbs: ["get", "list", "watch"] @@ -132,13 +141,13 @@ rules: - apiGroups: ["authorization.k8s.io"] resources: ["subjectaccessreviews"] verbs: ["create"] -{{- if or .Values.global.externalIstiod .Values.global.centralIstiod }} +{{- if or .Values.global.externalIstiod }} - apiGroups: [""] resources: ["configmaps"] verbs: ["create", "get", "list", "watch", "update"] - apiGroups: ["admissionregistration.k8s.io"] resources: ["mutatingwebhookconfigurations"] - verbs: ["get", "list", "watch", "patch"] + verbs: ["get", "list", "watch", "update", "patch"] - apiGroups: ["admissionregistration.k8s.io"] resources: ["validatingwebhookconfigurations"] verbs: ["get", "list", "watch", "update"] diff --git a/charts/kubezero-istio/charts/base/values.yaml b/charts/kubezero-istio/charts/base/values.yaml index af16510..8f86ba0 100644 --- a/charts/kubezero-istio/charts/base/values.yaml +++ b/charts/kubezero-istio/charts/base/values.yaml @@ -12,6 +12,8 @@ global: enableAnalysis: false configValidation: true + externalIstiod: false + remotePilotAddress: "" base: # Used for helm2 to add the CRDs to templates. @@ -19,4 +21,7 @@ base: # Validation webhook configuration url # For example: https://$remotePilotAddress:15017/validate - validationURL: "" \ No newline at end of file + validationURL: "" + + # For istioctl usage to disable istio config crds in base + enableIstioConfigCRDs: true diff --git a/charts/kubezero-istio/charts/istio-discovery/Chart.yaml b/charts/kubezero-istio/charts/istio-discovery/Chart.yaml index daed6bd..a43db08 100644 --- a/charts/kubezero-istio/charts/istio-discovery/Chart.yaml +++ b/charts/kubezero-istio/charts/istio-discovery/Chart.yaml @@ -1,7 +1,6 @@ apiVersion: v1 name: istio-discovery -version: 1.2.0 -appVersion: 1.2.0 +version: 1.9.1 tillerVersion: ">=2.7.2" description: Helm chart for istio control plane keywords: diff --git a/charts/kubezero-istio/charts/istio-discovery/files/gen-istio.yaml b/charts/kubezero-istio/charts/istio-discovery/files/gen-istio.yaml index da98e33..4a4077a 100644 --- a/charts/kubezero-istio/charts/istio-discovery/files/gen-istio.yaml +++ b/charts/kubezero-istio/charts/istio-discovery/files/gen-istio.yaml @@ -39,8 +39,6 @@ data: mesh: |- defaultConfig: discoveryAddress: istiod.istio-system.svc:15012 - proxyMetadata: - DNS_AGENT: "" tracing: zipkin: address: zipkin.istio-system:9411 @@ -64,16 +62,7 @@ data: values: |- { "global": { - "arch": { - "amd64": 2, - "ppc64le": 2, - "s390x": 2 - }, "caAddress": "", - "centralIstiod": false, - "configValidation": true, - "defaultConfigVisibilitySettings": [], - "defaultNodeSelector": {}, "defaultPodDisruptionBudget": { "enabled": true }, @@ -82,7 +71,6 @@ data: "cpu": "10m" } }, - "defaultTolerations": [], "externalIstiod": false, "hub": "gcr.io/istio-testing", "imagePullPolicy": "", @@ -96,18 +84,12 @@ data: "logging": { "level": "default:info" }, - "meshExpansion": { - "enabled": false, - "useILB": false - }, "meshID": "", "meshNetworks": {}, "mountMtlsCerts": false, "multiCluster": { "clusterName": "", - "enabled": false, - "globalDomainSuffix": "global", - "includeEnvoyFilter": true + "enabled": false }, "network": "", "omitSidecarInjectorConfigMap": false, @@ -191,14 +173,17 @@ data: "revision": "", "sidecarInjectorWebhook": { "alwaysInjectSelector": [], + "defaultTemplates": [], "enableNamespacesByDefault": false, "injectedAnnotations": {}, "neverInjectSelector": [], "objectSelector": { "autoInject": true, - "enabled": false + "enabled": true }, - "rewriteAppHTTPProbe": true + "rewriteAppHTTPProbe": true, + "templates": {}, + "useLegacySelectors": true } } @@ -208,476 +193,508 @@ data: # New fields should not use Values - it is a 'primary' config object, users should be able # to fine tune it or use it with kube-inject. config: |- + # defaultTemplates defines the default template to use for pods that do not explicitly specify a template + defaultTemplates: [sidecar] policy: enabled alwaysInjectSelector: [] neverInjectSelector: [] injectedAnnotations: - - template: | - {{- $holdProxy := or .ProxyConfig.HoldApplicationUntilProxyStarts.GetValue .Values.global.proxy.holdApplicationUntilProxyStarts }} - rewriteAppHTTPProbe: {{ valueOrDefault .Values.sidecarInjectorWebhook.rewriteAppHTTPProbe false }} - initContainers: - {{ if ne (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) `NONE` }} - {{ if .Values.istio_cni.enabled -}} - - name: istio-validation - {{ else -}} - - name: istio-init - {{ end -}} - {{- if contains "/" .Values.global.proxy_init.image }} - image: "{{ .Values.global.proxy_init.image }}" - {{- else }} - image: "{{ .Values.global.hub }}/{{ .Values.global.proxy_init.image }}:{{ .Values.global.tag }}" - {{- end }} - args: - - istio-iptables - - "-p" - - "15001" - - "-z" - - "15006" - - "-u" - - "1337" - - "-m" - - "{{ annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode }}" - - "-i" - - "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundIPRanges` .Values.global.proxy.includeIPRanges }}" - - "-x" - - "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundIPRanges` .Values.global.proxy.excludeIPRanges }}" - - "-b" - - "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeInboundPorts` `*` }}" - - "-d" - {{- if excludeInboundPort (annotation .ObjectMeta `status.sidecar.istio.io/port` .Values.global.proxy.statusPort) (annotation .ObjectMeta `traffic.sidecar.istio.io/excludeInboundPorts` .Values.global.proxy.excludeInboundPorts) }} - - "15090,15021,{{ excludeInboundPort (annotation .ObjectMeta `status.sidecar.istio.io/port` .Values.global.proxy.statusPort) (annotation .ObjectMeta `traffic.sidecar.istio.io/excludeInboundPorts` .Values.global.proxy.excludeInboundPorts) }}" - {{- else }} - - "15090,15021" - {{- end }} - {{ if or (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/includeOutboundPorts`) (ne (valueOrDefault .Values.global.proxy.includeOutboundPorts "") "") -}} - - "-q" - - "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundPorts` .Values.global.proxy.includeOutboundPorts }}" - {{ end -}} - {{ if or (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/excludeOutboundPorts`) (ne (valueOrDefault .Values.global.proxy.excludeOutboundPorts "") "") -}} - - "-o" - - "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundPorts` .Values.global.proxy.excludeOutboundPorts }}" - {{ end -}} - {{ if (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/kubevirtInterfaces`) -}} - - "-k" - - "{{ index .ObjectMeta.Annotations `traffic.sidecar.istio.io/kubevirtInterfaces` }}" - {{ end -}} - {{ if .Values.istio_cni.enabled -}} - - "--run-validation" - - "--skip-rule-apply" - {{ end -}} - imagePullPolicy: "{{ valueOrDefault .Values.global.imagePullPolicy `Always` }}" - {{- if .ProxyConfig.ProxyMetadata }} - env: - {{- range $key, $value := .ProxyConfig.ProxyMetadata }} - - name: {{ $key }} - value: "{{ $value }}" - {{- end }} - {{- end }} - resources: - {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) }} - {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) }} - requests: - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) -}} - cpu: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU` }}" + template: "{{ Template_Version_And_Istio_Version_Mismatched_Check_Installation }}" + templates: + sidecar: | + {{- $containers := list }} + {{- range $index, $container := .Spec.Containers }}{{ if not (eq $container.Name "istio-proxy") }}{{ $containers = append $containers $container.Name }}{{end}}{{- end}} + metadata: + labels: + security.istio.io/tlsMode: {{ index .ObjectMeta.Labels `security.istio.io/tlsMode` | default "istio" | quote }} + service.istio.io/canonical-name: {{ index .ObjectMeta.Labels `service.istio.io/canonical-name` | default (index .ObjectMeta.Labels `app.kubernetes.io/name`) | default (index .ObjectMeta.Labels `app`) | default .DeploymentMeta.Name | quote }} + service.istio.io/canonical-revision: {{ index .ObjectMeta.Labels `service.istio.io/canonical-revision` | default (index .ObjectMeta.Labels `app.kubernetes.io/version`) | default (index .ObjectMeta.Labels `version`) | default "latest" | quote }} + istio.io/rev: {{ .Revision | default "default" | quote }} + annotations: { + {{- if eq (len $containers) 1 }} + kubectl.kubernetes.io/default-logs-container: "{{ index $containers 0 }}", {{ end }} - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) -}} - memory: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory` }}" - {{ end }} - {{- end }} - {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) }} - limits: - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) -}} - cpu: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit` }}" - {{ end }} - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) -}} - memory: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit` }}" - {{ end }} - {{- end }} - {{- else }} - {{- if .Values.global.proxy.resources }} - {{ toYaml .Values.global.proxy.resources | indent 4 }} - {{- end }} - {{- end }} - securityContext: - allowPrivilegeEscalation: {{ .Values.global.proxy.privileged }} - privileged: {{ .Values.global.proxy.privileged }} - capabilities: - {{- if not .Values.istio_cni.enabled }} - add: - - NET_ADMIN - - NET_RAW - {{- end }} - drop: - - ALL - {{- if not .Values.istio_cni.enabled }} - readOnlyRootFilesystem: false - runAsGroup: 0 - runAsNonRoot: false - runAsUser: 0 - {{- else }} - readOnlyRootFilesystem: true - runAsGroup: 1337 - runAsUser: 1337 - runAsNonRoot: true - {{- end }} - restartPolicy: Always - {{ end -}} - {{- if eq .Values.global.proxy.enableCoreDump true }} - - name: enable-core-dump - args: - - -c - - sysctl -w kernel.core_pattern=/var/lib/istio/data/core.proxy && ulimit -c unlimited - command: - - /bin/sh - {{- if contains "/" .Values.global.proxy_init.image }} - image: "{{ .Values.global.proxy_init.image }}" - {{- else }} - image: "{{ .Values.global.hub }}/{{ .Values.global.proxy_init.image }}:{{ .Values.global.tag }}" - {{- end }} - imagePullPolicy: "{{ valueOrDefault .Values.global.imagePullPolicy `Always` }}" - resources: {} - securityContext: - allowPrivilegeEscalation: true - capabilities: - add: - - SYS_ADMIN - drop: - - ALL - privileged: true - readOnlyRootFilesystem: false - runAsGroup: 0 - runAsNonRoot: false - runAsUser: 0 - {{ end }} - containers: - - name: istio-proxy - {{- if contains "/" (annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy.image) }} - image: "{{ annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy.image }}" - {{- else }} - image: "{{ .Values.global.hub }}/{{ .Values.global.proxy.image }}:{{ .Values.global.tag }}" - {{- end }} - ports: - - containerPort: 15090 - protocol: TCP - name: http-envoy-prom - args: - - proxy - - sidecar - - --domain - - $(POD_NAMESPACE).svc.{{ .Values.global.proxy.clusterDomain }} - - --serviceCluster - {{ if ne "" (index .ObjectMeta.Labels "app") -}} - - "{{ index .ObjectMeta.Labels `app` }}.$(POD_NAMESPACE)" - {{ else -}} - - "{{ valueOrDefault .DeploymentMeta.Name `istio-proxy` }}.{{ valueOrDefault .DeploymentMeta.Namespace `default` }}" - {{ end -}} - - --proxyLogLevel={{ annotation .ObjectMeta `sidecar.istio.io/logLevel` .Values.global.proxy.logLevel}} - - --proxyComponentLogLevel={{ annotation .ObjectMeta `sidecar.istio.io/componentLogLevel` .Values.global.proxy.componentLogLevel}} - {{- if .Values.global.sts.servicePort }} - - --stsPort={{ .Values.global.sts.servicePort }} - {{- end }} - {{- if .Values.global.logAsJson }} - - --log_as_json - {{- end }} - {{- if gt .ProxyConfig.Concurrency.GetValue 0 }} - - --concurrency - - "{{ .ProxyConfig.Concurrency.GetValue }}" - {{- end -}} - {{- if .Values.global.proxy.lifecycle }} - lifecycle: - {{ toYaml .Values.global.proxy.lifecycle | indent 4 }} - {{- else if $holdProxy }} - lifecycle: - postStart: - exec: - command: - - pilot-agent - - wait - {{- end }} - env: - - name: JWT_POLICY - value: {{ .Values.global.jwtPolicy }} - - name: PILOT_CERT_PROVIDER - value: {{ .Values.global.pilotCertProvider }} - - name: CA_ADDR - {{- if .Values.global.caAddress }} - value: {{ .Values.global.caAddress }} - {{- else }} - value: istiod{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }}.{{ .Values.global.istioNamespace }}.svc:15012 - {{- end }} - - name: POD_NAME - valueFrom: - fieldRef: - fieldPath: metadata.name - - name: POD_NAMESPACE - valueFrom: - fieldRef: - fieldPath: metadata.namespace - - name: INSTANCE_IP - valueFrom: - fieldRef: - fieldPath: status.podIP - - name: SERVICE_ACCOUNT - valueFrom: - fieldRef: - fieldPath: spec.serviceAccountName - - name: HOST_IP - valueFrom: - fieldRef: - fieldPath: status.hostIP - - name: CANONICAL_SERVICE - valueFrom: - fieldRef: - fieldPath: metadata.labels['service.istio.io/canonical-name'] - - name: CANONICAL_REVISION - valueFrom: - fieldRef: - fieldPath: metadata.labels['service.istio.io/canonical-revision'] - - name: PROXY_CONFIG - value: | - {{ protoToJSON .ProxyConfig }} - - name: ISTIO_META_POD_PORTS - value: |- - [ - {{- $first := true }} - {{- range $index1, $c := .Spec.Containers }} - {{- range $index2, $p := $c.Ports }} - {{- if (structToJSON $p) }} - {{if not $first}},{{end}}{{ structToJSON $p }} - {{- $first = false }} - {{- end }} - {{- end}} - {{- end}} - ] - - name: ISTIO_META_APP_CONTAINERS - value: "{{- range $index, $container := .Spec.Containers }}{{- if ne $index 0}},{{- end}}{{ $container.Name }}{{- end}}" - - name: ISTIO_META_CLUSTER_ID - value: "{{ valueOrDefault .Values.global.multiCluster.clusterName `Kubernetes` }}" - - name: ISTIO_META_INTERCEPTION_MODE - value: "{{ or (index .ObjectMeta.Annotations `sidecar.istio.io/interceptionMode`) .ProxyConfig.InterceptionMode.String }}" - {{- if .Values.global.network }} - - name: ISTIO_META_NETWORK - value: "{{ .Values.global.network }}" - {{- end }} - {{ if .ObjectMeta.Annotations }} - - name: ISTIO_METAJSON_ANNOTATIONS - value: | - {{ toJSON .ObjectMeta.Annotations }} - {{ end }} - {{- if .DeploymentMeta.Name }} - - name: ISTIO_META_WORKLOAD_NAME - value: "{{ .DeploymentMeta.Name }}" - {{ end }} - {{- if and .TypeMeta.APIVersion .DeploymentMeta.Name }} - - name: ISTIO_META_OWNER - value: kubernetes://apis/{{ .TypeMeta.APIVersion }}/namespaces/{{ valueOrDefault .DeploymentMeta.Namespace `default` }}/{{ toLower .TypeMeta.Kind}}s/{{ .DeploymentMeta.Name }} - {{- end}} - {{- if (isset .ObjectMeta.Annotations `sidecar.istio.io/bootstrapOverride`) }} - - name: ISTIO_BOOTSTRAP_OVERRIDE - value: "/etc/istio/custom-bootstrap/custom_bootstrap.json" - {{- end }} - {{- if .Values.global.meshID }} - - name: ISTIO_META_MESH_ID - value: "{{ .Values.global.meshID }}" - {{- else if (valueOrDefault .MeshConfig.TrustDomain .Values.global.trustDomain) }} - - name: ISTIO_META_MESH_ID - value: "{{ (valueOrDefault .MeshConfig.TrustDomain .Values.global.trustDomain) }}" - {{- end }} - {{- with (valueOrDefault .MeshConfig.TrustDomain .Values.global.trustDomain) }} - - name: TRUST_DOMAIN - value: "{{ . }}" - {{- end }} - {{- if and (eq .Values.global.proxy.tracer "datadog") (isset .ObjectMeta.Annotations `apm.datadoghq.com/env`) }} - {{- range $key, $value := fromJSON (index .ObjectMeta.Annotations `apm.datadoghq.com/env`) }} - - name: {{ $key }} - value: "{{ $value }}" - {{- end }} - {{- end }} - {{- range $key, $value := .ProxyConfig.ProxyMetadata }} - - name: {{ $key }} - value: "{{ $value }}" - {{- end }} - imagePullPolicy: "{{ valueOrDefault .Values.global.imagePullPolicy `Always` }}" - {{ if ne (annotation .ObjectMeta `status.sidecar.istio.io/port` .Values.global.proxy.statusPort) `0` }} - readinessProbe: - httpGet: - path: /healthz/ready - port: 15021 - initialDelaySeconds: {{ annotation .ObjectMeta `readiness.status.sidecar.istio.io/initialDelaySeconds` .Values.global.proxy.readinessInitialDelaySeconds }} - periodSeconds: {{ annotation .ObjectMeta `readiness.status.sidecar.istio.io/periodSeconds` .Values.global.proxy.readinessPeriodSeconds }} - timeoutSeconds: 3 - failureThreshold: {{ annotation .ObjectMeta `readiness.status.sidecar.istio.io/failureThreshold` .Values.global.proxy.readinessFailureThreshold }} - {{ end -}} - securityContext: - allowPrivilegeEscalation: {{ .Values.global.proxy.privileged }} - capabilities: - {{ if or (eq (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) `TPROXY`) (eq (annotation .ObjectMeta `sidecar.istio.io/capNetBindService` .Values.global.proxy.capNetBindService) `true`) -}} - add: - {{ if eq (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) `TPROXY` -}} - - NET_ADMIN + {{- if .Values.istio_cni.enabled }} + {{- if not .Values.istio_cni.chained }} + k8s.v1.cni.cncf.io/networks: '{{ appendMultusNetwork (index .ObjectMeta.Annotations `k8s.v1.cni.cncf.io/networks`) `istio-cni` }}', {{- end }} - {{ if eq (annotation .ObjectMeta `sidecar.istio.io/capNetBindService` .Values.global.proxy.capNetBindService) `true` -}} - - NET_BIND_SERVICE + sidecar.istio.io/interceptionMode: "{{ annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode }}", + {{ with annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundIPRanges` .Values.global.proxy.includeIPRanges }}traffic.sidecar.istio.io/includeOutboundIPRanges: "{{.}}",{{ end }} + {{ with annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundIPRanges` .Values.global.proxy.excludeIPRanges }}traffic.sidecar.istio.io/excludeOutboundIPRanges: "{{.}}",{{ end }} + traffic.sidecar.istio.io/includeInboundPorts: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeInboundPorts` `*` }}", + traffic.sidecar.istio.io/excludeInboundPorts: "{{ excludeInboundPort (annotation .ObjectMeta `status.sidecar.istio.io/port` .Values.global.proxy.statusPort) (annotation .ObjectMeta `traffic.sidecar.istio.io/excludeInboundPorts` .Values.global.proxy.excludeInboundPorts) }}", + {{ if or (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/includeOutboundPorts`) (ne (valueOrDefault .Values.global.proxy.includeOutboundPorts "") "") }} + traffic.sidecar.istio.io/includeOutboundPorts: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundPorts` .Values.global.proxy.includeOutboundPorts }}", {{- end }} + {{ if or (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/excludeOutboundPorts`) (ne .Values.global.proxy.excludeOutboundPorts "") }} + traffic.sidecar.istio.io/excludeOutboundPorts: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundPorts` .Values.global.proxy.excludeOutboundPorts }}", {{- end }} - drop: - - ALL - privileged: {{ .Values.global.proxy.privileged }} - readOnlyRootFilesystem: {{ not .Values.global.proxy.enableCoreDump }} - runAsGroup: 1337 - fsGroup: 1337 - {{ if or (eq (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) `TPROXY`) (eq (annotation .ObjectMeta `sidecar.istio.io/capNetBindService` .Values.global.proxy.capNetBindService) `true`) -}} - runAsNonRoot: false - runAsUser: 0 - {{- else -}} - runAsNonRoot: true - runAsUser: 1337 - {{- end }} - resources: - {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) }} - {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) }} - requests: - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) -}} - cpu: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU` }}" - {{ end }} - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) -}} - memory: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory` }}" - {{ end }} + {{ with index .ObjectMeta.Annotations `traffic.sidecar.istio.io/kubevirtInterfaces` }}traffic.sidecar.istio.io/kubevirtInterfaces: "{{.}}",{{ end }} {{- end }} - {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) }} - limits: - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) -}} - cpu: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit` }}" - {{ end }} - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) -}} - memory: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit` }}" - {{ end }} - {{- end }} - {{- else }} - {{- if .Values.global.proxy.resources }} - {{ toYaml .Values.global.proxy.resources | indent 4 }} - {{- end }} - {{- end }} - volumeMounts: - {{- if eq .Values.global.pilotCertProvider "istiod" }} - - mountPath: /var/run/secrets/istio - name: istiod-ca-cert - {{- end }} - - mountPath: /var/lib/istio/data - name: istio-data - {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/bootstrapOverride`) }} - - mountPath: /etc/istio/custom-bootstrap - name: custom-bootstrap-volume - {{- end }} - # SDS channel between istioagent and Envoy - - mountPath: /etc/istio/proxy - name: istio-envoy - {{- if eq .Values.global.jwtPolicy "third-party-jwt" }} - - mountPath: /var/run/secrets/tokens - name: istio-token - {{- end }} - {{- if .Values.global.mountMtlsCerts }} - # Use the key and cert mounted to /etc/certs/ for the in-cluster mTLS communications. - - mountPath: /etc/certs/ - name: istio-certs - readOnly: true - {{- end }} - - name: istio-podinfo - mountPath: /etc/istio/pod - {{- if and (eq .Values.global.proxy.tracer "lightstep") .ProxyConfig.GetTracing.GetTlsSettings }} - - mountPath: {{ directory .ProxyConfig.GetTracing.GetTlsSettings.GetCaCertificates }} - name: lightstep-certs - readOnly: true - {{- end }} - {{- if isset .ObjectMeta.Annotations `sidecar.istio.io/userVolumeMount` }} - {{ range $index, $value := fromJSON (index .ObjectMeta.Annotations `sidecar.istio.io/userVolumeMount`) }} - - name: "{{ $index }}" - {{ toYaml $value | indent 4 }} - {{ end }} - {{- end }} - volumes: - {{- if (isset .ObjectMeta.Annotations `sidecar.istio.io/bootstrapOverride`) }} - - name: custom-bootstrap-volume - configMap: - name: {{ annotation .ObjectMeta `sidecar.istio.io/bootstrapOverride` "" }} - {{- end }} - # SDS channel between istioagent and Envoy - - emptyDir: - medium: Memory - name: istio-envoy - - name: istio-data - emptyDir: {} - - name: istio-podinfo - downwardAPI: - items: - - path: "labels" - fieldRef: - fieldPath: metadata.labels - - path: "annotations" - fieldRef: - fieldPath: metadata.annotations - {{- if eq .Values.global.jwtPolicy "third-party-jwt" }} - - name: istio-token - projected: - sources: - - serviceAccountToken: - path: istio-token - expirationSeconds: 43200 - audience: {{ .Values.global.sds.token.aud }} - {{- end }} - {{- if eq .Values.global.pilotCertProvider "istiod" }} - - name: istiod-ca-cert - configMap: - name: istio-ca-root-cert - {{- end }} - {{- if .Values.global.mountMtlsCerts }} - # Use the key and cert mounted to /etc/certs/ for the in-cluster mTLS communications. - - name: istio-certs - secret: - optional: true - {{ if eq .Spec.ServiceAccountName "" }} - secretName: istio.default + } + spec: + {{- $holdProxy := or .ProxyConfig.HoldApplicationUntilProxyStarts.GetValue .Values.global.proxy.holdApplicationUntilProxyStarts }} + initContainers: + {{ if ne (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) `NONE` }} + {{ if .Values.istio_cni.enabled -}} + - name: istio-validation {{ else -}} - secretName: {{ printf "istio.%s" .Spec.ServiceAccountName }} - {{ end -}} - {{- end }} - {{- if isset .ObjectMeta.Annotations `sidecar.istio.io/userVolume` }} - {{range $index, $value := fromJSON (index .ObjectMeta.Annotations `sidecar.istio.io/userVolume`) }} - - name: "{{ $index }}" - {{ toYaml $value | indent 2 }} - {{ end }} - {{ end }} - {{- if and (eq .Values.global.proxy.tracer "lightstep") .ProxyConfig.GetTracing.GetTlsSettings }} - - name: lightstep-certs - secret: - optional: true - secretName: lightstep.cacert - {{- end }} - podRedirectAnnot: - {{- if and (.Values.istio_cni.enabled) (not .Values.istio_cni.chained) }} - k8s.v1.cni.cncf.io/networks: '{{ appendMultusNetwork (index .ObjectMeta.Annotations `k8s.v1.cni.cncf.io/networks`) `istio-cni` }}' - {{- end }} - sidecar.istio.io/interceptionMode: "{{ annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode }}" - traffic.sidecar.istio.io/includeOutboundIPRanges: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundIPRanges` .Values.global.proxy.includeIPRanges }}" - traffic.sidecar.istio.io/excludeOutboundIPRanges: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundIPRanges` .Values.global.proxy.excludeIPRanges }}" - traffic.sidecar.istio.io/includeInboundPorts: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeInboundPorts` (includeInboundPorts .Spec.Containers) }}" - traffic.sidecar.istio.io/excludeInboundPorts: "{{ excludeInboundPort (annotation .ObjectMeta `status.sidecar.istio.io/port` .Values.global.proxy.statusPort) (annotation .ObjectMeta `traffic.sidecar.istio.io/excludeInboundPorts` .Values.global.proxy.excludeInboundPorts) }}" - {{ if or (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/includeOutboundPorts`) (ne (valueOrDefault .Values.global.proxy.includeOutboundPorts "") "") }} - traffic.sidecar.istio.io/includeOutboundPorts: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundPorts` .Values.global.proxy.includeOutboundPorts }}" - {{- end }} - {{ if or (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/excludeOutboundPorts`) (ne .Values.global.proxy.excludeOutboundPorts "") }} - traffic.sidecar.istio.io/excludeOutboundPorts: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundPorts` .Values.global.proxy.excludeOutboundPorts }}" - {{- end }} - traffic.sidecar.istio.io/kubevirtInterfaces: "{{ index .ObjectMeta.Annotations `traffic.sidecar.istio.io/kubevirtInterfaces` }}" - {{- if .Values.global.imagePullSecrets }} - imagePullSecrets: - {{- range .Values.global.imagePullSecrets }} - - name: {{ . }} - {{- end }} - {{- end }} + - name: istio-init + {{ end -}} + {{- if contains "/" (annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy_init.image) }} + image: "{{ annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy_init.image }}" + {{- else }} + image: "{{ .Values.global.hub }}/{{ .Values.global.proxy_init.image }}:{{ .Values.global.tag }}" + {{- end }} + args: + - istio-iptables + - "-p" + - "15001" + - "-z" + - "15006" + - "-u" + - "1337" + - "-m" + - "{{ annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode }}" + - "-i" + - "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundIPRanges` .Values.global.proxy.includeIPRanges }}" + - "-x" + - "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundIPRanges` .Values.global.proxy.excludeIPRanges }}" + - "-b" + - "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeInboundPorts` `*` }}" + - "-d" + {{- if excludeInboundPort (annotation .ObjectMeta `status.sidecar.istio.io/port` .Values.global.proxy.statusPort) (annotation .ObjectMeta `traffic.sidecar.istio.io/excludeInboundPorts` .Values.global.proxy.excludeInboundPorts) }} + - "15090,15021,{{ excludeInboundPort (annotation .ObjectMeta `status.sidecar.istio.io/port` .Values.global.proxy.statusPort) (annotation .ObjectMeta `traffic.sidecar.istio.io/excludeInboundPorts` .Values.global.proxy.excludeInboundPorts) }}" + {{- else }} + - "15090,15021" + {{- end }} + {{ if or (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/includeOutboundPorts`) (ne (valueOrDefault .Values.global.proxy.includeOutboundPorts "") "") -}} + - "-q" + - "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundPorts` .Values.global.proxy.includeOutboundPorts }}" + {{ end -}} + {{ if or (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/excludeOutboundPorts`) (ne (valueOrDefault .Values.global.proxy.excludeOutboundPorts "") "") -}} + - "-o" + - "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundPorts` .Values.global.proxy.excludeOutboundPorts }}" + {{ end -}} + {{ if (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/kubevirtInterfaces`) -}} + - "-k" + - "{{ index .ObjectMeta.Annotations `traffic.sidecar.istio.io/kubevirtInterfaces` }}" + {{ end -}} + {{ if .Values.istio_cni.enabled -}} + - "--run-validation" + - "--skip-rule-apply" + {{ end -}} + imagePullPolicy: "{{ valueOrDefault .Values.global.imagePullPolicy `Always` }}" + {{- if .ProxyConfig.ProxyMetadata }} + env: + {{- range $key, $value := .ProxyConfig.ProxyMetadata }} + - name: {{ $key }} + value: "{{ $value }}" + {{- end }} + {{- end }} + resources: + {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) }} + {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) }} + requests: + {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) -}} + cpu: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU` }}" + {{ end }} + {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) -}} + memory: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory` }}" + {{ end }} + {{- end }} + {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) }} + limits: + {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) -}} + cpu: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit` }}" + {{ end }} + {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) -}} + memory: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit` }}" + {{ end }} + {{- end }} + {{- else }} + {{- if .Values.global.proxy.resources }} + {{ toYaml .Values.global.proxy.resources | indent 6 }} + {{- end }} + {{- end }} + securityContext: + allowPrivilegeEscalation: {{ .Values.global.proxy.privileged }} + privileged: {{ .Values.global.proxy.privileged }} + capabilities: + {{- if not .Values.istio_cni.enabled }} + add: + - NET_ADMIN + - NET_RAW + {{- end }} + drop: + - ALL + {{- if not .Values.istio_cni.enabled }} + readOnlyRootFilesystem: false + runAsGroup: 0 + runAsNonRoot: false + runAsUser: 0 + {{- else }} + readOnlyRootFilesystem: true + runAsGroup: 1337 + runAsUser: 1337 + runAsNonRoot: true + {{- end }} + restartPolicy: Always + {{ end -}} + {{- if eq .Values.global.proxy.enableCoreDump true }} + - name: enable-core-dump + args: + - -c + - sysctl -w kernel.core_pattern=/var/lib/istio/data/core.proxy && ulimit -c unlimited + command: + - /bin/sh + {{- if contains "/" (annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy_init.image) }} + image: "{{ annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy_init.image }}" + {{- else }} + image: "{{ .Values.global.hub }}/{{ .Values.global.proxy_init.image }}:{{ .Values.global.tag }}" + {{- end }} + imagePullPolicy: "{{ valueOrDefault .Values.global.imagePullPolicy `Always` }}" + resources: {} + securityContext: + allowPrivilegeEscalation: true + capabilities: + add: + - SYS_ADMIN + drop: + - ALL + privileged: true + readOnlyRootFilesystem: false + runAsGroup: 0 + runAsNonRoot: false + runAsUser: 0 + {{ end }} + containers: + - name: istio-proxy + {{- if contains "/" (annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy.image) }} + image: "{{ annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy.image }}" + {{- else }} + image: "{{ .Values.global.hub }}/{{ .Values.global.proxy.image }}:{{ .Values.global.tag }}" + {{- end }} + ports: + - containerPort: 15090 + protocol: TCP + name: http-envoy-prom + args: + - proxy + - sidecar + - --domain + - $(POD_NAMESPACE).svc.{{ .Values.global.proxy.clusterDomain }} + - --serviceCluster + {{ if ne "" (index .ObjectMeta.Labels "app") -}} + - "{{ index .ObjectMeta.Labels `app` }}.$(POD_NAMESPACE)" + {{ else -}} + - "{{ valueOrDefault .DeploymentMeta.Name `istio-proxy` }}.{{ valueOrDefault .DeploymentMeta.Namespace `default` }}" + {{ end -}} + - --proxyLogLevel={{ annotation .ObjectMeta `sidecar.istio.io/logLevel` .Values.global.proxy.logLevel }} + - --proxyComponentLogLevel={{ annotation .ObjectMeta `sidecar.istio.io/componentLogLevel` .Values.global.proxy.componentLogLevel }} + - --log_output_level={{ annotation .ObjectMeta `sidecar.istio.io/agentLogLevel` .Values.global.logging.level }} + {{- if .Values.global.sts.servicePort }} + - --stsPort={{ .Values.global.sts.servicePort }} + {{- end }} + {{- if .Values.global.logAsJson }} + - --log_as_json + {{- end }} + {{- if gt .ProxyConfig.Concurrency.GetValue 0 }} + - --concurrency + - "{{ .ProxyConfig.Concurrency.GetValue }}" + {{- end -}} + {{- if .Values.global.proxy.lifecycle }} + lifecycle: + {{ toYaml .Values.global.proxy.lifecycle | indent 6 }} + {{- else if $holdProxy }} + lifecycle: + postStart: + exec: + command: + - pilot-agent + - wait + {{- end }} + env: + - name: JWT_POLICY + value: {{ .Values.global.jwtPolicy }} + - name: PILOT_CERT_PROVIDER + value: {{ .Values.global.pilotCertProvider }} + - name: CA_ADDR + {{- if .Values.global.caAddress }} + value: {{ .Values.global.caAddress }} + {{- else }} + value: istiod{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }}.{{ .Values.global.istioNamespace }}.svc:15012 + {{- end }} + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + - name: INSTANCE_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - name: SERVICE_ACCOUNT + valueFrom: + fieldRef: + fieldPath: spec.serviceAccountName + - name: HOST_IP + valueFrom: + fieldRef: + fieldPath: status.hostIP + - name: CANONICAL_SERVICE + valueFrom: + fieldRef: + fieldPath: metadata.labels['service.istio.io/canonical-name'] + - name: CANONICAL_REVISION + valueFrom: + fieldRef: + fieldPath: metadata.labels['service.istio.io/canonical-revision'] + - name: PROXY_CONFIG + value: | + {{ protoToJSON .ProxyConfig }} + - name: ISTIO_META_POD_PORTS + value: |- + [ + {{- $first := true }} + {{- range $index1, $c := .Spec.Containers }} + {{- range $index2, $p := $c.Ports }} + {{- if (structToJSON $p) }} + {{if not $first}},{{end}}{{ structToJSON $p }} + {{- $first = false }} + {{- end }} + {{- end}} + {{- end}} + ] + - name: ISTIO_META_APP_CONTAINERS + value: "{{ $containers | join "," }}" + - name: ISTIO_META_CLUSTER_ID + value: "{{ valueOrDefault .Values.global.multiCluster.clusterName `Kubernetes` }}" + - name: ISTIO_META_INTERCEPTION_MODE + value: "{{ or (index .ObjectMeta.Annotations `sidecar.istio.io/interceptionMode`) .ProxyConfig.InterceptionMode.String }}" + {{- if .Values.global.network }} + - name: ISTIO_META_NETWORK + value: "{{ .Values.global.network }}" + {{- end }} + {{ if .ObjectMeta.Annotations }} + - name: ISTIO_METAJSON_ANNOTATIONS + value: | + {{ toJSON .ObjectMeta.Annotations }} + {{ end }} + {{- if .DeploymentMeta.Name }} + - name: ISTIO_META_WORKLOAD_NAME + value: "{{ .DeploymentMeta.Name }}" + {{ end }} + {{- if and .TypeMeta.APIVersion .DeploymentMeta.Name }} + - name: ISTIO_META_OWNER + value: kubernetes://apis/{{ .TypeMeta.APIVersion }}/namespaces/{{ valueOrDefault .DeploymentMeta.Namespace `default` }}/{{ toLower .TypeMeta.Kind}}s/{{ .DeploymentMeta.Name }} + {{- end}} + {{- if (isset .ObjectMeta.Annotations `sidecar.istio.io/bootstrapOverride`) }} + - name: ISTIO_BOOTSTRAP_OVERRIDE + value: "/etc/istio/custom-bootstrap/custom_bootstrap.json" + {{- end }} + {{- if .Values.global.meshID }} + - name: ISTIO_META_MESH_ID + value: "{{ .Values.global.meshID }}" + {{- else if (valueOrDefault .MeshConfig.TrustDomain .Values.global.trustDomain) }} + - name: ISTIO_META_MESH_ID + value: "{{ (valueOrDefault .MeshConfig.TrustDomain .Values.global.trustDomain) }}" + {{- end }} + {{- with (valueOrDefault .MeshConfig.TrustDomain .Values.global.trustDomain) }} + - name: TRUST_DOMAIN + value: "{{ . }}" + {{- end }} + {{- if and (eq .Values.global.proxy.tracer "datadog") (isset .ObjectMeta.Annotations `apm.datadoghq.com/env`) }} + {{- range $key, $value := fromJSON (index .ObjectMeta.Annotations `apm.datadoghq.com/env`) }} + - name: {{ $key }} + value: "{{ $value }}" + {{- end }} + {{- end }} + {{- range $key, $value := .ProxyConfig.ProxyMetadata }} + - name: {{ $key }} + value: "{{ $value }}" + {{- end }} + imagePullPolicy: "{{ valueOrDefault .Values.global.imagePullPolicy `Always` }}" + {{ if ne (annotation .ObjectMeta `status.sidecar.istio.io/port` .Values.global.proxy.statusPort) `0` }} + readinessProbe: + httpGet: + path: /healthz/ready + port: 15021 + initialDelaySeconds: {{ annotation .ObjectMeta `readiness.status.sidecar.istio.io/initialDelaySeconds` .Values.global.proxy.readinessInitialDelaySeconds }} + periodSeconds: {{ annotation .ObjectMeta `readiness.status.sidecar.istio.io/periodSeconds` .Values.global.proxy.readinessPeriodSeconds }} + timeoutSeconds: 3 + failureThreshold: {{ annotation .ObjectMeta `readiness.status.sidecar.istio.io/failureThreshold` .Values.global.proxy.readinessFailureThreshold }} + {{ end -}} + securityContext: + allowPrivilegeEscalation: {{ .Values.global.proxy.privileged }} + capabilities: + {{ if or (eq (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) `TPROXY`) (eq (annotation .ObjectMeta `sidecar.istio.io/capNetBindService` .Values.global.proxy.capNetBindService) `true`) -}} + add: + {{ if eq (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) `TPROXY` -}} + - NET_ADMIN + {{- end }} + {{ if eq (annotation .ObjectMeta `sidecar.istio.io/capNetBindService` .Values.global.proxy.capNetBindService) `true` -}} + - NET_BIND_SERVICE + {{- end }} + {{- end }} + drop: + - ALL + privileged: {{ .Values.global.proxy.privileged }} + readOnlyRootFilesystem: {{ not .Values.global.proxy.enableCoreDump }} + runAsGroup: 1337 + fsGroup: 1337 + {{ if or (eq (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) `TPROXY`) (eq (annotation .ObjectMeta `sidecar.istio.io/capNetBindService` .Values.global.proxy.capNetBindService) `true`) -}} + runAsNonRoot: false + runAsUser: 0 + {{- else -}} + runAsNonRoot: true + runAsUser: 1337 + {{- end }} + resources: + {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) }} + {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) }} + requests: + {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) -}} + cpu: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU` }}" + {{ end }} + {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) -}} + memory: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory` }}" + {{ end }} + {{- end }} + {{- if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) }} + limits: + {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit`) -}} + cpu: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyCPULimit` }}" + {{ end }} + {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit`) -}} + memory: "{{ index .ObjectMeta.Annotations `sidecar.istio.io/proxyMemoryLimit` }}" + {{ end }} + {{- end }} + {{- else }} + {{- if .Values.global.proxy.resources }} + {{ toYaml .Values.global.proxy.resources | indent 6 }} + {{- end }} + {{- end }} + volumeMounts: + {{- if eq .Values.global.pilotCertProvider "istiod" }} + - mountPath: /var/run/secrets/istio + name: istiod-ca-cert + {{- end }} + - mountPath: /var/lib/istio/data + name: istio-data + {{ if (isset .ObjectMeta.Annotations `sidecar.istio.io/bootstrapOverride`) }} + - mountPath: /etc/istio/custom-bootstrap + name: custom-bootstrap-volume + {{- end }} + # SDS channel between istioagent and Envoy + - mountPath: /etc/istio/proxy + name: istio-envoy + {{- if eq .Values.global.jwtPolicy "third-party-jwt" }} + - mountPath: /var/run/secrets/tokens + name: istio-token + {{- end }} + {{- if .Values.global.mountMtlsCerts }} + # Use the key and cert mounted to /etc/certs/ for the in-cluster mTLS communications. + - mountPath: /etc/certs/ + name: istio-certs + readOnly: true + {{- end }} + - name: istio-podinfo + mountPath: /etc/istio/pod + {{- if and (eq .Values.global.proxy.tracer "lightstep") .ProxyConfig.GetTracing.GetTlsSettings }} + - mountPath: {{ directory .ProxyConfig.GetTracing.GetTlsSettings.GetCaCertificates }} + name: lightstep-certs + readOnly: true + {{- end }} + {{- if isset .ObjectMeta.Annotations `sidecar.istio.io/userVolumeMount` }} + {{ range $index, $value := fromJSON (index .ObjectMeta.Annotations `sidecar.istio.io/userVolumeMount`) }} + - name: "{{ $index }}" + {{ toYaml $value | indent 6 }} + {{ end }} + {{- end }} + volumes: + {{- if (isset .ObjectMeta.Annotations `sidecar.istio.io/bootstrapOverride`) }} + - name: custom-bootstrap-volume + configMap: + name: {{ annotation .ObjectMeta `sidecar.istio.io/bootstrapOverride` "" }} + {{- end }} + # SDS channel between istioagent and Envoy + - emptyDir: + medium: Memory + name: istio-envoy + - name: istio-data + emptyDir: {} + - name: istio-podinfo + downwardAPI: + items: + - path: "labels" + fieldRef: + fieldPath: metadata.labels + - path: "annotations" + fieldRef: + fieldPath: metadata.annotations + - path: "cpu-limit" + resourceFieldRef: + containerName: istio-proxy + resource: limits.cpu + divisor: 1m + - path: "cpu-request" + resourceFieldRef: + containerName: istio-proxy + resource: requests.cpu + divisor: 1m + {{- if eq .Values.global.jwtPolicy "third-party-jwt" }} + - name: istio-token + projected: + sources: + - serviceAccountToken: + path: istio-token + expirationSeconds: 43200 + audience: {{ .Values.global.sds.token.aud }} + {{- end }} + {{- if eq .Values.global.pilotCertProvider "istiod" }} + - name: istiod-ca-cert + configMap: + name: istio-ca-root-cert + {{- end }} + {{- if .Values.global.mountMtlsCerts }} + # Use the key and cert mounted to /etc/certs/ for the in-cluster mTLS communications. + - name: istio-certs + secret: + optional: true + {{ if eq .Spec.ServiceAccountName "" }} + secretName: istio.default + {{ else -}} + secretName: {{ printf "istio.%s" .Spec.ServiceAccountName }} + {{ end -}} + {{- end }} + {{- if isset .ObjectMeta.Annotations `sidecar.istio.io/userVolume` }} + {{range $index, $value := fromJSON (index .ObjectMeta.Annotations `sidecar.istio.io/userVolume`) }} + - name: "{{ $index }}" + {{ toYaml $value | indent 4 }} + {{ end }} + {{ end }} + {{- if and (eq .Values.global.proxy.tracer "lightstep") .ProxyConfig.GetTracing.GetTlsSettings }} + - name: lightstep-certs + secret: + optional: true + secretName: lightstep.cacert + {{- end }} + {{- if .Values.global.imagePullSecrets }} + imagePullSecrets: + {{- range .Values.global.imagePullSecrets }} + - name: {{ . }} + {{- end }} + {{- end }} + {{- if eq (env "ENABLE_LEGACY_FSGROUP_INJECTION" "true") "true" }} + securityContext: + fsGroup: 1337 + {{- end }} --- # Source: istio-discovery/templates/service.yaml apiVersion: v1 @@ -740,6 +757,7 @@ spec: app: istiod istio.io/rev: default install.operator.istio.io/owning-resource: unknown + sidecar.istio.io/inject: "false" operator.istio.io/component: "Pilot" istio: pilot annotations: @@ -768,8 +786,6 @@ spec: protocol: TCP - containerPort: 15017 protocol: TCP - - containerPort: 15053 - protocol: TCP readinessProbe: httpGet: path: /ready @@ -807,8 +823,6 @@ spec: value: "true" - name: PILOT_ENABLE_PROTOCOL_SNIFFING_FOR_INBOUND value: "true" - - name: INJECTION_WEBHOOK_CONFIG_NAME - value: istio-sidecar-injector - name: ISTIOD_ADDR value: istiod.istio-system.svc:15012 - name: PILOT_ENABLE_ANALYSIS @@ -817,10 +831,6 @@ spec: value: "Kubernetes" - name: EXTERNAL_ISTIOD value: "false" - - name: CENTRAL_ISTIOD - value: "false" - - name: PILOT_ENDPOINT_TELEMETRY_LABEL - value: "true" resources: requests: cpu: 500m @@ -904,687 +914,6 @@ spec: name: cpu targetAverageUtilization: 80 --- -# Source: istio-discovery/templates/telemetryv2_1.6.yaml -apiVersion: networking.istio.io/v1alpha3 -kind: EnvoyFilter -metadata: - name: metadata-exchange-1.6 - namespace: istio-system - labels: - istio.io/rev: default - install.operator.istio.io/owning-resource: unknown - operator.istio.io/component: "Pilot" -spec: - configPatches: - - applyTo: HTTP_FILTER - match: - context: ANY # inbound, outbound, and gateway - proxy: - proxyVersion: '^1\.6.*' - listener: - filterChain: - filter: - name: "envoy.http_connection_manager" - patch: - operation: INSERT_BEFORE - value: - name: istio.metadata_exchange - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm - value: - config: - configuration: | - {} - vm_config: - runtime: envoy.wasm.runtime.null - code: - local: - inline_string: envoy.wasm.metadata_exchange ---- -# Source: istio-discovery/templates/telemetryv2_1.6.yaml -apiVersion: networking.istio.io/v1alpha3 -kind: EnvoyFilter -metadata: - name: tcp-metadata-exchange-1.6 - namespace: istio-system - labels: - istio.io/rev: default -spec: - configPatches: - - applyTo: NETWORK_FILTER - match: - context: SIDECAR_INBOUND - proxy: - proxyVersion: '^1\.6.*' - listener: {} - patch: - operation: INSERT_BEFORE - value: - name: istio.metadata_exchange - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange - value: - protocol: istio-peer-exchange - - applyTo: CLUSTER - match: - context: SIDECAR_OUTBOUND - proxy: - proxyVersion: '^1\.6.*' - cluster: {} - patch: - operation: MERGE - value: - filters: - - name: istio.metadata_exchange - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange - value: - protocol: istio-peer-exchange - - applyTo: CLUSTER - match: - context: GATEWAY - proxy: - proxyVersion: '^1\.6.*' - cluster: {} - patch: - operation: MERGE - value: - filters: - - name: istio.metadata_exchange - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange - value: - protocol: istio-peer-exchange ---- -# Source: istio-discovery/templates/telemetryv2_1.6.yaml -apiVersion: networking.istio.io/v1alpha3 -kind: EnvoyFilter -metadata: - name: stats-filter-1.6 - namespace: istio-system - labels: - istio.io/rev: default -spec: - configPatches: - - applyTo: HTTP_FILTER - match: - context: SIDECAR_OUTBOUND - proxy: - proxyVersion: '^1\.6.*' - listener: - filterChain: - filter: - name: "envoy.http_connection_manager" - subFilter: - name: "envoy.router" - patch: - operation: INSERT_BEFORE - value: - name: istio.stats - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm - value: - config: - root_id: stats_outbound - configuration: | - { - "debug": "false", - "stat_prefix": "istio" - } - vm_config: - vm_id: stats_outbound - runtime: envoy.wasm.runtime.null - code: - local: - inline_string: envoy.wasm.stats - - applyTo: HTTP_FILTER - match: - context: SIDECAR_INBOUND - proxy: - proxyVersion: '^1\.6.*' - listener: - filterChain: - filter: - name: "envoy.http_connection_manager" - subFilter: - name: "envoy.router" - patch: - operation: INSERT_BEFORE - value: - name: istio.stats - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm - value: - config: - root_id: stats_inbound - configuration: | - { - "debug": "false", - "stat_prefix": "istio" - } - vm_config: - vm_id: stats_inbound - runtime: envoy.wasm.runtime.null - code: - local: - inline_string: envoy.wasm.stats - - applyTo: HTTP_FILTER - match: - context: GATEWAY - proxy: - proxyVersion: '^1\.6.*' - listener: - filterChain: - filter: - name: "envoy.http_connection_manager" - subFilter: - name: "envoy.router" - patch: - operation: INSERT_BEFORE - value: - name: istio.stats - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm - value: - config: - root_id: stats_outbound - configuration: | - { - "debug": "false", - "stat_prefix": "istio", - "disable_host_header_fallback": true - } - vm_config: - vm_id: stats_outbound - runtime: envoy.wasm.runtime.null - code: - local: - inline_string: envoy.wasm.stats ---- -# Source: istio-discovery/templates/telemetryv2_1.6.yaml -apiVersion: networking.istio.io/v1alpha3 -kind: EnvoyFilter -metadata: - name: tcp-stats-filter-1.6 - namespace: istio-system - labels: - istio.io/rev: default -spec: - configPatches: - - applyTo: NETWORK_FILTER - match: - context: SIDECAR_INBOUND - proxy: - proxyVersion: '^1\.6.*' - listener: - filterChain: - filter: - name: "envoy.tcp_proxy" - patch: - operation: INSERT_BEFORE - value: - name: istio.stats - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm - value: - config: - root_id: stats_inbound - configuration: | - { - "debug": "false", - "stat_prefix": "istio" - } - vm_config: - vm_id: tcp_stats_inbound - runtime: envoy.wasm.runtime.null - code: - local: - inline_string: "envoy.wasm.stats" - - applyTo: NETWORK_FILTER - match: - context: SIDECAR_OUTBOUND - proxy: - proxyVersion: '^1\.6.*' - listener: - filterChain: - filter: - name: "envoy.tcp_proxy" - patch: - operation: INSERT_BEFORE - value: - name: istio.stats - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm - value: - config: - root_id: stats_outbound - configuration: | - { - "debug": "false", - "stat_prefix": "istio" - } - vm_config: - vm_id: tcp_stats_outbound - runtime: envoy.wasm.runtime.null - code: - local: - inline_string: "envoy.wasm.stats" - - applyTo: NETWORK_FILTER - match: - context: GATEWAY - proxy: - proxyVersion: '^1\.6.*' - listener: - filterChain: - filter: - name: "envoy.tcp_proxy" - patch: - operation: INSERT_BEFORE - value: - name: istio.stats - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm - value: - config: - root_id: stats_outbound - configuration: | - { - "debug": "false", - "stat_prefix": "istio" - } - vm_config: - vm_id: tcp_stats_outbound - runtime: envoy.wasm.runtime.null - code: - local: - inline_string: "envoy.wasm.stats" ---- -# Source: istio-discovery/templates/telemetryv2_1.7.yaml -# Note: metadata exchange filter is wasm enabled only in sidecars. -apiVersion: networking.istio.io/v1alpha3 -kind: EnvoyFilter -metadata: - name: metadata-exchange-1.7 - namespace: istio-system - labels: - istio.io/rev: default - install.operator.istio.io/owning-resource: unknown - operator.istio.io/component: "Pilot" -spec: - configPatches: - - applyTo: HTTP_FILTER - match: - context: SIDECAR_INBOUND - proxy: - proxyVersion: '^1\.7.*' - listener: - filterChain: - filter: - name: "envoy.http_connection_manager" - patch: - operation: INSERT_BEFORE - value: - name: istio.metadata_exchange - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm - value: - config: - configuration: - "@type": "type.googleapis.com/google.protobuf.StringValue" - value: | - {} - vm_config: - runtime: envoy.wasm.runtime.null - code: - local: - inline_string: envoy.wasm.metadata_exchange - - applyTo: HTTP_FILTER - match: - context: SIDECAR_OUTBOUND - proxy: - proxyVersion: '^1\.7.*' - listener: - filterChain: - filter: - name: "envoy.http_connection_manager" - patch: - operation: INSERT_BEFORE - value: - name: istio.metadata_exchange - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm - value: - config: - configuration: - "@type": "type.googleapis.com/google.protobuf.StringValue" - value: | - {} - vm_config: - runtime: envoy.wasm.runtime.null - code: - local: - inline_string: envoy.wasm.metadata_exchange - - applyTo: HTTP_FILTER - match: - context: GATEWAY - proxy: - proxyVersion: '^1\.7.*' - listener: - filterChain: - filter: - name: "envoy.http_connection_manager" - patch: - operation: INSERT_BEFORE - value: - name: istio.metadata_exchange - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm - value: - config: - configuration: - "@type": "type.googleapis.com/google.protobuf.StringValue" - value: | - {} - vm_config: - runtime: envoy.wasm.runtime.null - code: - local: - inline_string: envoy.wasm.metadata_exchange ---- -# Source: istio-discovery/templates/telemetryv2_1.7.yaml -apiVersion: networking.istio.io/v1alpha3 -kind: EnvoyFilter -metadata: - name: tcp-metadata-exchange-1.7 - namespace: istio-system - labels: - istio.io/rev: default -spec: - configPatches: - - applyTo: NETWORK_FILTER - match: - context: SIDECAR_INBOUND - proxy: - proxyVersion: '^1\.7.*' - listener: {} - patch: - operation: INSERT_BEFORE - value: - name: istio.metadata_exchange - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange - value: - protocol: istio-peer-exchange - - applyTo: CLUSTER - match: - context: SIDECAR_OUTBOUND - proxy: - proxyVersion: '^1\.7.*' - cluster: {} - patch: - operation: MERGE - value: - filters: - - name: istio.metadata_exchange - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange - value: - protocol: istio-peer-exchange - - applyTo: CLUSTER - match: - context: GATEWAY - proxy: - proxyVersion: '^1\.7.*' - cluster: {} - patch: - operation: MERGE - value: - filters: - - name: istio.metadata_exchange - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange - value: - protocol: istio-peer-exchange ---- -# Source: istio-discovery/templates/telemetryv2_1.7.yaml -# Note: http stats filter is wasm enabled only in sidecars. -apiVersion: networking.istio.io/v1alpha3 -kind: EnvoyFilter -metadata: - name: stats-filter-1.7 - namespace: istio-system - labels: - istio.io/rev: default -spec: - configPatches: - - applyTo: HTTP_FILTER - match: - context: SIDECAR_OUTBOUND - proxy: - proxyVersion: '^1\.7.*' - listener: - filterChain: - filter: - name: "envoy.http_connection_manager" - subFilter: - name: "envoy.router" - patch: - operation: INSERT_BEFORE - value: - name: istio.stats - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm - value: - config: - root_id: stats_outbound - configuration: - "@type": "type.googleapis.com/google.protobuf.StringValue" - value: | - { - "debug": "false", - "stat_prefix": "istio" - } - vm_config: - vm_id: stats_outbound - runtime: envoy.wasm.runtime.null - code: - local: - inline_string: envoy.wasm.stats - - applyTo: HTTP_FILTER - match: - context: SIDECAR_INBOUND - proxy: - proxyVersion: '^1\.7.*' - listener: - filterChain: - filter: - name: "envoy.http_connection_manager" - subFilter: - name: "envoy.router" - patch: - operation: INSERT_BEFORE - value: - name: istio.stats - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm - value: - config: - root_id: stats_inbound - configuration: - "@type": "type.googleapis.com/google.protobuf.StringValue" - value: | - { - "debug": "false", - "stat_prefix": "istio" - } - vm_config: - vm_id: stats_inbound - runtime: envoy.wasm.runtime.null - code: - local: - inline_string: envoy.wasm.stats - - applyTo: HTTP_FILTER - match: - context: GATEWAY - proxy: - proxyVersion: '^1\.7.*' - listener: - filterChain: - filter: - name: "envoy.http_connection_manager" - subFilter: - name: "envoy.router" - patch: - operation: INSERT_BEFORE - value: - name: istio.stats - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm - value: - config: - root_id: stats_outbound - configuration: - "@type": "type.googleapis.com/google.protobuf.StringValue" - value: | - { - "debug": "false", - "stat_prefix": "istio", - "disable_host_header_fallback": true - } - vm_config: - vm_id: stats_outbound - runtime: envoy.wasm.runtime.null - code: - local: - inline_string: envoy.wasm.stats ---- -# Source: istio-discovery/templates/telemetryv2_1.7.yaml -# Note: tcp stats filter is wasm enabled only in sidecars. -apiVersion: networking.istio.io/v1alpha3 -kind: EnvoyFilter -metadata: - name: tcp-stats-filter-1.7 - namespace: istio-system - labels: - istio.io/rev: default -spec: - configPatches: - - applyTo: NETWORK_FILTER - match: - context: SIDECAR_INBOUND - proxy: - proxyVersion: '^1\.7.*' - listener: - filterChain: - filter: - name: "envoy.tcp_proxy" - patch: - operation: INSERT_BEFORE - value: - name: istio.stats - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm - value: - config: - root_id: stats_inbound - configuration: - "@type": "type.googleapis.com/google.protobuf.StringValue" - value: | - { - "debug": "false", - "stat_prefix": "istio" - } - vm_config: - vm_id: tcp_stats_inbound - runtime: envoy.wasm.runtime.null - code: - local: - inline_string: "envoy.wasm.stats" - - applyTo: NETWORK_FILTER - match: - context: SIDECAR_OUTBOUND - proxy: - proxyVersion: '^1\.7.*' - listener: - filterChain: - filter: - name: "envoy.tcp_proxy" - patch: - operation: INSERT_BEFORE - value: - name: istio.stats - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm - value: - config: - root_id: stats_outbound - configuration: - "@type": "type.googleapis.com/google.protobuf.StringValue" - value: | - { - "debug": "false", - "stat_prefix": "istio" - } - vm_config: - vm_id: tcp_stats_outbound - runtime: envoy.wasm.runtime.null - code: - local: - inline_string: "envoy.wasm.stats" - - applyTo: NETWORK_FILTER - match: - context: GATEWAY - proxy: - proxyVersion: '^1\.7.*' - listener: - filterChain: - filter: - name: "envoy.tcp_proxy" - patch: - operation: INSERT_BEFORE - value: - name: istio.stats - typed_config: - "@type": type.googleapis.com/udpa.type.v1.TypedStruct - type_url: type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm - value: - config: - root_id: stats_outbound - configuration: - "@type": "type.googleapis.com/google.protobuf.StringValue" - value: | - { - "debug": "false", - "stat_prefix": "istio" - } - vm_config: - vm_id: tcp_stats_outbound - runtime: envoy.wasm.runtime.null - code: - local: - inline_string: "envoy.wasm.stats" ---- # Source: istio-discovery/templates/telemetryv2_1.8.yaml # Note: metadata exchange filter is wasm enabled only in sidecars. apiVersion: networking.istio.io/v1alpha3 @@ -1606,7 +935,7 @@ spec: listener: filterChain: filter: - name: "envoy.http_connection_manager" + name: "envoy.filters.network.http_connection_manager" patch: operation: INSERT_BEFORE value: @@ -1633,7 +962,7 @@ spec: listener: filterChain: filter: - name: "envoy.http_connection_manager" + name: "envoy.filters.network.http_connection_manager" patch: operation: INSERT_BEFORE value: @@ -1660,7 +989,7 @@ spec: listener: filterChain: filter: - name: "envoy.http_connection_manager" + name: "envoy.filters.network.http_connection_manager" patch: operation: INSERT_BEFORE value: @@ -1757,9 +1086,9 @@ spec: listener: filterChain: filter: - name: "envoy.http_connection_manager" + name: "envoy.filters.network.http_connection_manager" subFilter: - name: "envoy.router" + name: "envoy.filters.http.router" patch: operation: INSERT_BEFORE value: @@ -1789,9 +1118,9 @@ spec: listener: filterChain: filter: - name: "envoy.http_connection_manager" + name: "envoy.filters.network.http_connection_manager" subFilter: - name: "envoy.router" + name: "envoy.filters.http.router" patch: operation: INSERT_BEFORE value: @@ -1821,9 +1150,9 @@ spec: listener: filterChain: filter: - name: "envoy.http_connection_manager" + name: "envoy.filters.network.http_connection_manager" subFilter: - name: "envoy.router" + name: "envoy.filters.http.router" patch: operation: INSERT_BEFORE value: @@ -1866,7 +1195,7 @@ spec: listener: filterChain: filter: - name: "envoy.tcp_proxy" + name: "envoy.filters.network.tcp_proxy" patch: operation: INSERT_BEFORE value: @@ -1896,7 +1225,7 @@ spec: listener: filterChain: filter: - name: "envoy.tcp_proxy" + name: "envoy.filters.network.tcp_proxy" patch: operation: INSERT_BEFORE value: @@ -1926,7 +1255,7 @@ spec: listener: filterChain: filter: - name: "envoy.tcp_proxy" + name: "envoy.filters.network.tcp_proxy" patch: operation: INSERT_BEFORE value: @@ -1949,13 +1278,435 @@ spec: local: inline_string: "envoy.wasm.stats" --- +# Source: istio-discovery/templates/telemetryv2_1.9.yaml +# Note: metadata exchange filter is wasm enabled only in sidecars. +apiVersion: networking.istio.io/v1alpha3 +kind: EnvoyFilter +metadata: + name: metadata-exchange-1.9 + namespace: istio-system + labels: + istio.io/rev: default + install.operator.istio.io/owning-resource: unknown + operator.istio.io/component: "Pilot" +spec: + configPatches: + - applyTo: HTTP_FILTER + match: + context: SIDECAR_INBOUND + proxy: + proxyVersion: '^1\.9.*' + listener: + filterChain: + filter: + name: "envoy.filters.network.http_connection_manager" + patch: + operation: INSERT_BEFORE + value: + name: istio.metadata_exchange + typed_config: + "@type": type.googleapis.com/udpa.type.v1.TypedStruct + type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm + value: + config: + configuration: + "@type": "type.googleapis.com/google.protobuf.StringValue" + value: | + {} + vm_config: + runtime: envoy.wasm.runtime.null + code: + local: + inline_string: envoy.wasm.metadata_exchange + - applyTo: HTTP_FILTER + match: + context: SIDECAR_OUTBOUND + proxy: + proxyVersion: '^1\.9.*' + listener: + filterChain: + filter: + name: "envoy.filters.network.http_connection_manager" + patch: + operation: INSERT_BEFORE + value: + name: istio.metadata_exchange + typed_config: + "@type": type.googleapis.com/udpa.type.v1.TypedStruct + type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm + value: + config: + configuration: + "@type": "type.googleapis.com/google.protobuf.StringValue" + value: | + {} + vm_config: + runtime: envoy.wasm.runtime.null + code: + local: + inline_string: envoy.wasm.metadata_exchange + - applyTo: HTTP_FILTER + match: + context: GATEWAY + proxy: + proxyVersion: '^1\.9.*' + listener: + filterChain: + filter: + name: "envoy.filters.network.http_connection_manager" + patch: + operation: INSERT_BEFORE + value: + name: istio.metadata_exchange + typed_config: + "@type": type.googleapis.com/udpa.type.v1.TypedStruct + type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm + value: + config: + configuration: + "@type": "type.googleapis.com/google.protobuf.StringValue" + value: | + {} + vm_config: + runtime: envoy.wasm.runtime.null + code: + local: + inline_string: envoy.wasm.metadata_exchange +--- +# Source: istio-discovery/templates/telemetryv2_1.9.yaml +apiVersion: networking.istio.io/v1alpha3 +kind: EnvoyFilter +metadata: + name: tcp-metadata-exchange-1.9 + namespace: istio-system + labels: + istio.io/rev: default +spec: + configPatches: + - applyTo: NETWORK_FILTER + match: + context: SIDECAR_INBOUND + proxy: + proxyVersion: '^1\.9.*' + listener: {} + patch: + operation: INSERT_BEFORE + value: + name: istio.metadata_exchange + typed_config: + "@type": type.googleapis.com/udpa.type.v1.TypedStruct + type_url: type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange + value: + protocol: istio-peer-exchange + - applyTo: CLUSTER + match: + context: SIDECAR_OUTBOUND + proxy: + proxyVersion: '^1\.9.*' + cluster: {} + patch: + operation: MERGE + value: + filters: + - name: istio.metadata_exchange + typed_config: + "@type": type.googleapis.com/udpa.type.v1.TypedStruct + type_url: type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange + value: + protocol: istio-peer-exchange + - applyTo: CLUSTER + match: + context: GATEWAY + proxy: + proxyVersion: '^1\.9.*' + cluster: {} + patch: + operation: MERGE + value: + filters: + - name: istio.metadata_exchange + typed_config: + "@type": type.googleapis.com/udpa.type.v1.TypedStruct + type_url: type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange + value: + protocol: istio-peer-exchange +--- +# Source: istio-discovery/templates/telemetryv2_1.9.yaml +# Note: http stats filter is wasm enabled only in sidecars. +apiVersion: networking.istio.io/v1alpha3 +kind: EnvoyFilter +metadata: + name: stats-filter-1.9 + namespace: istio-system + labels: + istio.io/rev: default +spec: + configPatches: + - applyTo: HTTP_FILTER + match: + context: SIDECAR_OUTBOUND + proxy: + proxyVersion: '^1\.9.*' + listener: + filterChain: + filter: + name: "envoy.filters.network.http_connection_manager" + subFilter: + name: "envoy.filters.http.router" + patch: + operation: INSERT_BEFORE + value: + name: istio.stats + typed_config: + "@type": type.googleapis.com/udpa.type.v1.TypedStruct + type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm + value: + config: + root_id: stats_outbound + configuration: + "@type": "type.googleapis.com/google.protobuf.StringValue" + value: | + { + "debug": "false", + "stat_prefix": "istio", + "metrics": [ + { + "dimensions": { + "source_cluster": "node.metadata['CLUSTER_ID']", + "destination_cluster": "upstream_peer.cluster_id" + } + } + ] + } + vm_config: + vm_id: stats_outbound + runtime: envoy.wasm.runtime.null + code: + local: + inline_string: envoy.wasm.stats + - applyTo: HTTP_FILTER + match: + context: SIDECAR_INBOUND + proxy: + proxyVersion: '^1\.9.*' + listener: + filterChain: + filter: + name: "envoy.filters.network.http_connection_manager" + subFilter: + name: "envoy.filters.http.router" + patch: + operation: INSERT_BEFORE + value: + name: istio.stats + typed_config: + "@type": type.googleapis.com/udpa.type.v1.TypedStruct + type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm + value: + config: + root_id: stats_inbound + configuration: + "@type": "type.googleapis.com/google.protobuf.StringValue" + value: | + { + "debug": "false", + "stat_prefix": "istio", + "metrics": [ + { + "dimensions": { + "destination_cluster": "node.metadata['CLUSTER_ID']", + "source_cluster": "downstream_peer.cluster_id" + } + } + ] + } + vm_config: + vm_id: stats_inbound + runtime: envoy.wasm.runtime.null + code: + local: + inline_string: envoy.wasm.stats + - applyTo: HTTP_FILTER + match: + context: GATEWAY + proxy: + proxyVersion: '^1\.9.*' + listener: + filterChain: + filter: + name: "envoy.filters.network.http_connection_manager" + subFilter: + name: "envoy.filters.http.router" + patch: + operation: INSERT_BEFORE + value: + name: istio.stats + typed_config: + "@type": type.googleapis.com/udpa.type.v1.TypedStruct + type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm + value: + config: + root_id: stats_outbound + configuration: + "@type": "type.googleapis.com/google.protobuf.StringValue" + value: | + { + "debug": "false", + "stat_prefix": "istio", + "disable_host_header_fallback": true, + "metrics": [ + { + "dimensions": { + "source_cluster": "node.metadata['CLUSTER_ID']", + "destination_cluster": "upstream_peer.cluster_id" + } + } + ] + } + vm_config: + vm_id: stats_outbound + runtime: envoy.wasm.runtime.null + code: + local: + inline_string: envoy.wasm.stats +--- +# Source: istio-discovery/templates/telemetryv2_1.9.yaml +# Note: tcp stats filter is wasm enabled only in sidecars. +apiVersion: networking.istio.io/v1alpha3 +kind: EnvoyFilter +metadata: + name: tcp-stats-filter-1.9 + namespace: istio-system + labels: + istio.io/rev: default +spec: + configPatches: + - applyTo: NETWORK_FILTER + match: + context: SIDECAR_INBOUND + proxy: + proxyVersion: '^1\.9.*' + listener: + filterChain: + filter: + name: "envoy.filters.network.tcp_proxy" + patch: + operation: INSERT_BEFORE + value: + name: istio.stats + typed_config: + "@type": type.googleapis.com/udpa.type.v1.TypedStruct + type_url: type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm + value: + config: + root_id: stats_inbound + configuration: + "@type": "type.googleapis.com/google.protobuf.StringValue" + value: | + { + "debug": "false", + "stat_prefix": "istio", + "metrics": [ + { + "dimensions": { + "destination_cluster": "node.metadata['CLUSTER_ID']", + "source_cluster": "downstream_peer.cluster_id" + } + } + ] + } + vm_config: + vm_id: tcp_stats_inbound + runtime: envoy.wasm.runtime.null + code: + local: + inline_string: "envoy.wasm.stats" + - applyTo: NETWORK_FILTER + match: + context: SIDECAR_OUTBOUND + proxy: + proxyVersion: '^1\.9.*' + listener: + filterChain: + filter: + name: "envoy.filters.network.tcp_proxy" + patch: + operation: INSERT_BEFORE + value: + name: istio.stats + typed_config: + "@type": type.googleapis.com/udpa.type.v1.TypedStruct + type_url: type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm + value: + config: + root_id: stats_outbound + configuration: + "@type": "type.googleapis.com/google.protobuf.StringValue" + value: | + { + "debug": "false", + "stat_prefix": "istio", + "metrics": [ + { + "dimensions": { + "source_cluster": "node.metadata['CLUSTER_ID']", + "destination_cluster": "upstream_peer.cluster_id" + } + } + ] + } + vm_config: + vm_id: tcp_stats_outbound + runtime: envoy.wasm.runtime.null + code: + local: + inline_string: "envoy.wasm.stats" + - applyTo: NETWORK_FILTER + match: + context: GATEWAY + proxy: + proxyVersion: '^1\.9.*' + listener: + filterChain: + filter: + name: "envoy.filters.network.tcp_proxy" + patch: + operation: INSERT_BEFORE + value: + name: istio.stats + typed_config: + "@type": type.googleapis.com/udpa.type.v1.TypedStruct + type_url: type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm + value: + config: + root_id: stats_outbound + configuration: + "@type": "type.googleapis.com/google.protobuf.StringValue" + value: | + { + "debug": "false", + "stat_prefix": "istio", + "metrics": [ + { + "dimensions": { + "source_cluster": "node.metadata['CLUSTER_ID']", + "destination_cluster": "upstream_peer.cluster_id" + } + } + ] + } + vm_config: + vm_id: tcp_stats_outbound + runtime: envoy.wasm.runtime.null + code: + local: + inline_string: "envoy.wasm.stats" +--- # Source: istio-discovery/templates/mutatingwebhook.yaml -# Installed for each revision - not installed for cluster resources ( cluster roles, bindings, crds) apiVersion: admissionregistration.k8s.io/v1beta1 kind: MutatingWebhookConfiguration metadata: name: istio-sidecar-injector - labels: istio.io/rev: default install.operator.istio.io/owning-resource: unknown @@ -1963,21 +1714,27 @@ metadata: app: sidecar-injector release: istio webhooks: - - name: sidecar-injector.istio.io - clientConfig: - service: - name: istiod - namespace: istio-system - path: "/inject" - caBundle: "" - sideEffects: None - rules: - - operations: [ "CREATE" ] - apiGroups: [""] - apiVersions: ["v1"] - resources: ["pods"] - failurePolicy: Fail - admissionReviewVersions: ["v1beta1", "v1"] - namespaceSelector: - matchLabels: - istio-injection: enabled +- name: sidecar-injector.istio.io + clientConfig: + service: + name: istiod + namespace: istio-system + path: "/inject" + caBundle: "" + sideEffects: None + rules: + - operations: [ "CREATE" ] + apiGroups: [""] + apiVersions: ["v1"] + resources: ["pods"] + failurePolicy: Fail + admissionReviewVersions: ["v1beta1", "v1"] + namespaceSelector: + matchLabels: + istio-injection: enabled + objectSelector: + matchExpressions: + - key: "sidecar.istio.io/inject" + operator: NotIn + values: + - "false" diff --git a/charts/kubezero-istio/charts/istio-discovery/files/injection-template.yaml b/charts/kubezero-istio/charts/istio-discovery/files/injection-template.yaml index 10dfbbe..b9b9866 100644 --- a/charts/kubezero-istio/charts/istio-discovery/files/injection-template.yaml +++ b/charts/kubezero-istio/charts/istio-discovery/files/injection-template.yaml @@ -1,6 +1,35 @@ -template: | + {{- $containers := list }} +{{- range $index, $container := .Spec.Containers }}{{ if not (eq $container.Name "istio-proxy") }}{{ $containers = append $containers $container.Name }}{{end}}{{- end}} +metadata: + labels: + security.istio.io/tlsMode: {{ index .ObjectMeta.Labels `security.istio.io/tlsMode` | default "istio" | quote }} + service.istio.io/canonical-name: {{ index .ObjectMeta.Labels `service.istio.io/canonical-name` | default (index .ObjectMeta.Labels `app.kubernetes.io/name`) | default (index .ObjectMeta.Labels `app`) | default .DeploymentMeta.Name | quote }} + service.istio.io/canonical-revision: {{ index .ObjectMeta.Labels `service.istio.io/canonical-revision` | default (index .ObjectMeta.Labels `app.kubernetes.io/version`) | default (index .ObjectMeta.Labels `version`) | default "latest" | quote }} + istio.io/rev: {{ .Revision | default "default" | quote }} + annotations: { + {{- if eq (len $containers) 1 }} + kubectl.kubernetes.io/default-logs-container: "{{ index $containers 0 }}", + {{ end }} +{{- if .Values.istio_cni.enabled }} + {{- if not .Values.istio_cni.chained }} + k8s.v1.cni.cncf.io/networks: '{{ appendMultusNetwork (index .ObjectMeta.Annotations `k8s.v1.cni.cncf.io/networks`) `istio-cni` }}', + {{- end }} + sidecar.istio.io/interceptionMode: "{{ annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode }}", + {{ with annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundIPRanges` .Values.global.proxy.includeIPRanges }}traffic.sidecar.istio.io/includeOutboundIPRanges: "{{.}}",{{ end }} + {{ with annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundIPRanges` .Values.global.proxy.excludeIPRanges }}traffic.sidecar.istio.io/excludeOutboundIPRanges: "{{.}}",{{ end }} + traffic.sidecar.istio.io/includeInboundPorts: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeInboundPorts` `*` }}", + traffic.sidecar.istio.io/excludeInboundPorts: "{{ excludeInboundPort (annotation .ObjectMeta `status.sidecar.istio.io/port` .Values.global.proxy.statusPort) (annotation .ObjectMeta `traffic.sidecar.istio.io/excludeInboundPorts` .Values.global.proxy.excludeInboundPorts) }}", + {{ if or (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/includeOutboundPorts`) (ne (valueOrDefault .Values.global.proxy.includeOutboundPorts "") "") }} + traffic.sidecar.istio.io/includeOutboundPorts: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundPorts` .Values.global.proxy.includeOutboundPorts }}", + {{- end }} + {{ if or (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/excludeOutboundPorts`) (ne .Values.global.proxy.excludeOutboundPorts "") }} + traffic.sidecar.istio.io/excludeOutboundPorts: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundPorts` .Values.global.proxy.excludeOutboundPorts }}", + {{- end }} + {{ with index .ObjectMeta.Annotations `traffic.sidecar.istio.io/kubevirtInterfaces` }}traffic.sidecar.istio.io/kubevirtInterfaces: "{{.}}",{{ end }} +{{- end }} + } +spec: {{- $holdProxy := or .ProxyConfig.HoldApplicationUntilProxyStarts.GetValue .Values.global.proxy.holdApplicationUntilProxyStarts }} - rewriteAppHTTPProbe: {{ valueOrDefault .Values.sidecarInjectorWebhook.rewriteAppHTTPProbe false }} initContainers: {{ if ne (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) `NONE` }} {{ if .Values.istio_cni.enabled -}} @@ -8,8 +37,8 @@ template: | {{ else -}} - name: istio-init {{ end -}} - {{- if contains "/" .Values.global.proxy_init.image }} - image: "{{ .Values.global.proxy_init.image }}" + {{- if contains "/" (annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy_init.image) }} + image: "{{ annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy_init.image }}" {{- else }} image: "{{ .Values.global.hub }}/{{ .Values.global.proxy_init.image }}:{{ .Values.global.tag }}" {{- end }} @@ -81,7 +110,7 @@ template: | {{- end }} {{- else }} {{- if .Values.global.proxy.resources }} - {{ toYaml .Values.global.proxy.resources | indent 4 }} + {{ toYaml .Values.global.proxy.resources | indent 6 }} {{- end }} {{- end }} securityContext: @@ -115,8 +144,8 @@ template: | - sysctl -w kernel.core_pattern=/var/lib/istio/data/core.proxy && ulimit -c unlimited command: - /bin/sh - {{- if contains "/" .Values.global.proxy_init.image }} - image: "{{ .Values.global.proxy_init.image }}" + {{- if contains "/" (annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy_init.image) }} + image: "{{ annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy_init.image }}" {{- else }} image: "{{ .Values.global.hub }}/{{ .Values.global.proxy_init.image }}:{{ .Values.global.tag }}" {{- end }} @@ -157,8 +186,9 @@ template: | {{ else -}} - "{{ valueOrDefault .DeploymentMeta.Name `istio-proxy` }}.{{ valueOrDefault .DeploymentMeta.Namespace `default` }}" {{ end -}} - - --proxyLogLevel={{ annotation .ObjectMeta `sidecar.istio.io/logLevel` .Values.global.proxy.logLevel}} - - --proxyComponentLogLevel={{ annotation .ObjectMeta `sidecar.istio.io/componentLogLevel` .Values.global.proxy.componentLogLevel}} + - --proxyLogLevel={{ annotation .ObjectMeta `sidecar.istio.io/logLevel` .Values.global.proxy.logLevel }} + - --proxyComponentLogLevel={{ annotation .ObjectMeta `sidecar.istio.io/componentLogLevel` .Values.global.proxy.componentLogLevel }} + - --log_output_level={{ annotation .ObjectMeta `sidecar.istio.io/agentLogLevel` .Values.global.logging.level }} {{- if .Values.global.sts.servicePort }} - --stsPort={{ .Values.global.sts.servicePort }} {{- end }} @@ -171,7 +201,7 @@ template: | {{- end -}} {{- if .Values.global.proxy.lifecycle }} lifecycle: - {{ toYaml .Values.global.proxy.lifecycle | indent 4 }} + {{ toYaml .Values.global.proxy.lifecycle | indent 6 }} {{- else if $holdProxy }} lifecycle: postStart: @@ -236,7 +266,7 @@ template: | {{- end}} ] - name: ISTIO_META_APP_CONTAINERS - value: "{{- range $index, $container := .Spec.Containers }}{{- if ne $index 0}},{{- end}}{{ $container.Name }}{{- end}}" + value: "{{ $containers | join "," }}" - name: ISTIO_META_CLUSTER_ID value: "{{ valueOrDefault .Values.global.multiCluster.clusterName `Kubernetes` }}" - name: ISTIO_META_INTERCEPTION_MODE @@ -341,7 +371,7 @@ template: | {{- end }} {{- else }} {{- if .Values.global.proxy.resources }} - {{ toYaml .Values.global.proxy.resources | indent 4 }} + {{ toYaml .Values.global.proxy.resources | indent 6 }} {{- end }} {{- end }} volumeMounts: @@ -378,7 +408,7 @@ template: | {{- if isset .ObjectMeta.Annotations `sidecar.istio.io/userVolumeMount` }} {{ range $index, $value := fromJSON (index .ObjectMeta.Annotations `sidecar.istio.io/userVolumeMount`) }} - name: "{{ $index }}" - {{ toYaml $value | indent 4 }} + {{ toYaml $value | indent 6 }} {{ end }} {{- end }} volumes: @@ -402,6 +432,16 @@ template: | - path: "annotations" fieldRef: fieldPath: metadata.annotations + - path: "cpu-limit" + resourceFieldRef: + containerName: istio-proxy + resource: limits.cpu + divisor: 1m + - path: "cpu-request" + resourceFieldRef: + containerName: istio-proxy + resource: requests.cpu + divisor: 1m {{- if eq .Values.global.jwtPolicy "third-party-jwt" }} - name: istio-token projected: @@ -430,7 +470,7 @@ template: | {{- if isset .ObjectMeta.Annotations `sidecar.istio.io/userVolume` }} {{range $index, $value := fromJSON (index .ObjectMeta.Annotations `sidecar.istio.io/userVolume`) }} - name: "{{ $index }}" - {{ toYaml $value | indent 2 }} + {{ toYaml $value | indent 4 }} {{ end }} {{ end }} {{- if and (eq .Values.global.proxy.tracer "lightstep") .ProxyConfig.GetTracing.GetTlsSettings }} @@ -439,25 +479,13 @@ template: | optional: true secretName: lightstep.cacert {{- end }} - podRedirectAnnot: - {{- if and (.Values.istio_cni.enabled) (not .Values.istio_cni.chained) }} - k8s.v1.cni.cncf.io/networks: '{{ appendMultusNetwork (index .ObjectMeta.Annotations `k8s.v1.cni.cncf.io/networks`) `istio-cni` }}' - {{- end }} - sidecar.istio.io/interceptionMode: "{{ annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode }}" - traffic.sidecar.istio.io/includeOutboundIPRanges: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundIPRanges` .Values.global.proxy.includeIPRanges }}" - traffic.sidecar.istio.io/excludeOutboundIPRanges: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundIPRanges` .Values.global.proxy.excludeIPRanges }}" - traffic.sidecar.istio.io/includeInboundPorts: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeInboundPorts` (includeInboundPorts .Spec.Containers) }}" - traffic.sidecar.istio.io/excludeInboundPorts: "{{ excludeInboundPort (annotation .ObjectMeta `status.sidecar.istio.io/port` .Values.global.proxy.statusPort) (annotation .ObjectMeta `traffic.sidecar.istio.io/excludeInboundPorts` .Values.global.proxy.excludeInboundPorts) }}" - {{ if or (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/includeOutboundPorts`) (ne (valueOrDefault .Values.global.proxy.includeOutboundPorts "") "") }} - traffic.sidecar.istio.io/includeOutboundPorts: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundPorts` .Values.global.proxy.includeOutboundPorts }}" - {{- end }} - {{ if or (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/excludeOutboundPorts`) (ne .Values.global.proxy.excludeOutboundPorts "") }} - traffic.sidecar.istio.io/excludeOutboundPorts: "{{ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundPorts` .Values.global.proxy.excludeOutboundPorts }}" - {{- end }} - traffic.sidecar.istio.io/kubevirtInterfaces: "{{ index .ObjectMeta.Annotations `traffic.sidecar.istio.io/kubevirtInterfaces` }}" {{- if .Values.global.imagePullSecrets }} imagePullSecrets: {{- range .Values.global.imagePullSecrets }} - name: {{ . }} {{- end }} {{- end }} + {{- if eq (env "ENABLE_LEGACY_FSGROUP_INJECTION" "true") "true" }} + securityContext: + fsGroup: 1337 + {{- end }} diff --git a/charts/kubezero-istio/charts/istio-discovery/templates/configmap.yaml b/charts/kubezero-istio/charts/istio-discovery/templates/configmap.yaml index 17174f1..3a6bb68 100644 --- a/charts/kubezero-istio/charts/istio-discovery/templates/configmap.yaml +++ b/charts/kubezero-istio/charts/istio-discovery/templates/configmap.yaml @@ -4,6 +4,12 @@ # Refer to https://github.com/spiffe/spiffe/blob/master/standards/SPIFFE-ID.md#21-trust-domain trustDomain: {{ .Values.global.trustDomain | default "cluster.local" | quote }} + # The namespace to treat as the administrative root namespace for Istio configuration. + # When processing a leaf namespace Istio will search for declarations in that namespace first + # and if none are found it will search in the root namespace. Any matching declaration found in the root namespace + # is processed as if it were declared in the leaf namespace. + rootNamespace: {{ .Values.meshConfig.rootNamespace | default .Values.global.istioNamespace }} + defaultConfig: {{- if .Values.global.meshID }} meshId: {{ .Values.global.meshID }} @@ -47,10 +53,13 @@ {{- /* Fill in openCensusAgent configuration from meshConfig so it isn't overwritten below */ -}} {{ toYaml $.Values.meshConfig.defaultConfig.tracing }} {{- end }} - {{- if .Values.global.remotePilotAddress }} + {{- if .Values.pilot.enabled }} discoveryAddress: {{ printf "istiod-remote.%s.svc" .Release.Namespace }}:15012 {{- else }} + discoveryAddress: {{ printf "istiod.%s.svc" .Release.Namespace }}:15012 + {{- end }} + {{- else }} discoveryAddress: istiod{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }}.{{.Release.Namespace}}.svc:15012 {{- end }} {{- end }} diff --git a/charts/kubezero-istio/charts/istio-discovery/templates/deployment.yaml b/charts/kubezero-istio/charts/istio-discovery/templates/deployment.yaml index ba586de..c7a42c0 100644 --- a/charts/kubezero-istio/charts/istio-discovery/templates/deployment.yaml +++ b/charts/kubezero-istio/charts/istio-discovery/templates/deployment.yaml @@ -37,6 +37,7 @@ spec: app: istiod istio.io/rev: {{ .Values.revision | default "default" }} install.operator.istio.io/owning-resource: {{ .Values.ownerName | default "unknown" }} + sidecar.istio.io/inject: "false" operator.istio.io/component: "Pilot" {{- if eq .Values.revision ""}} istio: pilot @@ -101,8 +102,6 @@ spec: protocol: TCP - containerPort: 15017 protocol: TCP - - containerPort: 15053 - protocol: TCP readinessProbe: httpGet: path: /ready @@ -148,14 +147,6 @@ spec: value: "{{ .Values.pilot.enableProtocolSniffingForOutbound }}" - name: PILOT_ENABLE_PROTOCOL_SNIFFING_FOR_INBOUND value: "{{ .Values.pilot.enableProtocolSniffingForInbound }}" -{{- if not (hasKey .Values.pilot.env "INJECTION_WEBHOOK_CONFIG_NAME") }} - - name: INJECTION_WEBHOOK_CONFIG_NAME - {{- if eq .Release.Namespace "istio-system" }} - value: istio-sidecar-injector{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} - {{- else }} - value: istio-sidecar-injector{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }}-{{ .Release.Namespace }} - {{- end }} -{{- end }} - name: ISTIOD_ADDR value: istiod{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }}.{{ .Release.Namespace }}.svc:15012 - name: PILOT_ENABLE_ANALYSIS @@ -164,11 +155,9 @@ spec: value: "{{ $.Values.global.multiCluster.clusterName | default `Kubernetes` }}" - name: EXTERNAL_ISTIOD value: "{{ $.Values.global.externalIstiod | default "false" }}" - - name: CENTRAL_ISTIOD - value: "{{ $.Values.global.centralIstiod | default "false" }}" -{{- if .Values.telemetry.v2.enabled }} +{{- if not .Values.telemetry.v2.enabled }} - name: PILOT_ENDPOINT_TELEMETRY_LABEL - value: "true" + value: "false" {{- end }} resources: {{- if .Values.pilot.resources }} diff --git a/charts/kubezero-istio/charts/istio-discovery/templates/istiod-injector-configmap.yaml b/charts/kubezero-istio/charts/istio-discovery/templates/istiod-injector-configmap.yaml index 8e660e9..15ece14 100644 --- a/charts/kubezero-istio/charts/istio-discovery/templates/istiod-injector-configmap.yaml +++ b/charts/kubezero-istio/charts/istio-discovery/templates/istiod-injector-configmap.yaml @@ -20,6 +20,15 @@ data: # New fields should not use Values - it is a 'primary' config object, users should be able # to fine tune it or use it with kube-inject. config: |- + # defaultTemplates defines the default template to use for pods that do not explicitly specify a template + {{- if .Values.sidecarInjectorWebhook.defaultTemplates }} + defaultTemplates: +{{- range .Values.sidecarInjectorWebhook.defaultTemplates}} + - {{ . }} +{{- end }} + {{- else }} + defaultTemplates: [sidecar] + {{- end }} policy: {{ .Values.global.proxy.autoInject }} alwaysInjectSelector: {{ toYaml .Values.sidecarInjectorWebhook.alwaysInjectSelector | trim | indent 6 }} @@ -29,7 +38,18 @@ data: {{- range $key, $val := .Values.sidecarInjectorWebhook.injectedAnnotations }} "{{ $key }}": "{{ $val }}" {{- end }} - -{{ .Files.Get "files/injection-template.yaml" | trim | indent 4 }} + {{- /* If someone ends up with this new template, but an older Istiod image, they will attempt to render this template + which will fail with "Pod injection failed: template: inject:1: function "Istio_1_9_Required_Template_And_Version_Mismatched" not defined". + This should make it obvious that their installation is broken. + */}} + template: {{ `{{ Template_Version_And_Istio_Version_Mismatched_Check_Installation }}` | quote }} + templates: +{{- if not (hasKey .Values.sidecarInjectorWebhook.templates "sidecar") }} + sidecar: | +{{ .Files.Get "files/injection-template.yaml" | trim | indent 8 }} +{{- end }} +{{- with .Values.sidecarInjectorWebhook.templates }} +{{ toYaml . | trim | indent 6 }} +{{- end }} {{- end }} diff --git a/charts/kubezero-istio/charts/istio-discovery/templates/mutatingwebhook.yaml b/charts/kubezero-istio/charts/istio-discovery/templates/mutatingwebhook.yaml index ca11b4f..f9fd67b 100644 --- a/charts/kubezero-istio/charts/istio-discovery/templates/mutatingwebhook.yaml +++ b/charts/kubezero-istio/charts/istio-discovery/templates/mutatingwebhook.yaml @@ -1,11 +1,35 @@ -# Installed for each revision - not installed for cluster resources ( cluster roles, bindings, crds) +{{- /* Core defines the common configuration used by all webhook segments */}} +{{- define "core" }} +{{- /* Kubernetes unfortunately requires a unique name for the webhook in some newer versions, so we assign +a unique prefix to each. */}} +- name: {{.Prefix}}sidecar-injector.istio.io + clientConfig: + {{- if .Values.istiodRemote.injectionURL }} + url: {{ .Values.istiodRemote.injectionURL }} + {{- else }} + service: + name: istiod{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} + namespace: {{ .Release.Namespace }} + path: "/inject" + {{- end }} + caBundle: "" + sideEffects: None + rules: + - operations: [ "CREATE" ] + apiGroups: [""] + apiVersions: ["v1"] + resources: ["pods"] + failurePolicy: Fail + admissionReviewVersions: ["v1beta1", "v1"] +{{- end }} +{{- /* Installed for each revision - not installed for cluster resources ( cluster roles, bindings, crds) */}} {{- if not .Values.global.operatorManageWebhooks }} apiVersion: admissionregistration.k8s.io/v1beta1 kind: MutatingWebhookConfiguration metadata: {{- if eq .Release.Namespace "istio-system"}} name: istio-sidecar-injector{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} -{{ else }} +{{- else }} name: istio-sidecar-injector{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }}-{{ .Release.Namespace }} {{- end }} labels: @@ -15,71 +39,151 @@ metadata: app: sidecar-injector release: {{ .Release.Name }} webhooks: - - name: sidecar-injector.istio.io - clientConfig: - {{- if .Values.istiodRemote.injectionURL }} - url: {{ .Values.istiodRemote.injectionURL }} - {{- else }} - service: - name: istiod{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} - namespace: {{ .Release.Namespace }} - path: "/inject" - {{- end }} - caBundle: "" - sideEffects: None - rules: - - operations: [ "CREATE" ] - apiGroups: [""] - apiVersions: ["v1"] - resources: ["pods"] - failurePolicy: Fail - admissionReviewVersions: ["v1beta1", "v1"] - namespaceSelector: +{{- if .Values.sidecarInjectorWebhook.useLegacySelectors}} +{{- /* Setup the "legacy" selectors. These are for backwards compatibility, will be removed in the future. */}} +{{- include "core" (mergeOverwrite (deepCopy .) (dict "Prefix" "")) }} + namespaceSelector: + {{- if .Values.sidecarInjectorWebhook.enableNamespacesByDefault }} + matchExpressions: + - key: name + operator: NotIn + values: + - {{ .Release.Namespace }} + - key: istio-injection + operator: NotIn + values: + - disabled + - key: istio-env + operator: DoesNotExist + - key: istio.io/rev + operator: DoesNotExist + {{- else if .Values.revision }} + matchExpressions: + - key: istio-injection + operator: DoesNotExist + - key: istio.io/rev + operator: In + values: + - {{ .Values.revision }} + {{- else }} + matchLabels: + istio-injection: enabled + {{- end }} + {{- if .Values.sidecarInjectorWebhook.objectSelector.enabled }} + objectSelector: + {{- if .Values.sidecarInjectorWebhook.objectSelector.autoInject }} + matchExpressions: + - key: "sidecar.istio.io/inject" + operator: NotIn + values: + - "false" + {{- else if .Values.revision }} + matchExpressions: + - key: "sidecar.istio.io/inject" + operator: DoesNotExist + - key: istio.io/rev + operator: In + values: + - {{ .Values.revision }} + {{- else }} + matchLabels: + "sidecar.istio.io/inject": "true" + {{- end }} + {{- end }} +{{- else }} + +{{- /* Set up the selectors. First section is for revision, rest is for "default" revision */}} +{{- if .Values.revision }} + +{{- /* Case 1: namespace selector matches, and object doesn't disable */}} +{{- /* Note: if both revision and legacy selector, we give precedence to the legacy one */}} +{{- include "core" (mergeOverwrite (deepCopy .) (dict "Prefix" "namespace.") ) }} + namespaceSelector: + matchExpressions: + - key: istio.io/rev + operator: In + values: + - "{{ .Values.revision }}" + - key: istio-injection + operator: DoesNotExist + objectSelector: + matchExpressions: + - key: sidecar.istio.io/inject + operator: NotIn + values: + - "false" + +{{- /* Case 2: No namespace selector, but object selects our revision (and doesn't disable) */}} +{{- include "core" (mergeOverwrite (deepCopy .) (dict "Prefix" "object.") ) }} + namespaceSelector: + matchExpressions: + - key: istio.io/rev + operator: DoesNotExist + - key: istio-injection + operator: DoesNotExist + objectSelector: + matchExpressions: + - key: sidecar.istio.io/inject + operator: NotIn + values: + - "false" + - key: istio.io/rev + operator: In + values: + - "{{ .Values.revision }}" + +{{- else }} +{{- /* "default" revision */}} + +{{- /* Case 1: Namespace selector enabled, and object selector is not injected */}} +{{- include "core" (mergeOverwrite (deepCopy .) (dict "Prefix" "namespace.") ) }} + namespaceSelector: + matchExpressions: + - key: istio-injection + operator: In + values: + - enabled + objectSelector: + matchExpressions: + - key: sidecar.istio.io/inject + operator: NotIn + values: + - "false" + +{{- /* Case 2: no namespace label, but object selector is enabled (and revision label is not, which has priority) */}} +{{- include "core" (mergeOverwrite (deepCopy .) (dict "Prefix" "object.") ) }} + namespaceSelector: + matchExpressions: + - key: istio-injection + operator: DoesNotExist + - key: istio.io/rev + operator: DoesNotExist + objectSelector: + matchExpressions: + - key: sidecar.istio.io/inject + operator: In + values: + - "true" + - key: istio.io/rev + operator: DoesNotExist + {{- if .Values.sidecarInjectorWebhook.enableNamespacesByDefault }} - matchExpressions: - - key: name - operator: NotIn - values: - - {{ .Release.Namespace }} - - key: istio-injection - operator: NotIn - values: - - disabled - - key: istio-env - operator: DoesNotExist - - key: istio.io/rev - operator: DoesNotExist -{{- else if .Values.revision }} - matchExpressions: - - key: istio-injection - operator: DoesNotExist - - key: istio.io/rev - operator: In - values: - - {{ .Values.revision }} -{{- else }} - matchLabels: - istio-injection: enabled +{{- /* Special case 3: no labels at all */}} +{{- include "core" (mergeOverwrite (deepCopy .) (dict "Prefix" "auto.") ) }} + namespaceSelector: + matchExpressions: + - key: istio-injection + operator: DoesNotExist + - key: istio.io/rev + operator: DoesNotExist + objectSelector: + matchExpressions: + - key: sidecar.istio.io/inject + operator: DoesNotExist + - key: istio.io/rev + operator: DoesNotExist {{- end }} -{{- if .Values.sidecarInjectorWebhook.objectSelector.enabled }} - objectSelector: -{{- if .Values.sidecarInjectorWebhook.objectSelector.autoInject }} - matchExpressions: - - key: "sidecar.istio.io/inject" - operator: NotIn - values: - - "false" -{{- else if .Values.revision }} - matchExpressions: - - key: "sidecar.istio.io/inject" - operator: DoesNotExist - - key: istio.io/rev - operator: In - values: - - {{ .Values.revision }} -{{- else }} - matchLabels: - "sidecar.istio.io/inject": "true" + {{- end }} {{- end }} {{- end }} diff --git a/charts/kubezero-istio/charts/istio-discovery/templates/telemetryv2_1.8.yaml b/charts/kubezero-istio/charts/istio-discovery/templates/telemetryv2_1.9.yaml similarity index 86% rename from charts/kubezero-istio/charts/istio-discovery/templates/telemetryv2_1.8.yaml rename to charts/kubezero-istio/charts/istio-discovery/templates/telemetryv2_1.9.yaml index f878321..b1db1b9 100644 --- a/charts/kubezero-istio/charts/istio-discovery/templates/telemetryv2_1.8.yaml +++ b/charts/kubezero-istio/charts/istio-discovery/templates/telemetryv2_1.9.yaml @@ -3,7 +3,7 @@ apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: - name: metadata-exchange-1.8{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} + name: metadata-exchange-1.9{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} {{- if .Values.meshConfig.rootNamespace }} namespace: {{ .Values.meshConfig.rootNamespace }} {{- else }} @@ -19,11 +19,11 @@ spec: match: context: SIDECAR_INBOUND proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' listener: filterChain: filter: - name: "envoy.http_connection_manager" + name: "envoy.filters.network.http_connection_manager" patch: operation: INSERT_BEFORE value: @@ -54,11 +54,11 @@ spec: match: context: SIDECAR_OUTBOUND proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' listener: filterChain: filter: - name: "envoy.http_connection_manager" + name: "envoy.filters.network.http_connection_manager" patch: operation: INSERT_BEFORE value: @@ -89,11 +89,11 @@ spec: match: context: GATEWAY proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' listener: filterChain: filter: - name: "envoy.http_connection_manager" + name: "envoy.filters.network.http_connection_manager" patch: operation: INSERT_BEFORE value: @@ -108,15 +108,23 @@ spec: value: | {} vm_config: + {{- if .Values.telemetry.v2.metadataExchange.wasmEnabled }} + runtime: envoy.wasm.runtime.v8 + allow_precompiled: true + code: + local: + filename: /etc/istio/extensions/metadata-exchange-filter.compiled.wasm + {{- else }} runtime: envoy.wasm.runtime.null code: local: inline_string: envoy.wasm.metadata_exchange + {{- end }} --- apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: - name: tcp-metadata-exchange-1.8{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} + name: tcp-metadata-exchange-1.9{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} {{- if .Values.meshConfig.rootNamespace }} namespace: {{ .Values.meshConfig.rootNamespace }} {{- else }} @@ -130,7 +138,7 @@ spec: match: context: SIDECAR_INBOUND proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' listener: {} patch: operation: INSERT_BEFORE @@ -145,7 +153,7 @@ spec: match: context: SIDECAR_OUTBOUND proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' cluster: {} patch: operation: MERGE @@ -161,7 +169,7 @@ spec: match: context: GATEWAY proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' cluster: {} patch: operation: MERGE @@ -179,7 +187,7 @@ spec: apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: - name: stats-filter-1.8{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} + name: stats-filter-1.9{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} {{- if .Values.meshConfig.rootNamespace }} namespace: {{ .Values.meshConfig.rootNamespace }} {{- else }} @@ -193,13 +201,13 @@ spec: match: context: SIDECAR_OUTBOUND proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' listener: filterChain: filter: - name: "envoy.http_connection_manager" + name: "envoy.filters.network.http_connection_manager" subFilter: - name: "envoy.router" + name: "envoy.filters.http.router" patch: operation: INSERT_BEFORE value: @@ -215,7 +223,8 @@ spec: value: | {{- if not .Values.telemetry.v2.prometheus.configOverride.outboundSidecar }} { - {{- if .Values.global.multiCluster.clusterName }} + "debug": "false", + "stat_prefix": "istio", "metrics": [ { "dimensions": { @@ -224,7 +233,6 @@ spec: } } ] - {{- end }} } {{- else }} {{ toJson .Values.telemetry.v2.prometheus.configOverride.outboundSidecar | indent 18 }} @@ -247,13 +255,13 @@ spec: match: context: SIDECAR_INBOUND proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' listener: filterChain: filter: - name: "envoy.http_connection_manager" + name: "envoy.filters.network.http_connection_manager" subFilter: - name: "envoy.router" + name: "envoy.filters.http.router" patch: operation: INSERT_BEFORE value: @@ -269,16 +277,16 @@ spec: value: | {{- if not .Values.telemetry.v2.prometheus.configOverride.inboundSidecar }} { - {{- if .Values.global.multiCluster.clusterName }} + "debug": "false", + "stat_prefix": "istio", "metrics": [ { "dimensions": { - "source_cluster": "downstream_peer.cluster_id", - "destination_cluster": "node.metadata['CLUSTER_ID']" + "destination_cluster": "node.metadata['CLUSTER_ID']", + "source_cluster": "downstream_peer.cluster_id" } } ] - {{- end }} } {{- else }} {{ toJson .Values.telemetry.v2.prometheus.configOverride.inboundSidecar | indent 18 }} @@ -301,13 +309,13 @@ spec: match: context: GATEWAY proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' listener: filterChain: filter: - name: "envoy.http_connection_manager" + name: "envoy.filters.network.http_connection_manager" subFilter: - name: "envoy.router" + name: "envoy.filters.http.router" patch: operation: INSERT_BEFORE value: @@ -323,7 +331,9 @@ spec: value: | {{- if not .Values.telemetry.v2.prometheus.configOverride.gateway }} { - "disable_host_header_fallback": true{{- if .Values.global.multiCluster.clusterName }}, + "debug": "false", + "stat_prefix": "istio", + "disable_host_header_fallback": true, "metrics": [ { "dimensions": { @@ -332,23 +342,30 @@ spec: } } ] - {{- end }} } {{- else }} {{ toJson .Values.telemetry.v2.prometheus.configOverride.gateway | indent 18 }} {{- end }} vm_config: vm_id: stats_outbound + {{- if .Values.telemetry.v2.prometheus.wasmEnabled }} + runtime: envoy.wasm.runtime.v8 + allow_precompiled: true + code: + local: + filename: /etc/istio/extensions/stats-filter.compiled.wasm + {{- else }} runtime: envoy.wasm.runtime.null code: local: inline_string: envoy.wasm.stats + {{- end }} --- # Note: tcp stats filter is wasm enabled only in sidecars. apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: - name: tcp-stats-filter-1.8{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} + name: tcp-stats-filter-1.9{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} {{- if .Values.meshConfig.rootNamespace }} namespace: {{ .Values.meshConfig.rootNamespace }} {{- else }} @@ -362,11 +379,11 @@ spec: match: context: SIDECAR_INBOUND proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' listener: filterChain: filter: - name: "envoy.tcp_proxy" + name: "envoy.filters.network.tcp_proxy" patch: operation: INSERT_BEFORE value: @@ -382,16 +399,16 @@ spec: value: | {{- if not .Values.telemetry.v2.prometheus.configOverride.inboundSidecar }} { - {{- if .Values.global.multiCluster.clusterName }} + "debug": "false", + "stat_prefix": "istio", "metrics": [ { "dimensions": { - "source_cluster": "downstream_peer.cluster_id", - "destination_cluster": "node.metadata['CLUSTER_ID']" + "destination_cluster": "node.metadata['CLUSTER_ID']", + "source_cluster": "downstream_peer.cluster_id" } } ] - {{- end }} } {{- else }} {{ toJson .Values.telemetry.v2.prometheus.configOverride.inboundSidecar | indent 18 }} @@ -414,11 +431,11 @@ spec: match: context: SIDECAR_OUTBOUND proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' listener: filterChain: filter: - name: "envoy.tcp_proxy" + name: "envoy.filters.network.tcp_proxy" patch: operation: INSERT_BEFORE value: @@ -434,7 +451,8 @@ spec: value: | {{- if not .Values.telemetry.v2.prometheus.configOverride.outboundSidecar }} { - {{- if .Values.global.multiCluster.clusterName }} + "debug": "false", + "stat_prefix": "istio", "metrics": [ { "dimensions": { @@ -443,7 +461,6 @@ spec: } } ] - {{- end }} } {{- else }} {{ toJson .Values.telemetry.v2.prometheus.configOverride.outboundSidecar | indent 18 }} @@ -466,11 +483,11 @@ spec: match: context: GATEWAY proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' listener: filterChain: filter: - name: "envoy.tcp_proxy" + name: "envoy.filters.network.tcp_proxy" patch: operation: INSERT_BEFORE value: @@ -486,7 +503,8 @@ spec: value: | {{- if not .Values.telemetry.v2.prometheus.configOverride.gateway }} { - {{- if .Values.global.multiCluster.clusterName }} + "debug": "false", + "stat_prefix": "istio", "metrics": [ { "dimensions": { @@ -495,26 +513,31 @@ spec: } } ] - {{- end }} } {{- else }} {{ toJson .Values.telemetry.v2.prometheus.configOverride.gateway | indent 18 }} {{- end }} vm_config: vm_id: tcp_stats_outbound + {{- if .Values.telemetry.v2.prometheus.wasmEnabled }} + runtime: envoy.wasm.runtime.v8 + allow_precompiled: true + code: + local: + filename: /etc/istio/extensions/stats-filter.compiled.wasm + {{- else }} runtime: envoy.wasm.runtime.null code: local: inline_string: "envoy.wasm.stats" + {{- end }} --- - {{- end }} - {{- if .Values.telemetry.v2.stackdriver.enabled }} apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: - name: stackdriver-filter-1.8{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} + name: stackdriver-filter-1.9{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} {{- if .Values.meshConfig.rootNamespace }} namespace: {{ .Values.meshConfig.rootNamespace }} {{- else }} @@ -529,13 +552,13 @@ spec: match: context: SIDECAR_OUTBOUND proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' listener: filterChain: filter: - name: "envoy.http_connection_manager" + name: "envoy.filters.network.http_connection_manager" subFilter: - name: "envoy.router" + name: "envoy.filters.http.router" patch: operation: INSERT_BEFORE value: @@ -564,13 +587,13 @@ spec: match: context: SIDECAR_INBOUND proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' listener: filterChain: filter: - name: "envoy.http_connection_manager" + name: "envoy.filters.network.http_connection_manager" subFilter: - name: "envoy.router" + name: "envoy.filters.http.router" patch: operation: INSERT_BEFORE value: @@ -598,13 +621,13 @@ spec: match: context: GATEWAY proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' listener: filterChain: filter: - name: "envoy.http_connection_manager" + name: "envoy.filters.network.http_connection_manager" subFilter: - name: "envoy.router" + name: "envoy.filters.http.router" patch: operation: INSERT_BEFORE value: @@ -629,11 +652,10 @@ spec: code: local: { inline_string: envoy.wasm.null.stackdriver } --- - apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: - name: tcp-stackdriver-filter-1.8{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} + name: tcp-stackdriver-filter-1.9{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} {{- if .Values.meshConfig.rootNamespace }} namespace: {{ .Values.meshConfig.rootNamespace }} {{- else }} @@ -648,11 +670,11 @@ spec: match: context: SIDECAR_OUTBOUND proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' listener: filterChain: filter: - name: "envoy.tcp_proxy" + name: "envoy.filters.network.tcp_proxy" patch: operation: INSERT_BEFORE value: @@ -681,11 +703,11 @@ spec: match: context: SIDECAR_INBOUND proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' listener: filterChain: filter: - name: "envoy.tcp_proxy" + name: "envoy.filters.network.tcp_proxy" patch: operation: INSERT_BEFORE value: @@ -713,11 +735,11 @@ spec: match: context: GATEWAY proxy: - proxyVersion: '^1\.8.*' + proxyVersion: '^1\.9.*' listener: filterChain: filter: - name: "envoy.tcp_proxy" + name: "envoy.filters.network.tcp_proxy" patch: operation: INSERT_BEFORE value: @@ -746,7 +768,7 @@ spec: apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: - name: stackdriver-sampling-accesslog-filter-1.8{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} + name: stackdriver-sampling-accesslog-filter-1.9{{- if not (eq .Values.revision "") }}-{{ .Values.revision }}{{- end }} {{- if .Values.meshConfig.rootNamespace }} namespace: {{ .Values.meshConfig.rootNamespace }} {{- else }} @@ -760,11 +782,11 @@ spec: match: context: SIDECAR_INBOUND proxy: - proxyVersion: '1\.8.*' + proxyVersion: '1\.9.*' listener: filterChain: filter: - name: "envoy.http_connection_manager" + name: "envoy.filters.network.http_connection_manager" subFilter: name: "istio.stackdriver" patch: diff --git a/charts/kubezero-istio/charts/istio-discovery/values.yaml b/charts/kubezero-istio/charts/istio-discovery/values.yaml index ac59dbd..6491504 100644 --- a/charts/kubezero-istio/charts/istio-discovery/values.yaml +++ b/charts/kubezero-istio/charts/istio-discovery/values.yaml @@ -65,6 +65,10 @@ pilot: sidecarInjectorWebhook: + # If enabled, the legacy webhook selection logic will be used. This relies on filtering of webhook + # requests in Istiod, rather than at the webhook selection level. + # This is option is intended for migration purposes only and will be removed in Istio 1.10. + useLegacySelectors: true # You can use the field called alwaysInjectSelector and neverInjectSelector which will always inject the sidecar or # always skip the injection on pods that match that label selector, regardless of the global policy. # See https://istio.io/docs/setup/kubernetes/additional-setup/sidecar-injection/#more-control-adding-exceptions @@ -90,13 +94,41 @@ sidecarInjectorWebhook: # Only one environment should have this enabled. enableNamespacesByDefault: false - # Enable objectSelector to filter out pods with no need for sidecar before calling istio-sidecar-injector. - # It is disabled by default since this function will only work after k8s v1.15. + # Enable objectSelector to filter out pods with no need for sidecar before calling istiod. + # It is enabled by default as the minimum supported Kubernetes version is 1.15+ objectSelector: - enabled: false + enabled: true autoInject: true rewriteAppHTTPProbe: true + + # Templates defines a set of custom injection templates that can be used. For example, defining: + # + # templates: + # hello: | + # metadata: + # labels: + # hello: world + # + # Then starting a pod with the `inject.istio.io/templates: hello` annotation, will result in the pod + # being injected with the hello=world labels. + # This is intended for advanced configuration only; most users should use the built in template + templates: {} + + # Default templates specifies a set of default templates that are used in sidecar injection. + # By default, a template `sidecar` is always provided, which contains the template of default sidecar. + # To inject other additional templates, define it using the `templates` option, and add it to + # the default templates list. + # For example: + # + # templates: + # hello: | + # metadata: + # labels: + # hello: world + # + # defaultTemplates: ["sidecar", "hello"] + defaultTemplates: [] istiodRemote: # Sidecar injector mutating webhook configuration url # For example: https://$remotePilotAddress:15017/inject @@ -150,26 +182,20 @@ ownerName: "" # meshConfig defines runtime configuration of components, including Istiod and istio-agent behavior # See https://istio.io/docs/reference/config/istio.mesh.v1alpha1/ for all available options meshConfig: - + enablePrometheusMerge: true # Config for the default ProxyConfig. # Initially using directly the proxy metadata - can also be activated using annotations # on the pod. This is an unsupported low-level API, pending review and decisions on # enabling the feature. Enabling the DNS listener is safe - and allows further testing # and gradual adoption by setting capture only on specific workloads. It also allows # VMs to use other DNS options, like dnsmasq or unbound. - defaultConfig: - proxyMetadata: - # If empty, agent will not start :15013 DNS listener and will not attempt - # to connect to Istiod DNS-TLS. This will also disable the core dns sidecar in - # istiod and the dns-over-tls listener. - # DNS_AGENT: DNS-TLS - DNS_AGENT: "" # The namespace to treat as the administrative root namespace for Istio configuration. # When processing a leaf namespace Istio will search for declarations in that namespace first # and if none are found it will search in the root namespace. Any matching declaration found in the root namespace # is processed as if it were declared in the leaf namespace. - rootNamespace: "istio-system" + + rootNamespace: # TODO: the intent is to eventually have this enabled by default when security is used. # It is not clear if user should normally need to configure - the metadata is typically @@ -180,6 +206,8 @@ meshConfig: # No hurry to do this in 1.6, we're trying to prove the code. global: + # Used to locate istiod. + istioNamespace: istio-system # enable pod disruption budget for the control plane, which is used to # ensure Istio control plane components are gradually upgraded or recovered. defaultPodDisruptionBudget: @@ -199,15 +227,12 @@ global: # cpu: 100m # memory: 128Mi - # Used to locate istiod. - istioNamespace: istio-system - # Default hub for Istio images. # Releases are published to docker hub under 'istio' project. # Dev builds from prow are on gcr.io - hub: gcr.io/istio-testing + hub: docker.io/istio # Default tag for Istio images. - tag: latest + tag: 1.9.1 # Specify image pull policy if default behavior isn't desired. # Default behavior: latest images will be Always else IfNotPresent. @@ -353,12 +378,11 @@ global: # The customized CA address to retrieve certificates for the pods in the cluster. # CSR clients such as the Istio Agent and ingress gateways can use this to specify the CA endpoint. + # If not set explicitly, default to the Istio discovery address. caAddress: "" # External istiod controls all remote clusters: disabled by default externalIstiod: false - # Central istiod controls all remote clusters: disabled by default - centralIstiod: false # Configure the policy for validating JWT. # Currently, two options are supported: "third-party-jwt" and "first-party-jwt". @@ -483,3 +507,6 @@ global: # Deprecated, use meshConfig.trustDomain trustDomain: "" +base: + # For istioctl usage to disable istio config crds in base + enableIstioConfigCRDs: true diff --git a/charts/kubezero-istio/update.sh b/charts/kubezero-istio/update.sh index 3993833..db27e79 100755 --- a/charts/kubezero-istio/update.sh +++ b/charts/kubezero-istio/update.sh @@ -1,7 +1,7 @@ #!/bin/bash set -ex -export ISTIO_VERSION=1.8.1 +export ISTIO_VERSION=1.9.1 if [ ! -d istio-$ISTIO_VERSION ]; then NAME="istio-$ISTIO_VERSION" @@ -16,10 +16,10 @@ cp -r istio-${ISTIO_VERSION}/manifests/charts/base charts/ cp -r istio-${ISTIO_VERSION}/manifests/charts/istio-control/istio-discovery charts/ # Patch for istiod to control plane -patch -p3 -i istio-discovery.patch +patch -p3 -i istio-discovery.patch --no-backup-if-mismatch # remove unused old telemetry filters -rm -f charts/istio-discovery/templates/telemetryv2_1.[67].yaml +rm -f charts/istio-discovery/templates/telemetryv2_1.[678].yaml # Ingress charts rm -rf ../kubezero-istio-ingress/charts/istio-* @@ -30,8 +30,8 @@ cp -r istio-${ISTIO_VERSION}/manifests/charts/gateways/istio-ingress ../kubezero sed -i -e 's/name: istio-ingress/name: istio-private-ingress/' ../kubezero-istio-ingress/charts/istio-private-ingress/Chart.yaml # Patch for ingress for extended termination grace period -patch -i ingress-terminationgraceperiod.patch ../kubezero-istio-ingress/charts/istio-ingress/templates/deployment.yaml -patch -i ingress-terminationgraceperiod.patch ../kubezero-istio-ingress/charts/istio-private-ingress/templates/deployment.yaml +patch -i ingress-terminationgraceperiod.patch ../kubezero-istio-ingress/charts/istio-ingress/templates/deployment.yaml --no-backup-if-mismatch +patch -i ingress-terminationgraceperiod.patch ../kubezero-istio-ingress/charts/istio-private-ingress/templates/deployment.yaml --no-backup-if-mismatch # Get matching istioctl [ -x istioctl ] && [ "$(./istioctl version --remote=false)" == $ISTIO_VERSION ] || { curl -sL https://github.com/istio/istio/releases/download/${ISTIO_VERSION}/istioctl-${ISTIO_VERSION}-linux-amd64.tar.gz | tar xz; chmod +x istioctl; } diff --git a/charts/kubezero-istio/values.yaml b/charts/kubezero-istio/values.yaml index 2b58c92..398dd39 100644 --- a/charts/kubezero-istio/values.yaml +++ b/charts/kubezero-istio/values.yaml @@ -1,6 +1,6 @@ global: - hub: docker.io/istio - tag: 1.8.0 + # hub: docker.io/istio + # tag: 1.9.0 logAsJson: true jwtPolicy: first-party-jwt diff --git a/charts/kubezero-kiam/Chart.yaml b/charts/kubezero-kiam/Chart.yaml index 8503fc5..b3d14ed 100644 --- a/charts/kubezero-kiam/Chart.yaml +++ b/charts/kubezero-kiam/Chart.yaml @@ -2,8 +2,8 @@ apiVersion: v2 name: kubezero-kiam description: KubeZero Umbrella Chart for Kiam type: application -version: 0.2.12 -appVersion: 3.6 +version: 0.3.2 +appVersion: "4.0" home: https://kubezero.com icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png keywords: @@ -16,7 +16,7 @@ dependencies: version: ">= 0.1.3" repository: https://zero-down-time.github.io/kubezero/ - name: kiam - version: 5.9.0 + version: 6.0.0 repository: https://uswitch.github.io/kiam-helm-charts/charts/ condition: kiam.enabled -kubeVersion: ">= 1.16.0" +kubeVersion: ">= 1.18.0" diff --git a/charts/kubezero-kiam/README.md b/charts/kubezero-kiam/README.md index a71327a..61c6a05 100644 --- a/charts/kubezero-kiam/README.md +++ b/charts/kubezero-kiam/README.md @@ -1,6 +1,6 @@ # kubezero-kiam -![Version: 0.2.12](https://img.shields.io/badge/Version-0.2.12-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 3.6](https://img.shields.io/badge/AppVersion-3.6-informational?style=flat-square) +![Version: 0.3.2](https://img.shields.io/badge/Version-0.3.2-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 4.0](https://img.shields.io/badge/AppVersion-4.0-informational?style=flat-square) KubeZero Umbrella Chart for Kiam @@ -14,11 +14,11 @@ KubeZero Umbrella Chart for Kiam ## Requirements -Kubernetes: `>= 1.16.0` +Kubernetes: `>= 1.18.0` | Repository | Name | Version | |------------|------|---------| -| https://uswitch.github.io/kiam-helm-charts/charts/ | kiam | 5.9.0 | +| https://uswitch.github.io/kiam-helm-charts/charts/ | kiam | 6.0.0 | | https://zero-down-time.github.io/kubezero/ | kubezero-lib | >= 0.1.3 | ## KubeZero default configuration @@ -44,6 +44,7 @@ By default all access to the meta-data service is blocked, expect for: | Key | Type | Default | Description | |-----|------|---------|-------------| | annotateKubeSystemNameSpace | bool | `false` | | +| kiam.agent.allowRouteRegexp | string | `"^/latest/(meta-data/instance-id|dynamic)"` | | | kiam.agent.gatewayTimeoutCreation | string | `"5s"` | | | kiam.agent.host.interface | string | `"cali+"` | | | kiam.agent.host.iptables | bool | `false` | | @@ -52,7 +53,6 @@ By default all access to the meta-data service is blocked, expect for: | kiam.agent.prometheus.servicemonitor.enabled | bool | `false` | | | kiam.agent.prometheus.servicemonitor.interval | string | `"30s"` | | | kiam.agent.prometheus.servicemonitor.labels.release | string | `"metrics"` | | -| kiam.agent.resources.limits.cpu | string | `"50m"` | | | kiam.agent.resources.limits.memory | string | `"20Mi"` | | | kiam.agent.resources.requests.cpu | string | `"50m"` | | | kiam.agent.resources.requests.memory | string | `"20Mi"` | | @@ -64,7 +64,6 @@ By default all access to the meta-data service is blocked, expect for: | kiam.agent.tolerations[0].effect | string | `"NoSchedule"` | | | kiam.agent.tolerations[0].key | string | `"node-role.kubernetes.io/master"` | | | kiam.agent.updateStrategy | string | `"RollingUpdate"` | | -| kiam.agent.whiteListRouteRegexp | string | `"^/latest/(meta-data/instance-id|dynamic)"` | | | kiam.enabled | bool | `true` | | | kiam.server.assumeRoleArn | string | `""` | kiam server IAM role to assume, required as we run the agents next to the servers normally, eg. arn:aws:iam::123456789012:role/kiam-server-role | | kiam.server.deployment.enabled | bool | `true` | | @@ -75,9 +74,8 @@ By default all access to the meta-data service is blocked, expect for: | kiam.server.prometheus.servicemonitor.enabled | bool | `false` | | | kiam.server.prometheus.servicemonitor.interval | string | `"30s"` | | | kiam.server.prometheus.servicemonitor.labels.release | string | `"metrics"` | | -| kiam.server.resources.limits.cpu | string | `"100m"` | | -| kiam.server.resources.limits.memory | string | `"50Mi"` | | -| kiam.server.resources.requests.cpu | string | `"100m"` | | +| kiam.server.resources.limits.memory | string | `"100Mi"` | | +| kiam.server.resources.requests.cpu | string | `"50m"` | | | kiam.server.resources.requests.memory | string | `"50Mi"` | | | kiam.server.service.port | int | `6444` | | | kiam.server.service.targetPort | int | `6444` | | diff --git a/charts/kubezero-kiam/values.yaml b/charts/kubezero-kiam/values.yaml index 3c5b38d..39eab1d 100644 --- a/charts/kubezero-kiam/values.yaml +++ b/charts/kubezero-kiam/values.yaml @@ -22,10 +22,10 @@ kiam: resources: requests: memory: "50Mi" - cpu: "100m" + cpu: "50m" limits: - memory: "50Mi" - cpu: "100m" + memory: "100Mi" + # cpu: "300m" tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule @@ -48,7 +48,7 @@ kiam: host: iptables: false interface: "cali+" - whiteListRouteRegexp: '^/latest/(meta-data/instance-id|dynamic)' + allowRouteRegexp: '^/latest/(meta-data/instance-id|dynamic)' sslCertHostPath: /etc/ssl/certs tlsSecret: kiam-agent-tls tlsCerts: @@ -61,7 +61,7 @@ kiam: cpu: "50m" limits: memory: "20Mi" - cpu: "50m" + # cpu: "50m" tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule diff --git a/charts/kubezero-local-volume-provisioner/Chart.yaml b/charts/kubezero-local-volume-provisioner/Chart.yaml index 303887f..35b4a86 100644 --- a/charts/kubezero-local-volume-provisioner/Chart.yaml +++ b/charts/kubezero-local-volume-provisioner/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: kubezero-local-volume-provisioner description: KubeZero Umbrella Chart for local-static-provisioner type: application -version: 0.1.0 +version: 0.1.1 appVersion: 2.3.4 home: https://kubezero.com icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png diff --git a/charts/kubezero-local-volume-provisioner/README.md b/charts/kubezero-local-volume-provisioner/README.md index a09236e..9fa10fe 100644 --- a/charts/kubezero-local-volume-provisioner/README.md +++ b/charts/kubezero-local-volume-provisioner/README.md @@ -1,6 +1,6 @@ # kubezero-local-volume-provisioner -![Version: 0.1.0](https://img.shields.io/badge/Version-0.1.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 2.3.4](https://img.shields.io/badge/AppVersion-2.3.4-informational?style=flat-square) +![Version: 0.1.1](https://img.shields.io/badge/Version-0.1.1-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 2.3.4](https://img.shields.io/badge/AppVersion-2.3.4-informational?style=flat-square) KubeZero Umbrella Chart for local-static-provisioner @@ -29,7 +29,8 @@ Kubernetes: `>= 1.16.0` | local-static-provisioner.classes[0].hostDir | string | `"/mnt/disks"` | | | local-static-provisioner.classes[0].name | string | `"local-sc-xfs"` | | | local-static-provisioner.common.namespace | string | `"kube-system"` | | -| local-static-provisioner.daemonset.nodeSelector."node.kubernetes.io/localVolume" | string | `"present"` | | +| local-static-provisioner.daemonset.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key | string | `"node.kubernetes.io/localVolumes"` | | +| local-static-provisioner.daemonset.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].operator | string | `"Exists"` | | | local-static-provisioner.prometheus.operator.enabled | bool | `false` | | ## KubeZero default configuration diff --git a/charts/kubezero-local-volume-provisioner/values.yaml b/charts/kubezero-local-volume-provisioner/values.yaml index c01ce93..7d60c09 100644 --- a/charts/kubezero-local-volume-provisioner/values.yaml +++ b/charts/kubezero-local-volume-provisioner/values.yaml @@ -5,8 +5,13 @@ local-static-provisioner: - name: local-sc-xfs hostDir: /mnt/disks daemonset: - nodeSelector: - node.kubernetes.io/localVolume: present + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node.kubernetes.io/localVolumes + operator: Exists prometheus: operator: enabled: false diff --git a/charts/kubezero-logging/Chart.yaml b/charts/kubezero-logging/Chart.yaml index 3776ccd..a8225fe 100644 --- a/charts/kubezero-logging/Chart.yaml +++ b/charts/kubezero-logging/Chart.yaml @@ -2,8 +2,8 @@ apiVersion: v2 name: kubezero-logging description: KubeZero Umbrella Chart for complete EFK stack type: application -version: 0.5.2 -appVersion: 1.3.1 +version: 0.6.2 +appVersion: 1.4.1 home: https://kubezero.com icon: https://cdn.zero-downtime.net/assets/kubezero/logo-small-64.png keywords: @@ -19,13 +19,13 @@ dependencies: version: ">= 0.1.3" repository: https://zero-down-time.github.io/kubezero/ - name: eck-operator - version: 1.3.1 + version: 1.4.1 repository: https://helm.elastic.co condition: eck-operator.enabled - name: fluentd - version: 2.5.3 + version: 0.2.2 condition: fluentd.enabled - name: fluent-bit - version: 0.7.14 + version: 0.12.3 condition: fluent-bit.enabled kubeVersion: ">= 1.16.0" diff --git a/charts/kubezero-logging/README.md b/charts/kubezero-logging/README.md index 9f47d63..89edc99 100644 --- a/charts/kubezero-logging/README.md +++ b/charts/kubezero-logging/README.md @@ -1,6 +1,6 @@ # kubezero-logging -![Version: 0.5.1](https://img.shields.io/badge/Version-0.5.1-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.3.1](https://img.shields.io/badge/AppVersion-1.3.1-informational?style=flat-square) +![Version: 0.6.2](https://img.shields.io/badge/Version-0.6.2-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.4.1](https://img.shields.io/badge/AppVersion-1.4.1-informational?style=flat-square) KubeZero Umbrella Chart for complete EFK stack @@ -18,8 +18,9 @@ Kubernetes: `>= 1.16.0` | Repository | Name | Version | |------------|------|---------| -| https://helm.elastic.co | eck-operator | 1.3.1 | -| https://kubernetes-charts.storage.googleapis.com/ | fluentd | 2.5.1 | +| | fluent-bit | 0.12.3 | +| | fluentd | 0.2.2 | +| https://helm.elastic.co | eck-operator | 1.4.1 | | https://zero-down-time.github.io/kubezero/ | kubezero-lib | >= 0.1.3 | ## Changes from upstream @@ -66,42 +67,49 @@ Kubernetes: `>= 1.16.0` | es.prometheus | bool | `false` | | | es.s3Snapshot.enabled | bool | `false` | | | es.s3Snapshot.iamrole | string | `""` | | -| fluent-bit.config.flushInterval | int | `1` | | +| fluent-bit.config.customParsers | string | `"[PARSER]\n Name cri-log\n Format regex\n Regex ^(?