KubeZero/charts/kubezero-ci/charts/jenkins/README.md

Jenkins

Jenkins is the leading open source automation server, Jenkins provides over 1800 plugins to support building, deploying and automating any project.

This chart installs a Jenkins server which spawns agents on Kubernetes utilizing the Jenkins Kubernetes plugin.

Inspired by the awesome work of Carlos Sanchez.

Get Repository Info

helm repo add jenkins https://charts.jenkins.io
helm repo update

See helm repo for command documentation.

Install Chart

# Helm 3
$ helm install [RELEASE_NAME] jenkins/jenkins [flags]

See configuration below.

See helm install for command documentation.

Uninstall Chart

# Helm 3
$ helm uninstall [RELEASE_NAME]

This removes all the Kubernetes components associated with the chart and deletes the release.

See helm uninstall for command documentation.

Upgrade Chart

# Helm 3
$ helm upgrade [RELEASE_NAME] jenkins/jenkins [flags]

See helm upgrade for command documentation.

Visit the chart's CHANGELOG to view the chart's release history. For migration between major version check migration guide.

Building weekly releases

The default charts target Long-Term-Support (LTS) releases of Jenkins. To use other versions the easiest way is to update the image tag to the version you want. You can also rebuild the chart if you want the appVersion field to match.

Configuration

See Customizing the Chart Before Installing. To see all configurable options with detailed comments, visit the chart's values.yaml, or run these configuration commands:

# Helm 3
$ helm show values jenkins/jenkins

For a summary of all configurable options, see VALUES_SUMMARY.md.

Configure Security Realm and Authorization Strategy

This chart configured a securityRealm and authorizationStrategy as shown below:

controller:
  JCasC:
    securityRealm: |-
      local:
        allowsSignup: false
        enableCaptcha: false
        users:
        - id: "${chart-admin-username}"
          name: "Jenkins Admin"
          password: "${chart-admin-password}"      
    authorizationStrategy: |-
      loggedInUsersCanDoAnything:
        allowAnonymousRead: false      

With the configuration above there is only a single user. This is fine for getting started quickly, but it needs to be adjusted for any serious environment.

So you should adjust this to suite your needs. That could be using LDAP / OIDC / .. as authorization strategy and use globalMatrix as authorization strategy to configure more fine-grained permissions.

Consider using a custom image

This chart allows the user to specify plugins which should be installed. However, for production use cases one should consider to build a custom Jenkins image which has all required plugins pre-installed. This way you can be sure which plugins Jenkins is using when starting up and you avoid trouble in case of connectivity issues to the Jenkins update site.

The docker repository for the Jenkins image contains documentation how to do it.

Here is an example how that can be done:

FROM jenkins/jenkins:lts
RUN jenkins-plugin-cli --plugins kubernetes workflow-aggregator git configuration-as-code

NOTE: If you want a reproducible build then you should specify a non-floating tag for the image jenkins/jenkins:2.249.3 and specify plugin versions.

Once you built the image and pushed it to your registry you can specify it in your values file like this:

controller:
  image: "registry/my-jenkins"
  tag: "v1.2.3"
  installPlugins: false

Notice: installPlugins is set to false to disable plugin download. In this case, the image registry/my-jenkins:v1.2.3 must have the plugins specified as default value for the controller.installPlugins directive to ensure that the configuration side-car system works as expected.

In case you are using a private registry you can use 'imagePullSecretName' to specify the name of the secret to use when pulling the image:

controller:
  image: "registry/my-jenkins"
  tag: "v1.2.3"
  imagePullSecretName: registry-secret
  installPlugins: false

External URL Configuration

If you are using the ingress definitions provided by this chart via the controller.ingress block the configured hostname will be the ingress hostname starting with https:// or http:// depending on the tls configuration. The Protocol can be overwritten by specifying controller.jenkinsUrlProtocol.

If you are not using the provided ingress you can specify controller.jenkinsUrl to change the URL definition.

Configuration as Code

Jenkins Configuration as Code (JCasC) is now a standard component in the Jenkins project. To allow JCasC's configuration from the helm values, the plugin configuration-as-code must be installed in the Jenkins Controller's Docker image (which is the case by default as specified by the default value of the directive controller.installPlugins).

JCasc configuration is passed through Helm values under the key controller.JCasC. The section "Jenkins Configuration as Code (JCasC)" of the page "VALUES_SUMMARY.md" lists all the possible directives.

In particular, you may specify custom JCasC scripts by adding sub-key under the controller.JCasC.configScripts for each configuration area where each corresponds to a plugin or section of the UI.

The sub-keys (prior to | character) are only labels used to give the section a meaningful name. The only restriction is they must conform to RFC 1123 definition of a DNS label, so they may only contain lowercase letters, numbers, and hyphens.

Each key will become the name of a configuration yaml file on the controller in /var/jenkins_home/casc_configs (by default) and will be processed by the Configuration as Code Plugin during Jenkins startup.

The lines after each | become the content of the configuration yaml file.

The first line after this is a JCasC root element, e.g. jenkins, credentials, etc.

Best reference is the Documentation link here: https://<jenkins_url>/configuration-as-code.

The example below sets custom systemMessage:

controller:
  JCasC:
    configScripts:
      welcome-message: |
        jenkins:
          systemMessage: Welcome to our CI\CD server.        

More complex example that creates ldap settings:

controller:
  JCasC:
    configScripts:
      ldap-settings: |
        jenkins:
          securityRealm:
            ldap:
              configurations:
                - server: ldap.acme.com
                  rootDN: dc=acme,dc=uk
                  managerPasswordSecret: ${LDAP_PASSWORD}
                  groupMembershipStrategy:
                    fromUserRecord:
                      attributeName: "memberOf"        

Keep in mind that default configuration file already contains some values that you won't be able to override under configScripts section.

For example, you can not configure Jenkins URL and System Admin email address like this because of conflicting configuration error.

Incorrect:

controller:
  JCasC:
    configScripts:
      jenkins-url: |
        unclassified:
          location:
            url: https://example.com/jenkins
            adminAddress: example@mail.com        

Correct:

controller:
  jenkinsUrl: https://example.com/jenkins
  jenkinsAdminEmail: example@mail.com

Further JCasC examples can be found here.

Breaking out large Config as Code scripts

Jenkins Config as Code scripts can become quite large, and maintaining all of your scripts within one yaml file can be difficult. The Config as Code plugin itself suggests updating the CASC_JENKINS_CONFIG environment variable to be a comma separated list of paths for the plugin to traverse, picking up the yaml files as needed.
However, under the Jenkins helm chart, this CASC_JENKINS_CONFIG value is maintained through the templates. A better solution is to split your controller.JCasC.configScripts into separate values files, and provide each file during the helm install.

For example, you can have a values file (e.g values_main.yaml) that defines the values described in the VALUES_SUMMARY.md for your Jenkins configuration:

jenkins:
  controller:
    jenkinsUrlProtocol: https
    installPlugins: false
    ...

In a second file (e.g values_jenkins_casc.yaml), you can define a section of your config scripts:

jenkins:
  controller:
    JCasC:
      configScripts:
        jenkinsCasc:  |
          jenkins:
            disableRememberMe: false
            mode: NORMAL
            ...          

And keep extending your config scripts by creating more files (so not all config scripts are located in one yaml file for better maintenance):

values_jenkins_unclassified.yaml

jenkins:
  controller:
    JCasC:
      configScripts:
        unclassifiedCasc: |
          unclassified:
            ...          

When installing, you provide all relevant yaml files (e.g helm install -f values_main.yaml -f values_jenkins_casc.yaml -f values_jenkins_unclassified.yaml ...). Instead of updating the CASC_JENKINS_CONFIG environment variable to include multiple paths, multiple CasC yaml files will be created in the same path var/jenkins_home/casc_configs.

Config as Code With or Without Auto-Reload

Config as Code changes (to controller.JCasC.configScripts) can either force a new pod to be created and only be applied at next startup, or can be auto-reloaded on-the-fly. If you set controller.sidecars.configAutoReload.enabled to true, a second, auxiliary container will be installed into the Jenkins controller pod, known as a "sidecar". This watches for changes to configScripts, copies the content onto the Jenkins file-system and issues a POST to http://<jenkins_url>/reload-configuration-as-code with a pre-shared key. You can monitor this sidecar's logs using command kubectl logs <controller_pod> -c config-reload -f. If you want to enable auto-reload then you also need to configure rbac as the container which triggers the reload needs to watch the config maps:

controller:
  sidecars:
    configAutoReload:
      enabled: true
rbac:
  create: true

Allow Limited HTML Markup in User-Submitted Text

Some third-party systems (e.g. GitHub) use HTML-formatted data in their payload sent to a Jenkins webhook (e.g. URL of a pull-request being built). To display such data as processed HTML instead of raw text set controller.enableRawHtmlMarkupFormatter to true. This option requires installation of the OWASP Markup Formatter Plugin (antisamy-markup-formatter). This plugin is not installed by default but may be added to controller.additionalPlugins.

Change max connections to Kubernetes API

When using agents with containers other than JNLP, The kubernetes plugin will communicate with those containers using the Kubernetes API. this changes the maximum concurrent connections

agent:
  maxRequestsPerHostStr: "32"

This will change the configuration of the kubernetes "cloud" (as called by jenkins) that is created automatically as part of this helm chart.

Change container cleanup timeout API

For tasks that use very large images, this timeout can be increased to avoid early termination of the task while the Kubernetes pod is still deploying.

agent:
  retentionTimeout: "32"

This will change the configuration of the kubernetes "cloud" (as called by jenkins) that is created automatically as part of this helm chart.

Change seconds to wait for pod to be running

This will change how long Jenkins will wait (seconds) for pod to be in running state.

agent:
  waitForPodSec: "32"

This will change the configuration of the kubernetes "cloud" (as called by jenkins) that is created automatically as part of this helm chart.

Mounting Volumes into Agent Pods

Your Jenkins Agents will run as pods, and it's possible to inject volumes where needed:

agent:
  volumes:
  - type: Secret
    secretName: jenkins-mysecrets
    mountPath: /var/run/secrets/jenkins-mysecrets

The supported volume types are: ConfigMap, EmptyDir, HostPath, Nfs, PVC, Secret. Each type supports a different set of configurable attributes, defined by the corresponding Java class.

NetworkPolicy

To make use of the NetworkPolicy resources created by default, install a networking plugin that implements the Kubernetes NetworkPolicy spec.

Install helm chart with network policy enabled by setting networkPolicy.enabled to true.

You can use controller.networkPolicy.internalAgents and controller.networkPolicy.externalAgents stanzas for fine-grained controls over where internal/external agents can connect from. Internal ones are allowed based on pod labels and (optionally) namespaces, and external ones are allowed based on IP ranges.

Script approval list

controller.scriptApproval allows to pass function signatures that will be allowed in pipelines. Example:

controller:
  scriptApproval:
    - "method java.util.Base64$Decoder decode java.lang.String"
    - "new java.lang.String byte[]"
    - "staticMethod java.util.Base64 getDecoder"

Custom Labels

controller.serviceLabels can be used to add custom labels in jenkins-controller-svc.yaml. For example:

ServiceLabels:
  expose: true

Persistence

The Jenkins image stores persistence under /var/jenkins_home path of the container. A dynamically managed Persistent Volume Claim is used to keep the data across deployments, by default. This is known to work in GCE, AWS, and minikube. Alternatively, a previously configured Persistent Volume Claim can be used.

It is possible to mount several volumes using persistence.volumes and persistence.mounts parameters. See additional persistence values using configuration commands.

Existing PersistentVolumeClaim

1. Create the PersistentVolume
2. Create the PersistentVolumeClaim
3. Install the chart, setting persistence.existingClaim to PVC_NAME

Long Volume Attach/Mount Times

Certain volume type and filesystem format combinations may experience long attach/mount times, 10 or more minutes, when using fsGroup. This issue may result in the following entries in the pod's event history:

Warning  FailedMount  38m                kubelet, aks-default-41587790-2 Unable to attach or mount volumes: unmounted volumes=[jenkins-home], unattached volumes=[plugins plugin-dir jenkins-token-rmq2g sc-config-volume tmp jenkins-home jenkins-config secrets-dir]: timed out waiting for the condition

In these cases, experiment with replacing fsGroup with supplementalGroups in the pod's securityContext. This can be achieved by setting the controller.podSecurityContextOverride Helm chart value to something like:

controller:
  podSecurityContextOverride:
    runAsNonRoot: true
    runAsUser: 1000
    supplementalGroups: [1000]

This issue has been reported on azureDisk with ext4 and on Alibaba cloud.

Storage Class

It is possible to define which storage class to use, by setting persistence.storageClass to [customStorageClass]. If set to a dash (-), dynamic provisioning is disabled. If the storage class is set to null or left undefined (""), the default provisioner is used (gp2 on AWS, standard on GKE, AWS & OpenStack).

Additional Secrets

Additional secrets and Additional Existing Secrets, can be mounted into the Jenkins controller through the chart or created using controller.additionalSecrets or controller.additionalExistingSecrets.
A common use case might be identity provider credentials if using an external LDAP or OIDC-based identity provider. The secret may then be referenced in JCasC configuration (see JCasC configuration).

values.yaml controller section, referencing mounted secrets:

controller:
  # the 'name' and 'keyName' are concatenated with a '-' in between, so for example:
  # an existing secret "secret-credentials" and a key inside it named "github-password" should be used in Jcasc as ${secret-credentials-github-password}
  # 'name' and 'keyName' must be lowercase RFC 1123 label must consist of lower case alphanumeric characters or '-',
  # and must start and end with an alphanumeric character (e.g. 'my-name', or '123-abc')
  # existingSecret existing secret "secret-credentials" and a key inside it named "github-username" should be used in Jcasc as ${github-username}
  # When using existingSecret no need to specify the keyName under additionalExistingSecrets.
  existingSecret: secret-credentials
  
  additionalExistingSecrets:
    - name: secret-credentials
      keyName: github-username
    - name: secret-credentials
      keyName: github-password
    - name: secret-credentials
      keyName: token
  
  additionalSecrets:
    - name: client_id
      value: abc123
    - name: client_secret
      value: xyz999
  JCasC:
    securityRealm: |
      oic:
        clientId: ${client_id}
        clientSecret: ${client_secret}
        ...      
    configScripts:
      jenkins-casc-configs: |
        credentials:
          system:
            domainCredentials:
            - credentials:
              - string:
                  description: "github access token"
                  id: "github_app_token"
                  scope: GLOBAL
                  secret: ${secret-credentials-token}
              - usernamePassword:
                  description: "github access username password"
                  id: "github_username_pass"
                  password: ${secret-credentials-github-password}
                  scope: GLOBAL
                  username: ${secret-credentials-github-username}        

For more information, see JCasC documentation.

Secret Claims from HashiCorp Vault

It's possible for this chart to generate SecretClaim resources in order to automatically create and maintain Kubernetes Secrets from HashiCorp Vault via kube-vault-controller

These Secrets can then be referenced in the same manner as Additional Secrets above.

This can be achieved by defining required Secret Claims within controller.secretClaims, as follows:

controller:
  secretClaims:
    - name: jenkins-secret
      path: secret/path
    - name: jenkins-short-ttl
      path: secret/short-ttl-path
      renew: 60

RBAC

RBAC is enabled by default. If you want to disable it you will need to set rbac.create to false.

Adding Custom Pod Templates

It is possible to add custom pod templates for the default configured kubernetes cloud. Add a key under agent.podTemplates for each pod template. Each key (prior to | character) is just a label, and can be any value. Keys are only used to give the pod template a meaningful name. The only restriction is they may only contain RFC 1123 \ DNS label characters: lowercase letters, numbers, and hyphens. Each pod template can contain multiple containers. There's no need to add the jnlp container since the kubernetes plugin will automatically inject it into the pod. For this pod templates configuration to be loaded the following values must be set:

controller.JCasC.defaultConfig: true

The example below creates a python pod template in the kubernetes cloud:

agent:
  podTemplates:
    python: |
      - name: python
        label: jenkins-python
        serviceAccount: jenkins
        containers:
          - name: python
            image: python:3
            command: "/bin/sh -c"
            args: "cat"
            ttyEnabled: true
            privileged: true
            resourceRequestCpu: "400m"
            resourceRequestMemory: "512Mi"
            resourceLimitCpu: "1"
            resourceLimitMemory: "1024Mi"      

Best reference is https://<jenkins_url>/configuration-as-code/reference#Cloud-kubernetes.

Adding Pod Templates Using additionalAgents

additionalAgents may be used to configure additional kubernetes pod templates. Each additional agent corresponds to agent in terms of the configurable values and inherits all values from agent so you only need to specify values which differ. For example:

agent:
  podName: default
  customJenkinsLabels: default
  # set resources for additional agents to inherit
  resources:
    limits:
      cpu: "1"
      memory: "2048Mi"

additionalAgents:
  maven:
    podName: maven
    customJenkinsLabels: maven
    # An example of overriding the jnlp container
    # sideContainerName: jnlp
    image: jenkins/jnlp-agent-maven
    tag: latest
  python:
    podName: python
    customJenkinsLabels: python
    sideContainerName: python
    image: python
    tag: "3"
    command: "/bin/sh -c"
    args: "cat"
    TTYEnabled: true

Ingress Configuration

This chart provides ingress resources configurable via the controller.ingress block.

The simplest configuration looks like the following:

controller:
   ingress:
       enabled: true
       paths: []
       apiVersion: "extensions/v1beta1"
       hostName: jenkins.example.com

This snippet configures an ingress rule for exposing jenkins at jenkins.example.com

You can define labels and annotations via controller.ingress.labels and controller.ingress.annotations respectively. Additionally, you can configure the ingress tls via controller.ingress.tls. By default, this ingress rule exposes all paths. If needed this can be overwritten by specifying the wanted paths in controller.ingress.paths

If you want to configure a secondary ingress e.g. you don't want the jenkins instance exposed but still want to receive webhooks you can configure controller.secondaryingress. The secondaryingress doesn't expose anything by default and has to be configured via controller.secondaryingress.paths:

controller:
   ingress:
       enabled: true
       apiVersion: "extensions/v1beta1"
       hostName: "jenkins.internal.example.com"
       annotations:
           kubernetes.io/ingress.class: "internal"
   secondaryingress:
       enabled: true
       apiVersion: "extensions/v1beta1"
       hostName: "jenkins-scm.example.com"
       annotations:
           kubernetes.io/ingress.class: "public"
       paths:
        - /github-webhook

Prometheus Metrics

If you want to expose Prometheus metrics you need to install the Jenkins Prometheus Metrics Plugin. It will expose an endpoint (default /prometheus) with metrics where a Prometheus Server can scrape.

If you have implemented Prometheus Operator, you can set controller.prometheus.enabled to true to configure a ServiceMonitor and PrometheusRule. If you want to further adjust alerting rules you can do so by configuring controller.prometheus.alertingrules

If you have implemented Prometheus without using the operator, you can leave controller.prometheus.enabled set to false.

Running Behind a Forward Proxy

The controller pod uses an Init Container to install plugins etc. If you are behind a corporate proxy it may be useful to set controller.initContainerEnv to add environment variables such as http_proxy, so that these can be downloaded.

Additionally, you may want to add env vars for the init container, the Jenkins container, and the JVM (controller.javaOpts):

controller:
  initContainerEnv:
    - name: http_proxy
      value: "http://192.168.64.1:3128"
    - name: https_proxy
      value: "http://192.168.64.1:3128"
    - name: no_proxy
      value: ""
    - name: JAVA_OPTS
      value: "-Dhttps.proxyHost=proxy_host_name_without_protocol -Dhttps.proxyPort=3128"
  containerEnv:
    - name: http_proxy
      value: "http://192.168.64.1:3128"
    - name: https_proxy
      value: "http://192.168.64.1:3128"
  javaOpts: >-
    -Dhttp.proxyHost=192.168.64.1
    -Dhttp.proxyPort=3128
    -Dhttps.proxyHost=192.168.64.1
    -Dhttps.proxyPort=3128    

HTTPS Keystore Configuration

This configuration enables jenkins to use keystore in order to serve HTTPS. Here is the value file section related to keystore configuration. Keystore itself should be placed in front of jenkinsKeyStoreBase64Encoded key and in base64 encoded format. To achieve that after having keystore.jks file simply do this: cat keystore.jks | base64 and paste the output in front of jenkinsKeyStoreBase64Encoded. After enabling httpsKeyStore.enable make sure that httpPort and targetPort are not the same, as targetPort will serve HTTPS. Do not set controller.httpsKeyStore.httpPort to -1 because it will cause readiness and liveliness prob to fail. If you already have a kubernetes secret that has keystore and its password you can specify its' name in front of jenkinsHttpsJksSecretName, You need to remember that your secret should have proper data key names jenkins-jks-file (or override the key name using jenkinsHttpsJksSecretKey) and https-jks-password (or override the key name using jenkinsHttpsJksPasswordSecretKey; additionally you can make it get the password from a different secret using jenkinsHttpsJksPasswordSecretName). Example:

controller:
   httpsKeyStore:
       enable: true
       jenkinsHttpsJksSecretName: ''
       httpPort: 8081
       path: "/var/jenkins_keystore"
       fileName: "keystore.jks"
       password: "changeit"
       jenkinsKeyStoreBase64Encoded: ''

AWS Security Group Policies

To create SecurityGroupPolicies set awsSecurityGroupPolicies.enabled to true and add your policies. Each policy requires a name, array of securityGroupIds and a podSelector. Example:

awsSecurityGroupPolicies:
  enabled: true
  policies:
    - name: "jenkins-controller"
      securityGroupIds: 
        - sg-123456789
      podSelector:
        matchExpressions:
          - key: app.kubernetes.io/component
            operator: In
            values:
              - jenkins-controller

Agent Direct Connection

Set directConnection to true to allow agents to connect directly to a given TCP port without having to negotiate a HTTP(S) connection. This can allow you to have agent connections without an external HTTP(S) port. Example:

agent:
  jenkinsTunnel: "jenkinsci-agent:50000"
  directConnection: true

Migration Guide

From stable repository

Upgrade an existing release from stable/jenkins to jenkins/jenkins seamlessly by ensuring you have the latest repository info and running the upgrade commands specifying the jenkins/jenkins chart.

Major Version Upgrades

Chart release versions follow SemVer, where a MAJOR version change (example 1.0.0 -> 2.0.0) indicates an incompatible breaking change needing manual actions.

See UPGRADING.md for a list of breaking changes