NOTE: This is a Work-in-Progress

It is intended that this will eventually replace https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami as the offical multi-cloud image builder for Alpine Linux.

---

Alpine Linux Cloud Image Builder

This repository contains the code and and configs for the build system used to create official Alpine Linux images for various cloud providers, in various configurations. This build system is flexible, enabling others to build their own customized images.

---

Pre-Built Offical Cloud Images

To get started with offical pre-built Alpine Linux cloud images, visit https://alpinelinux.org/cloud. Currently, we build official images for the following providers:

• AWS

You should also be able to find the most recently published Alpine Linux images via your cloud provider's web console, or programatically query their API with a CLI tool or library.

(TODO: examples)

---

Build System

The build system consists of a number of components:

• the primary build script and related cloud-specific helpers
• a directory of configs/ defining the set of images to be built
• a Packer alpine.pkr.hcl orchestrating the images' local build, as well as importing them to cloud providers and publishing them to desitnation regions
• a directory of scripts/ which set up the images' contents during provisioning

Build Requirements

• Python (3.9.7 is known to work)
• Packer (1.7.6 is known to work)
• QEMU (6.1.0 is known to work)
• cloud provider account(s)

Cloud Credentials

This build system relies on the cloud providers' Python API libraries to find and use the necessary credentials -- via configuration in the user's home directory (i.e. ~/.aws/..., ~/.oci/..., etc.) or with special environment variables (i.e. AWS_..., OCI_..., etc.)

It is expected that each cloud provider's user/role will have been set up with sufficient permission in order to accomplish the operations necessary to query, import, and publish images; it is highly recommended that no permissions are granted beyond what is absolutely necessary.

The build Script

usage: build [-h] [--debug] [--clean] [--revise] {configs,local,import,publish}
         [--custom DIR [DIR ...]] [--skip KEY [KEY ...]] [--only KEY [KEY ...]]
         [--no-color] [--parallel N] [--vars FILE [FILE ...]]

build steps:
  configs                 resolve build configuration
  local                   build local images
  import                  import to cloud providers
  publish                 set permissions and publish to cloud regions

optional arguments:
  -h, --help              show this help message and exit
  --debug                 enable debug output (False)
  --clean                 start with a clean work environment (False)
  --revise                bump revisions if images already published (False)
  --custom DIR [DIR ...]  overlay custom directory in work environment
  --skip KEY [KEY ...]    skip variants with dimension key(s)
  --only KEY [KEY ...]    only variants with dimension key(s)
  --no-color              turn off Packer color output (False)
  --parallel N            build N images in parallel (1)
  --vars FILE [FILE ...]  supply Packer with additional -vars-file(s)

A work/ directory will be created for its Python virtual environment, any necessary Python libraries will be pip installed, and build will execute itself to ensure that it's running in the work environment.

This directory also contains configs/ and scripts/ subdirs (with custom overlays), UEFI firmware for QEMU, Packer cache, the generated configs.yaml and actions.yaml configs, and the images/ tree for local image builds.

Use --clean if you want to re-overlay, re-download, re-generate, or rebuild anything in the work/ directory. To redo the Python virtual environment, simply remove the work/ directory and its contents, and it will be recreated the next time build is run.

Build Steps

When executing build you also provide the target step you wish to reach. For example, if you only want to build local images, use build local. Any predecessor steps which haven't been done will also be executed -- that is, build local also implies build configs if that step hasn't completed yet.

The configs step determines the latest stable Alpine Linux release, and ensures that the configs/ and scripts/ overlays, UEFI firmware, and configs.yaml exist. This allows you to validate the generated build variant configuration before attempting to build any images locally.

If build is moving on past configs to other steps, it will determine which image variants to work on (based on --skip and --only values) and what actions will be taken, based on existence of local/imported/published images, and generate the actions.yaml file. Providing the --revise flag allows you to rebuild local images that were previously built, reimport unpublished images to cloud providers, and bump the "revision" value of previously published images -- this is useful if published images require fixes but the Alpine release itself isn't changing; published images are not removed (though they may be pruned once their "end-of-life" date has passed).

At this point, build executes Packer, which is responsible for the remaining local, import, and publish steps -- and also for parallelization, if the --parallel argument is given. Because build hardware varies, it is also possible to tune a number of QEMU timeouts and memory requirements by providing an HCL2 Packer Vars file and specifying --vars <filename> to override the defaults in alpine.pkr.hcl.

Packer and alpine.pkr.hcl

Packer loads and merges actions.yaml and configs.yaml, and iterates the resulting object in order to determine what it should do with each image variant configuration.

alpine.pkr.hcl defines two base source blocks -- null is used when an image variant is already built locally and/or already imported to the destination cloud provider; otherwise, the qemu source is used.

The qemu builder spins up a QEMU virtual machine with a blank virtual disk attached, using the latest stable Alpine Linux Virtual ISO, brings up the VM's network, enables the SSH daemon, and sets a random password for root.

If an image variant is to be built locally, the two dynamic provisioners copy the required data for the setup scripts to the VM's /tmp/ directory, and then run those setup scripts. It's these scripts that are ultimately responsible for installing and configuring the desired image on the attached virtual disk. When the setup scripts are complete, the virtual machine is shut down, and the resulting local disk image can be found at work/images/<cloud>/<build-name>/image.qcow2.

The dynamic post-processor uses the cloud_helper.py script to import a local image to the cloud provider, and/or publish an imported image to the cloud provider's destination regions, based on what actions are applicable for that image variant. When the publish step is reapplied to an already-published image, the script ensures that images have been copied to all destination regions (for example, if the cloud provider recently added a new region), and that all launch permissions are set as expected.

The cloud_helper.py Script

This script is only meant to be imported by build and called from Packer, and provides a normalized cloud-agnostic way of doing common cloud operations -- getting details about a variant's latest imported image, importing new local image to the cloud, removing a previouly imported (but unpublished) image so it can be replaced, or publishing an imported image to destination regions.

---

Build Configuration

The build script generates work/configs.yaml based on the contents of the top-level config file, work/configs/configs.conf; normally this is a symlink to alpine.conf, but can be overridden for custom builds. All configs are declared in HOCON format, which allows importing from other files, simple variable interpolation, and easy merging of objects. This flexibility helps keep configuration DRY.

The top-level build.conf has three main blocks, Default (default/starting values), Dimensions (with configs that apply in different circumstances), and Mandatory (required/final values). The configuration for these blocks are merged in this exact order.

Dimensions and Build Variants

Build variants (I was watching Loki™ at the time...) are the sets of dimensional "features" and related configuration details produced from a Cartesian product across each of the dimensional keys. Dimensional configs are merged together in the order they appear in build.conf.

If two dimensional keys are incompatible (for example, version/3.11 did not yet support arch/aarch64), an EXCLUDE directive indicates that such a variant is non-viable, and will be skipped.

Likewise, if one dimension's configuration depends on the value of a different dimensional key, the WHEN directive will supply the conditional config details when that other dimensional key is part of the variant.

Currently the base set of dimensions (and dimension keys) are...

version - current "release" value for each is autodetected, and always a component of an image's name

• edge ("release" value is the current UTC date)
• all non-EOL Alpine Linux versions

arch - machine architecture

• x86_64 (aka "amd64")
• aarch64 (aka "arm64")

firmware - machine boot firmware

• bios (legacy BIOS)
• uefi

bootstrap - image instantiation bootstrap is provided by...

• tiny (tiny-cloud-boostrap)
• cloudinit (cloud-init)

cloud - cloud provider or platform

• aws - Amazone Web Services / EC2
• oci - Oracle Cloud Infrastructure (WiP)
• gcp - Google Cloud Platform (WiP)
• azure - Microsoft Azure (WiP)

...each dimension may (or may not) contribute to the image name or description, if the dimensional key's config contributes to the name or description array values.

Customized Builds

(TODO)