236 lines
10 KiB
Markdown
236 lines
10 KiB
Markdown
# Alpine Linux Cloud Image Builder
|
|
|
|
This repository contains the code and and configs for the build system used to
|
|
create official Alpine Linux images for various cloud providers, in various
|
|
configurations. This build system is flexible, enabling others to build their
|
|
own customized images.
|
|
|
|
----
|
|
## Pre-Built Offical Cloud Images
|
|
|
|
To get started with offical pre-built Alpine Linux cloud images, visit
|
|
https://alpinelinux.org/cloud. Currently, we build official images for the
|
|
following cloud platforms...
|
|
* Amazon Web Services (AWS)
|
|
* Microsoft Azure
|
|
* GCP (Google Cloud Platform)
|
|
* OCI (Oracle Cloud Infrastructure)
|
|
* NoCloud
|
|
|
|
Each image's name contains the Alpine version release, architecture, firmware,
|
|
bootstrap, and image revision; a YAML metadata file containing these details
|
|
and more is downloadable.
|
|
|
|
| Key | Description / Values |
|
|
|-----|----------------------|
|
|
| name | `alpine-`_`release`_`-`_`arch`_`-`_`firmware`_`-`_`bootstrap`_`-r`_`revision`_ |
|
|
| project | `https://alpinelinux.org/cloud` |
|
|
| image_key | _`release`_`-`_`arch`_`-`_`firmware`_`-`_`bootstrap`_`-`_`cloud`_ |
|
|
| version | Alpine version (_`x.y`_ or `edge`) |
|
|
| release | Alpine release (_`x.y.z`_ or _`YYYYMMDD`_ for edge) |
|
|
| arch | architecture (`aarch64` or `x86_64`) |
|
|
| firmware | boot mode (`bios` or `uefi`) |
|
|
| bootstrap | initial bootstrap system (`tiny` = Tiny Cloud) |
|
|
| cloud | provider short name (`aws`) |
|
|
| revision | image revision number |
|
|
| built | image build timestamp |
|
|
| uploaded | image storage timestamp |
|
|
| imported | image import timestamp |
|
|
| import_id | imported image id |
|
|
| import_region | imported image region |
|
|
| signed | image signing timestamp |
|
|
| published | image publication timestamp |
|
|
| released | image release timestamp _(won't be set until second publish)_ |
|
|
| description | image description |
|
|
|
|
Published AWS images are also tagged with this data, but other AWS accounts
|
|
can't read these tags. However, the image name can still be used to filter
|
|
images to find what you're looking for. For example, to get a list of
|
|
available Alpine 3.x aarch64 images in AWS eu-west-2...
|
|
```
|
|
aws ec2 describe-images \
|
|
--region eu-west-2 \
|
|
--owners 538276064493 \
|
|
--filters \
|
|
Name=name,Values='alpine-3.*-aarch64-*' \
|
|
Name=state,Values=available \
|
|
--output text \
|
|
--query 'reverse(sort_by(Images, &CreationDate))[].[ImageId,Name,CreationDate]'
|
|
```
|
|
To get just the most recent matching image, use...
|
|
```
|
|
--query 'max_by(Image, &CreationDate).[ImageId,Name,CreationDate]'
|
|
```
|
|
|
|
----
|
|
## Build System
|
|
|
|
The build system consists of a number of components:
|
|
|
|
* the primary `build` script, and other related libararies...
|
|
* `clouds/` - specific cloud provider plugins
|
|
* `alpine.py` - for getting the latest Alpine information
|
|
* `image_config_manager.py` - manages collection of image configs
|
|
* `image_config.py` - individual image config functionality
|
|
* `image_storage.py` - persistent image/metadata storage
|
|
* `image_tags.py` - classes for working with image tags
|
|
|
|
* the `configs/` directory, defining the set of images to be built
|
|
|
|
* the `scripts/` directory, containing scripts and related data used to set up
|
|
image contents during provisioning
|
|
|
|
* the Packer `alpine.pkr.hcl`, which orchestrates the various build steps
|
|
from `local` and beyond.
|
|
|
|
* the `cloud_helper.py` script that Packer runs in order to do cloud-specific
|
|
per-image operations, such as image format conversion, upload, publishing,
|
|
etc.
|
|
|
|
### Build Requirements
|
|
|
|
* [Python](https://python.org) (3.9.9 is known to work)
|
|
* [Packer](https://packer.io) (1.9.4 is known to work)
|
|
* [QEMU](https://www.qemu.org) (8.1.2 is known to work)
|
|
* cloud provider account(s) _(for import/publish steps)_
|
|
|
|
### Cloud Credentials
|
|
|
|
Importing and publishing images relies on the cloud providers' Python API
|
|
libraries to find and use the necessary credentials, usually via configuration
|
|
under the user's home directory (i.e. `~/.aws/`, `~/.oci/`, etc.) or or via
|
|
environment variables (i.e. `AWS_...`, `OCI_...`, etc.)
|
|
|
|
_Note that presently, importing and publishing to cloud providers is only
|
|
supported for AWS images._
|
|
|
|
The credentials' user/role needs sufficient permission to query, import, and
|
|
publish images -- the exact details will vary from cloud to cloud. _It is
|
|
recommended that only the minimum required permissions are granted._
|
|
|
|
_We manage the credentials for publishing official Alpine images with an
|
|
"identity broker" service, and retrieve those credentials via the
|
|
`--use-broker` argument of the `build` script._
|
|
|
|
### The `build` Script
|
|
|
|
```
|
|
usage: build [-h] [--debug] [--clean] [--pad-uefi-bin-arch ARCH [ARCH ...]]
|
|
[--custom DIR [DIR ...]] [--skip KEY [KEY ...]] [--only KEY [KEY ...]]
|
|
[--revise] [--use-broker] [--no-color] [--parallel N]
|
|
[--vars FILE [FILE ...]]
|
|
{configs,state,rollback,local,upload,import,sign,publish,release}
|
|
|
|
positional arguments: (build up to and including this step)
|
|
configs resolve image build configuration
|
|
state report current build state of images
|
|
rollback remove local/uploaded/imported images if not published or released
|
|
local build images locally
|
|
upload upload images and metadata to storage
|
|
* import import local images to cloud provider default region (*)
|
|
sign cryptographically sign images
|
|
* publish set image permissions and publish to cloud regions (*)
|
|
release mark images as being officially relased
|
|
|
|
(*) may not apply to or be implemented for all cloud providers
|
|
|
|
optional arguments:
|
|
-h, --help show this help message and exit
|
|
--debug enable debug output
|
|
--clean start with a clean work environment
|
|
--pad-uefi-bin-arch ARCH [ARCH ...]
|
|
pad out UEFI firmware to 64 MiB ('aarch64')
|
|
--custom DIR [DIR ...] overlay custom directory in work environment
|
|
--skip KEY [KEY ...] skip variants with dimension key(s)
|
|
--only KEY [KEY ...] only variants with dimension key(s)
|
|
--revise bump revision and rebuild if published or released
|
|
--use-broker use the identity broker to get credentials
|
|
--no-color turn off Packer color output
|
|
--parallel N build N images in parallel
|
|
--vars FILE [FILE ...] supply Packer with -vars-file(s) (default: [])
|
|
--disable STEP [STEP ...] disable optional steps (default: [])
|
|
```
|
|
|
|
The `build` script will automatically create a `work/` directory containing a
|
|
Python virtual environment if one does not already exist. This directory also
|
|
hosts other data related to building images. The `--clean` argument will
|
|
remove everything in the `work/` directory except for things related to the
|
|
Python virtual environment.
|
|
|
|
If `work/configs/` or `work/scripts/` directories do not yet exist, they will
|
|
be populated with the base configuration and scripts from `configs/` and/or
|
|
`scripts/` directories. If any custom overlay directories are specified with
|
|
the `--custom` argument, their `configs/` and `scripts/` subdirectories are
|
|
also added to `work/configs/` and `work/scripts/`.
|
|
|
|
The "build step" positional argument deterimines the last step the `build`
|
|
script should execute -- all steps before this targeted step may also be
|
|
executed. That is, `build local` will first execute the `configs` step (if
|
|
necessary) and then the `state` step (always) before proceeding to the `local`
|
|
step.
|
|
|
|
The `configs` step resolves configuration for all buildable images, and writes
|
|
it to `work/images.yaml`, if it does not already exist.
|
|
|
|
The `state` step always checks the current state of the image builds,
|
|
determines what actions need to be taken, and updates `work/images.yaml`. A
|
|
subset of image builds can be targeted by using the `--skip` and `--only`
|
|
arguments.
|
|
|
|
The `rollback` step will remove any imported, uploaded, or local images, but
|
|
only if they are _unpublished_ and _unreleased_.
|
|
|
|
As _published_ and _released_ images can't be rolled back, `--revise` can be
|
|
used to increment the _`revision`_ value to rebuild newly revised images.
|
|
|
|
`local`, `upload`, `import`, `publish`, `sign` , and `release` steps are
|
|
orchestrated by Packer. By default, each image will be processed serially;
|
|
providing the `--parallel` argument with a value greater than 1 will
|
|
parallelize operations. The degree to which you can parallelize `local` image
|
|
builds will depend on the local build hardware -- as QEMU virtual machines are
|
|
launched for each image being built. Image `upload`, `import`, `publish`,
|
|
`sign`, and `release` steps are much more lightweight, and can support higher
|
|
parallelism.
|
|
|
|
The `local` step builds local images with QEMU, for those that are not already
|
|
built locally or have already been imported. Images are converted to formats
|
|
amenable for import into the cloud provider (if necessary) and checksums are
|
|
generated.
|
|
|
|
The `upload` step uploads the local image, checksum, and metadata to the
|
|
defined `storage_url`. The `import`, `publish`, and `release` steps will
|
|
also upload updated image metadata.
|
|
|
|
The `import` step imports the local images into the cloud providers' default
|
|
regions, unless they've already been imported. At this point the images are
|
|
not available publicly, allowing for additional testing prior to publishing.
|
|
|
|
The `sign` step will cryptographically sign the built image, using the command
|
|
specified by the `signing_cmd` config value.
|
|
|
|
The `publish` step copies the image from the default region to other regions,
|
|
if they haven't already been copied there. This step will always update
|
|
image permissions, descriptions, tags, and deprecation date (if applicable)
|
|
in all regions where the image has been published.
|
|
|
|
***NOTE:*** The `import` and `publish` steps are skipped for those cloud
|
|
providers where this does not make sense (i.e. NoCloud) or for those which
|
|
it has not yet been coded.
|
|
|
|
The `release` step simply marks the images as being fully released. If there
|
|
is a `release_cmd` specified, this is also executed, per image. _(For the
|
|
offical Alpine releases, we have a `gen_mksite_release.py` script to convert
|
|
the image data to a format that can be used by https://alpinelinux.org/cloud.)_
|
|
|
|
### The `cloud_helper.py` Script
|
|
|
|
This script is meant to be called only by Packer from its `post-processor`
|
|
block.
|
|
|
|
----
|
|
## Build Configuration
|
|
|
|
For more in-depth information about how the build system configuration works,
|
|
how to create custom config overlays, and details about individual config
|
|
settings, see [CONFIGURATION.md](CONFIGURATION.md).
|