Merge commit '5d0009b5f8878e33a8e9b0ab12f806179e2916d6' as 'alpine-cloud-images'

This commit is contained in:
Stefan Reimer 2023-04-28 10:12:12 +00:00
commit 3ba7b3dc1f
63 changed files with 4028 additions and 0 deletions

View File

@ -0,0 +1,2 @@
[flake8]
ignore = E265,E266,E402,E501

7
alpine-cloud-images/.gitignore vendored Normal file
View File

@ -0,0 +1,7 @@
*~
*.bak
*.swp
.DS_Store
.vscode/
/work/
releases*yaml

View File

@ -0,0 +1,314 @@
# Configuration
All the configuration for building image variants is defined by multiple
config files; the base configs for official Alpine Linux cloud images are in
the [`configs/`](configs/) directory.
We use [HOCON](https://github.com/lightbend/config/blob/main/HOCON.md) for
configuration -- this primarily facilitates importing deeper configs from
other files, but also allows the extension/concatenation of arrays and maps
(which can be a useful feature for customization), and inline comments.
----
## Resolving Work Environment Configs and Scripts
If `work/configs/` and `work/scripts/` don't exist, the `build` script will
install the contents of the base [`configs/`](configs/) and [`scripts/`](scripts/)
directories, and overlay additional `configs/` and `scripts/` subdirectories
from `--custom` directories (if any).
Files cannot be installed over existing files, with one exception -- the
[`configs/images.conf`](configs/images.conf) same-directory symlink. Because
the `build` script _always_ loads `work/configs/images.conf`, this is the hook
for "rolling your own" custom Alpine Linux cloud images.
The base [`configs/images.conf`](configs/images.conf) symlinks to
[`alpine.conf`](configs/images.conf), but this can be overridden using a
`--custom` directory containing a new `configs/images.conf` same-directory
symlink pointing to its custom top-level config.
For example, the configs and scripts in the [`overlays/testing/`](overlays/testing/)
directory can be resolved in a _clean_ work environment with...
```
./build configs --custom overlays/testing
```
This results in the `work/configs/images.conf` symlink to point to
`work/configs/alpine-testing.conf` instead of `work/configs/alpine.conf`.
If multiple directories are specified with `--custom`, they are applied in
the order given.
----
## Top-Level Config File
Examples of top-level config files are [`configs/alpine.conf`](configs/alpine.conf)
and [`overlays/testing/configs/alpine-testing.conf`](overlays/testing/configs/alpine-testing.conf).
There are three main blocks that need to exist (or be `import`ed into) the top
level HOCON configuration, and are merged in this exact order:
### `Default`
All image variant configs start with this block's contents as a starting point.
Arrays and maps can be appended by configs in `Dimensions` and `Mandatory`
blocks.
### `Dimensions`
The sub-blocks in `Dimensions` define the "dimensions" a variant config is
comprised of, and the different config values possible for that dimension.
The default [`alpine.conf`](configs/alpine.conf) defines the following
dimensional configs:
* `version` - Alpine Linux _x_._y_ (plus `edge`) versions
* `arch` - machine architectures, `x86_64` or `aarch64`
* `firmware` - supports launching via legacy BIOS or UEFI
* `bootstrap` - the system/scripts responsible for setting up an instance
during its initial launch
* `cloud` - for specific cloud platforms
The specific dimensional configs for an image variant are merged in the order
that the dimensions are listed.
### `Mandatory`
After a variant's dimensional configs have been applied, this is the last block
that's merged to the image variant configuration. This block is the ultimate
enforcer of any non-overrideable configuration across all variants, and can
also provide the last element to array config items.
----
## Dimensional Config Directives
Because a full cross-product across all dimensional configs may produce images
variants that are not viable (i.e. `aarch64` simply does not support legacy
`bios`), or may require further adjustments (i.e. the `aws` `aarch64` images
require an additional kernel module from `3.15` forward, which aren't available
in previous versions), we have two special directives which may appear in
dimensional configs.
### `EXCLUDE` array
This directive provides an array of dimensional config keys which are
incompatible with the current dimensional config. For example,
[`configs/arch/aarch64.conf`](configs/arch/aarch64.conf) specifies...
```
# aarch64 is UEFI only
EXCLUDE = [bios]
```
...which indicates that any image variant that includes both `aarch64` (the
current dimensional config) and `bios` configuration should be skipped.
### `WHEN` block
This directive conditionally merges additional configuration ***IF*** the
image variant also includes a specific dimensional config key (or keys). In
order to handle more complex situations, `WHEN` blocks may be nested. For
example, [`configs/cloud/aws.conf`](configs/cloud/aws.conf) has...
```
WHEN {
aarch64 {
# new AWS aarch64 default...
kernel_modules.gpio_pl061 = true
initfs_features.gpio_pl061 = true
WHEN {
"3.14 3.13 3.12" {
# ...but not supported for older versions
kernel_modules.gpio_pl061 = false
initfs_features.gpio_pl061 = false
}
}
}
```
This configures AWS `aarch64` images to use the `gpio_pl061` kernel module in
order to cleanly shutdown/reboot instances from the web console, CLI, or SDK.
However, this module is unavailable on older Alpine versions.
Spaces in `WHEN` block keys serve as an "OR" operator; nested `WHEN` blocks
function as "AND" operators.
----
## Config Settings
**Scalar** values can be simply overridden in later configs.
**Array** and **map** settings in later configs are merged with the previous
values, _or entirely reset if it's first set to `null`_, for example...
```
some_array = [ thing ]
# [...]
some_array = null
some_array = [ other_thing ]
```
Mostly in order of appearance, as we walk through
[`configs/alpine.conf`](configs/alpine.conf) and the deeper configs it
imports...
### `project` string
This is a unique identifier for the whole collection of images being built.
For the official Alpine Linux cloud images, this is set to
`https://alpinelinux.org/cloud`.
When building custom images, you **MUST** override **AT LEAST** this setting to
avoid image import and publishing collisions.
### `name` array
The ultimate contents of this array contribute to the overall naming of the
resultant image. Almost all dimensional configs will add to the `name` array,
with two notable exceptions: **version** configs' contribution to this array is
determined when `work/images.yaml` is resolved, and is set to the current
Alpine Linux release (_x.y.z_ or _YYYYMMDD_ for edge); also because
**cloud** images are isolated from each other, it's redundant to include that
in the image name.
### `description` array
Similar to the `name` array, the elements of this array contribute to the final
image description. However, for the official Alpine configs, only the
**version** dimension adds to this array, via the same mechanism that sets the
revision for the `name` array.
### `motd` map
This setting controls the contents of what ultimately gets written into the
variant image's `/etc/motd` file. Later configs can add additional messages,
replace existing contents, or remove them entirely (by setting the value to
`null`).
The `motd.release_notes` setting will be ignored if the Alpine release does
not have a release notes web page associated with it.
### `scripts` array
These are the scripts that will be executed by Packer, in order, to do various
setup tasks inside a variant's image. The `work/scripts/` directory contains
all scripts, including those that may have been added via `build --custom`.
### `script_dirs` array
Directories (under `work/scripts/`) that contain additional data that the
`scripts` will need. Packer will copy these to the VM responsible for setting
up the variant image.
### `size` string
The size of the image disk, by default we use `1G` (1 GiB). This disk may (or
may not) be further partitioned, based on other factors.
### `login` string
The image's primary login user, set to `alpine`.
### `repos` map
Defines the contents of the image's `/etc/apk/repositories` file. The map's
key is the URL of the repo, and the value determines how that URL will be
represented in the `repositories` file...
| value | result |
|-|-|
| `null` | make no reference to this repo |
| `false` | this repo is commented out (disabled) |
| `true` | this repo is enabled for use |
| _tag_ | enable this repo with `@`_`tag`_ |
### `packages` map
Defines what APK packages to add/delete. The map's key is the package
name, and the value determines whether (or not) to install/uninstall the
package...
| value | result |
|-|-|
| `null` | don't add or delete |
| `false` | explicitly delete |
| `true` | add from default repos |
| _tag_ | add from `@`_`tag`_ repo |
| `--no-scripts` | add with `--no-scripts` option |
| `--no-scripts` _tag_ | add from `@`_`tag`_ repo, with `--no-scripts` option |
### `services` map of maps
Defines what services are enabled/disabled at various runlevels. The first
map's key is the runlevel, the second key is the service. The service value
determines whether (or not) to enable/disable the service at that runlevel...
| value | result |
|-|-|
| `null` | don't enable or disable |
| `false` | explicitly disable |
| `true` | explicitly enable |
### `kernel_modules` map
Defines what kernel modules are specified in the boot loader. The key is the
kernel module, and the value determines whether or not it's in the final
list...
| value | result |
|-|-|
| `null` | skip |
| `false` | skip |
| `true` | include |
### `kernel_options` map
Defines what kernel options are specified on the kernel command line. The keys
are the kernel options, the value determines whether or not it's in the final
list...
| value | result |
|-|-|
| `null` | skip |
| `false` | skip |
| `true` | include |
### `initfs_features` map
Defines what initfs features are included when making the image's initramfs
file. The keys are the initfs features, and the values determine whether or
not they're included in the final list...
| value | result |
|-|-|
| `null` | skip |
| `false` | skip |
| `true` | include |
### `qemu.machine_type` string
The QEMU machine type to use when building local images. For x86_64, this is
set to `null`, for aarch64, we use `virt`.
### `qemu.args` list of lists
Additional QEMU arguments. For x86_64, this is set to `null`; but aarch64
requires several additional arguments to start an operational VM.
### `qemu.firmware` string
The path to the QEMU firmware (installed in `work/firmware/`). This is only
used when creating UEFI images.
### `bootloader` string
The bootloader to use, currently `extlinux` or `grub-efi`.
### `access` map
When images are published, this determines who has access to those images.
The key is the cloud account (or `PUBLIC`), and the value is whether or not
access is granted, `true` or `false`/`null`.
### `regions` map
Determines where images should be published. The key is the region
identifier (or `ALL`), and the value is whether or not to publish to that
region, `true` or `false`/`null`.
### `encrypted` string
Determines whether the image will be encrypted when imported and published.
Currently, only the **aws** cloud module supports this.
### `repo_keys` array
List of addtional repository keys to trust during the package installation phase.
This allows pulling in custom apk packages by simple specifying the repository name in packages block.

View File

@ -0,0 +1,19 @@
Copyright (c) 2017-2022 Jake Buchholz Göktürk, Michael Crute
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do
so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

View File

@ -0,0 +1,208 @@
# Alpine Linux Cloud Image Builder
This repository contains the code and and configs for the build system used to
create official Alpine Linux images for various cloud providers, in various
configurations. This build system is flexible, enabling others to build their
own customized images.
----
## Pre-Built Offical Cloud Images
To get started with offical pre-built Alpine Linux cloud images, visit
https://alpinelinux.org/cloud. Currently, we build official images for the
following cloud platforms...
* AWS
...we are working on also publishing offical images to other major cloud
providers.
Each published image's name contains the Alpine version release, architecture,
firmware, bootstrap, and image revision. These details (and more) are also
tagged on the images...
| Tag | Description / Values |
|-----|----------------------|
| name | `alpine-`_`release`_`-`_`arch`_`-`_`firmware`_`-`_`bootstrap`_`-r`_`revision`_ |
| project | `https://alpinelinux.org/cloud` |
| image_key | _`release`_`-`_`arch`_`-`_`firmware`_`-`_`bootstrap`_`-`_`cloud`_ |
| version | Alpine version (_`x.y`_ or `edge`) |
| release | Alpine release (_`x.y.z`_ or _`YYYYMMDD`_ for edge) |
| arch | architecture (`aarch64` or `x86_64`) |
| firmware | boot mode (`bios` or `uefi`) |
| bootstrap | initial bootstrap system (`tiny` = Tiny Cloud) |
| cloud | provider short name (`aws`) |
| revision | image revision number |
| imported | image import timestamp |
| import_id | imported image id |
| import_region | imported image region |
| published | image publication timestamp |
| description | image description |
Although AWS does not allow cross-account filtering by tags, the image name can
still be used to filter images. For example, to get a list of available Alpine
3.x aarch64 images in AWS eu-west-2...
```
aws ec2 describe-images \
--region eu-west-2 \
--owners 538276064493 \
--filters \
Name=name,Values='alpine-3.*-aarch64-*' \
Name=state,Values=available \
--output text \
--query 'reverse(sort_by(Images, &CreationDate))[].[ImageId,Name,CreationDate]'
```
To get just the most recent matching image, use...
```
--query 'max_by(Image, &CreationDate).[ImageId,Name,CreationDate]'
```
----
## Build System
The build system consists of a number of components:
* the primary `build` script
* the `configs/` directory, defining the set of images to be built
* the `scripts/` directory, containing scripts and related data used to set up
image contents during provisioning
* the Packer `alpine.pkr.hcl`, which orchestrates build, import, and publishing
of images
* the `cloud_helper.py` script that Packer runs in order to do cloud-specific
import and publish operations
### Build Requirements
* [Python](https://python.org) (3.9.7 is known to work)
* [Packer](https://packer.io) (1.7.6 is known to work)
* [QEMU](https://www.qemu.org) (6.1.0 is known to work)
* cloud provider account(s)
### Cloud Credentials
By default, the build system relies on the cloud providers' Python API
libraries to find and use the necessary credentials, usually via configuration
under the user's home directory (i.e. `~/.aws/`, `~/.oci/`, etc.) or or via
environment variables (i.e. `AWS_...`, `OCI_...`, etc.)
The credentials' user/role needs sufficient permission to query, import, and
publish images -- the exact details will vary from cloud to cloud. _It is
recommended that only the minimum required permissions are granted._
_We manage the credentials for publishing official Alpine images with an
"identity broker" service, and retrieve those credentials via the
`--use-broker` argument of the `build` script._
### The `build` Script
```
usage: build [-h] [--debug] [--clean] [--pad-uefi-bin-arch ARCH [ARCH ...]]
[--custom DIR [DIR ...]] [--skip KEY [KEY ...]] [--only KEY [KEY ...]]
[--revise] [--use-broker] [--no-color] [--parallel N]
[--vars FILE [FILE ...]]
{configs,state,rollback,local,upload,import,publish,release}
positional arguments: (build up to and including this step)
configs resolve image build configuration
state refresh current image build state
rollback remove existing local/uploaded/imported images if un-published/released
local build images locally
upload upload images and metadata to storage
* import import local images to cloud provider default region (*)
* publish set image permissions and publish to cloud regions (*)
release mark images as being officially relased
(*) may not apply to or be implemented for all cloud providers
optional arguments:
-h, --help show this help message and exit
--debug enable debug output
--clean start with a clean work environment
--pad-uefi-bin-arch ARCH [ARCH ...]
pad out UEFI firmware to 64 MiB ('aarch64')
--custom DIR [DIR ...] overlay custom directory in work environment
--skip KEY [KEY ...] skip variants with dimension key(s)
--only KEY [KEY ...] only variants with dimension key(s)
--revise remove existing local/uploaded/imported images if
un-published/released, or bump revision and rebuild
--use-broker use the identity broker to get credentials
--no-color turn off Packer color output
--parallel N build N images in parallel
--vars FILE [FILE ...] supply Packer with -vars-file(s) (default: [])
```
The `build` script will automatically create a `work/` directory containing a
Python virtual environment if one does not already exist. This directory also
hosts other data related to building images. The `--clean` argument will
remove everything in the `work/` directory except for things related to the
Python virtual environment.
If `work/configs/` or `work/scripts/` directories do not yet exist, they will
be populated with the base configuration and scripts from `configs/` and/or
`scripts/` directories. If any custom overlay directories are specified with
the `--custom` argument, their `configs/` and `scripts/` subdirectories are
also added to `work/configs/` and `work/scripts/`.
The "build step" positional argument deterimines the last step the `build`
script should execute -- all steps before this targeted step may also be
executed. That is, `build local` will first execute the `configs` step (if
necessary) and then the `state` step (always) before proceeding to the `local`
step.
The `configs` step resolves configuration for all buildable images, and writes
it to `work/images.yaml`, if it does not already exist.
The `state` step always checks the current state of the image builds,
determines what actions need to be taken, and updates `work/images.yaml`. A
subset of image builds can be targeted by using the `--skip` and `--only`
arguments.
The `rollback` step, when used with `--revise` argument indicates that any
_unpublished_ and _unreleased_ local, imported, or uploaded images should be
removed and rebuilt.
As _published_ and _released_ images can't be removed, `--revise` can be used
with `configs` or `state` to increment the _`revision`_ value to rebuild newly
revised images.
`local`, `upload`, `import`, `publish`, and `release` steps are orchestrated by
Packer. By default, each image will be processed serially; providing the
`--parallel` argument with a value greater than 1 will parallelize operations.
The degree to which you can parallelze `local` image builds will depend on the
local build hardware -- as QEMU virtual machines are launched for each image
being built. Image `upload`, `import`, `publish`, and `release` steps are much
more lightweight, and can support higher parallelism.
The `local` step builds local images with QEMU, for those that are not already
built locally or have already been imported. Images are converted to formats
amenable for import into the cloud provider (if necessary) and checksums are
generated.
The `upload` step uploads the local image, checksum, and metadata to the
defined `storage_url`. The `import`, `publish`, and `release` steps will
also upload updated image metadata.
The `import` step imports the local images into the cloud providers' default
regions, unless they've already been imported. At this point the images are
not available publicly, allowing for additional testing prior to publishing.
The `publish` step copies the image from the default region to other regions,
if they haven't already been copied there. This step will always update
image permissions, descriptions, tags, and deprecation date (if applicable)
in all regions where the image has been published.
***NOTE:*** The `import` and `publish` steps are skipped for those cloud
providers where this does not make sense (i.e. NoCloud) or for those which
it has not yet been coded.
The `release` step marks the images as being fully released.
### The `cloud_helper.py` Script
This script is meant to be called only by Packer from its `post-processor`
block.
----
## Build Configuration
For more in-depth information about how the build system configuration works,
how to create custom config overlays, and details about individual config
settings, see [CONFIGURATION.md](CONFIGURATION.md).

View File

@ -0,0 +1,201 @@
# Alpine Cloud Images Packer Configuration
### Variables
# include debug output from provisioning/post-processing scripts
variable "DEBUG" {
default = 0
}
# indicates cloud_helper.py should be run with --use-broker
variable "USE_BROKER" {
default = 0
}
# tuneable QEMU VM parameters, based on perfomance of the local machine;
# overrideable via build script --vars parameter referencing a Packer
# ".vars.hcl" file containing alternate settings
variable "qemu" {
default = {
boot_wait = {
aarch64 = "1m"
x86_64 = "1m"
}
cmd_wait = "5s"
ssh_timeout = "1m"
memory = 1024 # MiB
}
}
### Local Data
locals {
# possible actions for the post-processor
actions = [
"local", "upload", "import", "publish", "release"
]
debug_arg = var.DEBUG == 0 ? "" : "--debug"
broker_arg = var.USE_BROKER == 0 ? "" : "--use-broker"
# randomly generated password
password = uuidv4()
# resolve actionable build configs
configs = { for b, cfg in yamldecode(file("work/images.yaml")):
b => cfg if contains(keys(cfg), "actions")
}
}
### Build Sources
# Don't build
source null alpine {
communicator = "none"
}
# Common to all QEMU builds
source qemu alpine {
# qemu machine
headless = true
memory = var.qemu.memory
net_device = "virtio-net"
disk_interface = "virtio"
# build environment
boot_command = [
"root<enter>",
"setup-interfaces<enter><enter><enter><enter>",
"ifup eth0<enter><wait${var.qemu.cmd_wait}>",
"setup-sshd openssh<enter><wait${var.qemu.cmd_wait}>",
"echo PermitRootLogin yes >> /etc/ssh/sshd_config<enter>",
"service sshd restart<enter>",
"echo 'root:${local.password}' | chpasswd<enter>",
]
ssh_username = "root"
ssh_password = local.password
ssh_timeout = var.qemu.ssh_timeout
shutdown_command = "poweroff"
}
build {
name = "alpine"
## Builders
# QEMU builder
dynamic "source" {
for_each = { for b, c in local.configs:
b => c if contains(c.actions, "local")
}
iterator = B
labels = ["qemu.alpine"] # links us to the base source
content {
name = B.key
# qemu machine
qemu_binary = "qemu-system-${B.value.arch}"
qemuargs = B.value.qemu.args
machine_type = B.value.qemu.machine_type
firmware = B.value.qemu.firmware
# build environment
iso_url = B.value.qemu.iso_url
iso_checksum = "file:${B.value.qemu.iso_url}.sha512"
boot_wait = var.qemu.boot_wait[B.value.arch]
# results
output_directory = "work/images/${B.value.cloud}/${B.value.image_key}"
disk_size = B.value.size
format = "qcow2"
vm_name = "image.qcow2"
}
}
# Null builder (don't build, but we might do other actions)
dynamic "source" {
for_each = { for b, c in local.configs:
b => c if !contains(c.actions, "local")
}
iterator = B
labels = ["null.alpine"]
content {
name = B.key
}
}
## build provisioners
# install setup files
dynamic "provisioner" {
for_each = { for b, c in local.configs:
b => c if contains(c.actions, "local")
}
iterator = B
labels = ["file"]
content {
only = [ "qemu.${B.key}" ] # configs specific to one build
sources = [ for d in B.value.script_dirs: "work/scripts/${d}" ]
destination = "/tmp/"
}
}
# run setup scripts
dynamic "provisioner" {
for_each = { for b, c in local.configs:
b => c if contains(c.actions, "local")
}
iterator = B
labels = ["shell"]
content {
only = [ "qemu.${B.key}" ] # configs specific to one build
scripts = [ for s in B.value.scripts: "work/scripts/${s}" ]
use_env_var_file = true
environment_vars = [
"DEBUG=${var.DEBUG}",
"ARCH=${B.value.arch}",
"BOOTLOADER=${B.value.bootloader}",
"BOOTSTRAP=${B.value.bootstrap}",
"BUILD_NAME=${B.value.name}",
"BUILD_REVISION=${B.value.revision}",
"CLOUD=${B.value.cloud}",
"END_OF_LIFE=${B.value.end_of_life}",
"FIRMWARE=${B.value.firmware}",
"IMAGE_LOGIN=${B.value.login}",
"INITFS_FEATURES=${B.value.initfs_features}",
"KERNEL_MODULES=${B.value.kernel_modules}",
"KERNEL_OPTIONS=${B.value.kernel_options}",
"MOTD=${B.value.motd}",
"NTP_SERVER=${B.value.ntp_server}",
"PACKAGES_ADD=${B.value.packages.add}",
"PACKAGES_DEL=${B.value.packages.del}",
"PACKAGES_NOSCRIPTS=${B.value.packages.noscripts}",
"RELEASE=${B.value.release}",
"REPOS=${B.value.repos}",
"REPO_KEYS=${B.value.repo_keys}",
"SERVICES_ENABLE=${B.value.services.enable}",
"SERVICES_DISABLE=${B.value.services.disable}",
"VERSION=${B.value.version}",
]
}
}
## build post-processor
# import and/or publish cloud images
dynamic "post-processor" {
for_each = { for b, c in local.configs:
b => c if length(setintersection(c.actions, local.actions)) > 0
}
iterator = B
labels = ["shell-local"]
content {
only = [ "qemu.${B.key}", "null.${B.key}" ]
inline = [ for action in local.actions:
"./cloud_helper.py ${action} ${local.debug_arg} ${local.broker_arg} ${B.key}" if contains(B.value.actions, action)
]
}
}
}

View File

@ -0,0 +1,114 @@
# vim: ts=4 et:
import json
import re
from datetime import datetime, timedelta
from urllib.request import urlopen
class Alpine():
DEFAULT_RELEASES_URL = 'https://alpinelinux.org/releases.json'
DEFAULT_POSTS_URL = 'https://alpinelinux.org/posts/'
DEFAULT_CDN_URL = 'https://dl-cdn.alpinelinux.org/alpine'
DEFAULT_WEB_TIMEOUT = 5
def __init__(self, releases_url=None, posts_url=None, cdn_url=None, web_timeout=None):
self.now = datetime.utcnow()
self.release_today = self.now.strftime('%Y%m%d')
self.eol_tomorrow = (self.now + timedelta(days=1)).strftime('%F')
self.latest = None
self.versions = {}
self.releases_url = releases_url or self.DEFAULT_RELEASES_URL
self.posts_url = posts_url or self.DEFAULT_POSTS_URL
self.web_timeout = web_timeout or self.DEFAULT_WEB_TIMEOUT
self.cdn_url = cdn_url or self.DEFAULT_CDN_URL
# get all Alpine versions, and their EOL and latest release
res = urlopen(self.releases_url, timeout=self.web_timeout)
r = json.load(res)
branches = sorted(
r['release_branches'], reverse=True,
key=lambda x: x.get('branch_date', '0000-00-00')
)
for b in branches:
ver = b['rel_branch'].lstrip('v')
if not self.latest:
self.latest = ver
rel = None
notes = None
if releases := b.get('releases', None):
r = sorted(
releases, reverse=True, key=lambda x: x['date']
)[0]
rel = r['version']
notes = r.get('notes', None)
if notes:
notes = self.posts_url + notes.removeprefix('posts/').replace('.md', '.html')
elif ver == 'edge':
# edge "releases" is today's YYYYMMDD
rel = self.release_today
self.versions[ver] = {
'version': ver,
'release': rel,
'end_of_life': b.get('eol_date', self.eol_tomorrow),
'arches': b.get('arches'),
'notes': notes,
}
def _ver(self, ver=None):
if not ver or ver == 'latest' or ver == 'latest-stable':
ver = self.latest
return ver
def repo_url(self, repo, arch, ver=None):
ver = self._ver(ver)
if ver != 'edge':
ver = 'v' + ver
return f"{self.cdn_url}/{ver}/{repo}/{arch}"
def virt_iso_url(self, arch, ver=None):
ver = self._ver(ver)
rel = self.versions[ver]['release']
return f"{self.cdn_url}/v{ver}/releases/{arch}/alpine-virt-{rel}-{arch}.iso"
def version_info(self, ver=None):
ver = self._ver(ver)
if ver not in self.versions:
# perhaps a release candidate?
apk_ver = self.apk_version('main', 'x86_64', 'alpine-base', ver=ver)
rel = apk_ver.split('-')[0]
ver = '.'.join(rel.split('.')[:2])
self.versions[ver] = {
'version': ver,
'release': rel,
'end_of_life': self.eol_tomorrow,
'arches': self.versions['edge']['arches'], # reasonable assumption
'notes': None,
}
return self.versions[ver]
# TODO? maybe implement apk_info() to read from APKINDEX, but for now
# this apk_version() seems faster and gets what we need
def apk_version(self, repo, arch, apk, ver=None):
ver = self._ver(ver)
repo_url = self.repo_url(repo, arch, ver=ver)
apks_re = re.compile(f'"{apk}-(\\d.*)\\.apk"')
res = urlopen(repo_url, timeout=self.web_timeout)
for line in map(lambda x: x.decode('utf8'), res):
if not line.startswith('<a href="'):
continue
match = apks_re.search(line)
if match:
return match.group(1)
# didn't find it?
raise RuntimeError(f"Unable to find {apk} APK via {repo_url}")

349
alpine-cloud-images/build Executable file
View File

@ -0,0 +1,349 @@
#!/usr/bin/env python3
# vim: ts=4 et:
# Ensure we're using the Python virtual env with our installed dependencies
import os
import sys
import subprocess
sys.pycache_prefix = 'work/__pycache__'
# Create the work environment if it doesn't exist.
if not os.path.exists('work'):
import venv
PIP_LIBS = [
'mergedeep',
'pyhocon',
'python-dateutil',
'ruamel.yaml',
]
print('Work environment does not exist, creating...', file=sys.stderr)
venv.create('work', with_pip=True)
subprocess.run(['work/bin/pip', 'install', '-U', 'pip', 'wheel'])
subprocess.run(['work/bin/pip', 'install', '-U', *PIP_LIBS])
# Re-execute using the right virtual environment, if necessary.
venv_args = [os.path.join('work', 'bin', 'python3')] + sys.argv
if os.path.join(os.getcwd(), venv_args[0]) != sys.executable:
print("Re-executing with work environment's Python...\n", file=sys.stderr)
os.execv(venv_args[0], venv_args)
# We're now in the right Python environment...
import argparse
import io
import logging
import shutil
import time
from glob import glob
from subprocess import Popen, PIPE
from urllib.request import urlopen
import clouds
from alpine import Alpine
from image_config_manager import ImageConfigManager
### Constants & Variables
STEPS = ['configs', 'state', 'rollback', 'local', 'upload', 'import', 'publish', 'release']
LOGFORMAT = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
WORK_CLEAN = {'bin', 'include', 'lib', 'pyvenv.cfg', '__pycache__'}
WORK_OVERLAYS = ['configs', 'scripts']
UEFI_FIRMWARE = {
'aarch64': {
'apk': 'aavmf',
'bin': 'usr/share/AAVMF/QEMU_EFI.fd',
},
'x86_64': {
'apk': 'ovmf',
'bin': 'usr/share/OVMF/OVMF.fd',
}
}
alpine = Alpine()
### Functions
# ensure list has unique values, preserving order
def unique_list(x):
d = {e: 1 for e in x}
return list(d.keys())
def remove_dupe_args():
class RemoveDupeArgs(argparse.Action):
def __call__(self, parser, args, values, option_string=None):
setattr(args, self.dest, unique_list(values))
return RemoveDupeArgs
def are_args_valid(checker):
class AreArgsValid(argparse.Action):
def __call__(self, parser, args, values, option_string=None):
# remove duplicates
values = unique_list(values)
for x in values:
if not checker(x):
parser.error(f"{option_string} value is not a {self.metavar}: {x}")
setattr(args, self.dest, values)
return AreArgsValid
def clean_work():
log.info('Cleaning work environment')
for x in (set(os.listdir('work')) - WORK_CLEAN):
x = os.path.join('work', x)
log.debug('removing %s', x)
if os.path.isdir(x) and not os.path.islink(x):
shutil.rmtree(x)
else:
os.unlink(x)
def is_images_conf(o, x):
if not all([
o == 'configs',
x.endswith('/images.conf'),
os.path.islink(x),
]):
return False
# must also link to file in the same directory
x_link = os.path.normpath(os.readlink(x))
return x_link == os.path.basename(x_link)
def install_overlay(overlay):
log.info("Installing '%s' overlay in work environment", overlay)
dest_dir = os.path.join('work', overlay)
os.makedirs(dest_dir, exist_ok=True)
for src in unique_list(['.'] + args.custom):
src_dir = os.path.join(src, overlay)
if not os.path.exists(src_dir):
log.debug('%s does not exist, skipping', src_dir)
continue
for x in glob(os.path.join(src_dir, '**'), recursive=True):
x = x.removeprefix(src_dir + '/')
src_x = os.path.join(src_dir, x)
dest_x = os.path.join(dest_dir, x)
if is_images_conf(overlay, src_x):
rel_x = os.readlink(src_x)
if os.path.islink(dest_x):
log.debug('overriding %s', dest_x)
os.unlink(dest_x)
log.debug('ln -s %s %s', rel_x, dest_x)
os.symlink(rel_x, dest_x)
continue
if os.path.isdir(src_x):
if not os.path.exists(dest_x):
log.debug('makedirs %s', dest_x)
os.makedirs(dest_x)
if os.path.isdir(dest_x):
continue
if os.path.exists(dest_x):
log.critical('Unallowable destination overwirte detected: %s', dest_x)
sys.exit(1)
log.debug('cp -p %s %s', src_x, dest_x)
shutil.copy(src_x, dest_x)
def install_overlays():
for overlay in WORK_OVERLAYS:
if not os.path.isdir(os.path.join('work', overlay)):
install_overlay(overlay)
else:
log.info("Using existing '%s' in work environment", overlay)
def install_qemu_firmware():
firm_dir = 'work/firmware'
if os.path.isdir(firm_dir):
log.info('Using existing UEFI firmware in work environment')
return
log.info('Installing UEFI firmware in work environment')
os.makedirs(firm_dir)
for arch, a_cfg in UEFI_FIRMWARE.items():
apk = a_cfg['apk']
bin = a_cfg['bin']
v = alpine.apk_version('community', arch, apk)
apk_url = f"{alpine.repo_url('community', arch)}/{apk}-{v}.apk"
data = urlopen(apk_url).read()
# Python tarfile library can't extract from APKs
tar_cmd = ['tar', '-zxf', '-', '-C', firm_dir, bin]
p = Popen(tar_cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE)
out, err = p.communicate(input=data)
if p.returncode:
log.critical('Unable to untar %s to get %s', apk_url, bin)
log.error('%s = %s', p.returncode, ' '.join(tar_cmd))
log.error('STDOUT:\n%s', out.decode('utf8'))
log.error('STDERR:\n%s', err.decode('utf8'))
sys.exit(1)
firm_bin = os.path.join(firm_dir, f"uefi-{arch}.bin")
os.symlink(bin, firm_bin)
if arch in args.pad_uefi_bin_arch:
log.info('Padding "%s" to 67108864 bytes', firm_bin)
subprocess.run(['truncate', '-s', '67108864', firm_bin])
### Command Line & Logging
parser = argparse.ArgumentParser(
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
# general options
parser.add_argument(
'--debug', action='store_true', help='enable debug output')
parser.add_argument(
'--clean', action='store_true', help='start with a clean work environment')
parser.add_argument(
'--pad-uefi-bin-arch', metavar='ARCH', nargs='+', action=remove_dupe_args(),
default=['aarch64'], help='pad out UEFI firmware binaries to 64 MiB')
# config options
parser.add_argument(
'--custom', metavar='DIR', nargs='+', action=are_args_valid(os.path.isdir),
default=[], help='overlay custom directory in work environment')
# state options
parser.add_argument(
'--skip', metavar='KEY', nargs='+', action=remove_dupe_args(),
default=[], help='skip variants with dimension key(s)')
parser.add_argument(
'--only', metavar='KEY', nargs='+', action=remove_dupe_args(),
default=[], help='only variants with dimension key(s)')
parser.add_argument(
'--revise', action='store_true',
help='remove existing local/uploaded/imported image, or bump revision and '
' rebuild if published or released')
parser.add_argument(
'--use-broker', action='store_true',
help='use the identity broker to get credentials')
# packer options
parser.add_argument(
'--no-color', action='store_true', help='turn off Packer color output')
parser.add_argument(
'--parallel', metavar='N', type=int, default=1,
help='build N images in parallel')
parser.add_argument(
'--vars', metavar='FILE', nargs='+', action=are_args_valid(os.path.isfile),
default=[], help='supply Packer with -vars-file(s)')
# positional argument
parser.add_argument(
'step', choices=STEPS, help='build up to and including this step')
args = parser.parse_args()
log = logging.getLogger('build')
log.setLevel(logging.DEBUG if args.debug else logging.INFO)
console = logging.StreamHandler()
logfmt = logging.Formatter(LOGFORMAT, datefmt='%FT%TZ')
logfmt.converter = time.gmtime
console.setFormatter(logfmt)
log.addHandler(console)
log.debug(args)
if args.step == 'rollback':
log.warning('"rollback" step enables --revise option')
args.revise = True
# set up credential provider, if we're going to use it
if args.use_broker:
clouds.set_credential_provider(debug=args.debug)
### Setup Configs
latest = alpine.version_info()
log.info('Latest Alpine version %s, release %s, and notes: %s', latest['version'], latest['release'], latest['notes'])
if args.clean:
clean_work()
# install overlay(s) if missing
install_overlays()
image_configs = ImageConfigManager(
conf_path='work/configs/images.conf',
yaml_path='work/images.yaml',
log='build',
alpine=alpine,
)
log.info('Configuration Complete')
if args.step == 'configs':
sys.exit(0)
### What needs doing?
if not image_configs.refresh_state(
step=args.step, only=args.only, skip=args.skip, revise=args.revise):
log.info('No pending actions to take at this time.')
sys.exit(0)
if args.step == 'state' or args.step == 'rollback':
sys.exit(0)
# install firmware if missing
install_qemu_firmware()
### Build/Import/Publish with Packer
env = os.environ | {
'TZ': 'UTC',
'PACKER_CACHE_DIR': 'work/packer_cache'
}
packer_cmd = [
'packer', 'build', '-timestamp-ui',
'-parallel-builds', str(args.parallel)
]
if args.no_color:
packer_cmd.append('-color=false')
if args.use_broker:
packer_cmd += ['-var', 'USE_BROKER=1']
if args.debug:
# do not add '-debug', it will pause between steps
packer_cmd += ['-var', 'DEBUG=1']
for var_file in args.vars:
packer_cmd.append(f"-var-file={var_file}")
packer_cmd += ['.']
log.info('Executing Packer...')
log.debug(packer_cmd)
out = io.StringIO()
p = Popen(packer_cmd, stdout=PIPE, encoding='utf8', env=env)
while p.poll() is None:
text = p.stdout.readline()
out.write(text)
print(text, end="")
if p.returncode != 0:
log.critical('Packer Failure')
sys.exit(p.returncode)
log.info('Packer Completed')
# update final state in work/images.yaml
# TODO: do we need to do all of this or just save all the image_configs?
image_configs.refresh_state(
step='final',
only=args.only,
skip=args.skip
)
log.info('Build Finished')

View File

@ -0,0 +1,98 @@
#!/usr/bin/env python3
# vim: ts=4 et:
# Ensure we're using the Python virtual env with our installed dependencies
import os
import sys
import textwrap
NOTE = textwrap.dedent("""
This script is meant to be run as a Packer post-processor, and Packer is only
meant to be executed from the main 'build' script, which is responsible for
setting up the work environment.
""")
sys.pycache_prefix = 'work/__pycache__'
if not os.path.exists('work'):
print('FATAL: Work directory does not exist.', file=sys.stderr)
print(NOTE, file=sys.stderr)
exit(1)
# Re-execute using the right virtual environment, if necessary.
venv_args = [os.path.join('work', 'bin', 'python3')] + sys.argv
if os.path.join(os.getcwd(), venv_args[0]) != sys.executable:
print("Re-executing with work environment's Python...\n", file=sys.stderr)
os.execv(venv_args[0], venv_args)
# We're now in the right Python environment
import argparse
import logging
from ruamel.yaml import YAML
import clouds
from image_config_manager import ImageConfigManager
### Constants & Variables
ACTIONS = ['local', 'upload', 'import', 'publish', 'release']
LOGFORMAT = '%(name)s - %(levelname)s - %(message)s'
### Command Line & Logging
parser = argparse.ArgumentParser(description=NOTE)
parser.add_argument('--debug', action='store_true', help='enable debug output')
parser.add_argument(
'--use-broker', action='store_true',
help='use the identity broker to get credentials')
parser.add_argument('action', choices=ACTIONS)
parser.add_argument('image_keys', metavar='IMAGE_KEY', nargs='+')
args = parser.parse_args()
log = logging.getLogger(args.action)
log.setLevel(logging.DEBUG if args.debug else logging.INFO)
# log to STDOUT so that it's not all red when executed by packer
console = logging.StreamHandler(sys.stdout)
console.setFormatter(logging.Formatter(LOGFORMAT))
log.addHandler(console)
log.debug(args)
# set up credential provider, if we're going to use it
if args.use_broker:
clouds.set_credential_provider(debug=args.debug)
# load build configs
configs = ImageConfigManager(
conf_path='work/configs/images.conf',
yaml_path='work/images.yaml',
log=args.action
)
yaml = YAML()
yaml.explicit_start = True
for image_key in args.image_keys:
image_config = configs.get(image_key)
if args.action == 'local':
image_config.convert_image()
elif args.action == 'upload':
if image_config.storage:
image_config.upload_image()
elif args.action == 'import':
clouds.import_image(image_config)
elif args.action == 'publish':
clouds.publish_image(image_config)
elif args.action == 'release':
pass
# TODO: image_config.release_image() - configurable steps to take on remote host
# save per-image metadata
image_config.save_metadata(args.action)

View File

@ -0,0 +1,51 @@
# vim: ts=4 et:
from . import aws, nocloud, azure, gcp, oci
ADAPTERS = {}
def register(*mods):
for mod in mods:
cloud = mod.__name__.split('.')[-1]
if p := mod.register(cloud):
ADAPTERS[cloud] = p
register(
aws, # well-tested and fully supported
nocloud, # beta, but supported
azure, # alpha, needs testing, lacks import and publish
gcp, # alpha, needs testing, lacks import and publish
oci, # alpha, needs testing, lacks import and publish
)
# using a credential provider is optional, set across all adapters
def set_credential_provider(debug=False):
from .identity_broker_client import IdentityBrokerClient
cred_provider = IdentityBrokerClient(debug=debug)
for adapter in ADAPTERS.values():
adapter.cred_provider = cred_provider
### forward to the correct adapter
# TODO: latest_imported_tags(...)
def get_latest_imported_tags(config):
return ADAPTERS[config.cloud].get_latest_imported_tags(
config.project,
config.image_key
)
def import_image(config):
return ADAPTERS[config.cloud].import_image(config)
def delete_image(config, image_id):
return ADAPTERS[config.cloud].delete_image(image_id)
def publish_image(config):
return ADAPTERS[config.cloud].publish_image(config)

View File

@ -0,0 +1,397 @@
# NOTE: not meant to be executed directly
# vim: ts=4 et:
import logging
import hashlib
import os
import subprocess
import time
from datetime import datetime
from .interfaces.adapter import CloudAdapterInterface
from image_tags import DictObj, ImageTags
class AWSCloudAdapter(CloudAdapterInterface):
IMAGE_INFO = [
'revision', 'imported', 'import_id', 'import_region', 'published',
]
CRED_MAP = {
'access_key': 'aws_access_key_id',
'secret_key': 'aws_secret_access_key',
'session_token': 'aws_session_token',
}
ARCH = {
'aarch64': 'arm64',
'x86_64': 'x86_64',
}
BOOT_MODE = {
'bios': 'legacy-bios',
'uefi': 'uefi',
}
@property
def sdk(self):
# delayed import/install of SDK until we want to use it
if not self._sdk:
try:
import boto3
except ModuleNotFoundError:
subprocess.run(['work/bin/pip', 'install', '-U', 'boto3'])
import boto3
self._sdk = boto3
return self._sdk
def session(self, region=None):
if region not in self._sessions:
creds = {'region_name': region} | self.credentials(region)
self._sessions[region] = self.sdk.session.Session(**creds)
return self._sessions[region]
@property
def regions(self):
if self.cred_provider:
return self.cred_provider.get_regions(self.cloud)
# list of all subscribed regions
return {r['RegionName']: True for r in self.session().client('ec2').describe_regions()['Regions']}
@property
def default_region(self):
if self.cred_provider:
return self.cred_provider.get_default_region(self.cloud)
# rely on our env or ~/.aws config for the default
return None
def credentials(self, region=None):
if not self.cred_provider:
# use the cloud SDK's default credential discovery
return {}
creds = self.cred_provider.get_credentials(self.cloud, region)
# return dict suitable to use for session()
return {self.CRED_MAP[k]: v for k, v in creds.items() if k in self.CRED_MAP}
def _get_images_with_tags(self, project, image_key, tags={}, region=None):
ec2r = self.session(region).resource('ec2')
req = {'Owners': ['self'], 'Filters': []}
tags |= {
'project': project,
'image_key': image_key,
}
for k, v in tags.items():
req['Filters'].append({'Name': f"tag:{k}", 'Values': [str(v)]})
return sorted(
ec2r.images.filter(**req), key=lambda k: k.creation_date, reverse=True)
# necessary cloud-agnostic image info
# TODO: still necessary? maybe just incoroporate into new latest_imported_tags()?
def _image_info(self, i):
tags = ImageTags(from_list=i.tags)
return DictObj({k: tags.get(k, None) for k in self.IMAGE_INFO})
# get the latest imported image's tags for a given build key
def get_latest_imported_tags(self, project, image_key):
images = self._get_images_with_tags(
project=project,
image_key=image_key,
)
if images:
# first one is the latest
return ImageTags(from_list=images[0].tags)
return None
# import an image
# NOTE: requires 'vmimport' role with read/write of <s3_bucket>.* and its objects
def import_image(self, ic):
log = logging.getLogger('import')
description = ic.image_description
session = self.session()
s3r = session.resource('s3')
ec2c = session.client('ec2')
ec2r = session.resource('ec2')
bucket_name = 'alpine-cloud-images.' + hashlib.sha1(os.urandom(40)).hexdigest()
bucket = s3r.Bucket(bucket_name)
log.info('Creating S3 bucket %s', bucket.name)
bucket.create(
CreateBucketConfiguration={'LocationConstraint': ec2c.meta.region_name}
)
bucket.wait_until_exists()
s3_url = f"s3://{bucket.name}/{ic.image_file}"
try:
log.info('Uploading %s to %s', ic.image_path, s3_url)
bucket.upload_file(str(ic.image_path), ic.image_file)
# import snapshot from S3
log.info('Importing EC2 snapshot from %s', s3_url)
ss_import_opts = {
'DiskContainer': {
'Description': description, # https://github.com/boto/boto3/issues/2286
'Format': 'VHD',
'Url': s3_url,
},
'Encrypted': True if ic.encrypted else False,
# NOTE: TagSpecifications -- doesn't work with ResourceType: snapshot?
}
if type(ic.encrypted) is str:
ss_import_opts['KmsKeyId'] = ic.encrypted
ss_import = ec2c.import_snapshot(**ss_import_opts)
ss_task_id = ss_import['ImportTaskId']
while True:
ss_task = ec2c.describe_import_snapshot_tasks(
ImportTaskIds=[ss_task_id]
)
task_detail = ss_task['ImportSnapshotTasks'][0]['SnapshotTaskDetail']
if task_detail['Status'] not in ['pending', 'active', 'completed']:
msg = f"Bad EC2 snapshot import: {task_detail['Status']} - {task_detail['StatusMessage']}"
log.error(msg)
raise RuntimeError(msg)
if task_detail['Status'] == 'completed':
snapshot_id = task_detail['SnapshotId']
break
time.sleep(15)
except Exception:
log.error('Unable to import snapshot from S3:', exc_info=True)
raise
finally:
# always cleanup S3, even if there was an exception raised
log.info('Cleaning up %s', s3_url)
bucket.Object(ic.image_file).delete()
bucket.delete()
# tag snapshot
snapshot = ec2r.Snapshot(snapshot_id)
try:
log.info('Tagging EC2 snapshot %s', snapshot_id)
tags = ic.tags
tags.Name = tags.name # because AWS is special
snapshot.create_tags(Tags=tags.as_list())
except Exception:
log.error('Unable to tag snapshot:', exc_info=True)
log.info('Removing snapshot')
snapshot.delete()
raise
# register AMI
try:
log.info('Registering EC2 AMI from snapshot %s', snapshot_id)
img = ec2c.register_image(
Architecture=self.ARCH[ic.arch],
BlockDeviceMappings=[{
'DeviceName': '/dev/xvda',
'Ebs': {
'SnapshotId': snapshot_id,
'VolumeType': 'gp3'
}
}],
Description=description,
EnaSupport=True,
Name=ic.image_name,
RootDeviceName='/dev/xvda',
SriovNetSupport='simple',
VirtualizationType='hvm',
BootMode=self.BOOT_MODE[ic.firmware],
)
except Exception:
log.error('Unable to register image:', exc_info=True)
log.info('Removing snapshot')
snapshot.delete()
raise
image_id = img['ImageId']
image = ec2r.Image(image_id)
try:
# tag image (adds imported tag)
log.info('Tagging EC2 AMI %s', image_id)
tags.imported = datetime.utcnow().isoformat()
tags.import_id = image_id
tags.import_region = ec2c.meta.region_name
image.create_tags(Tags=tags.as_list())
except Exception:
log.error('Unable to tag image:', exc_info=True)
log.info('Removing image and snapshot')
image.delete()
snapshot.delete()
raise
# update ImageConfig with imported tag values, minus special AWS 'Name'
tags.pop('Name', None)
ic.__dict__ |= tags
# delete an (unpublished) image
def delete_image(self, image_id):
log = logging.getLogger('build')