diff --git a/CONFIGURATION.md b/CONFIGURATION.md new file mode 100644 index 0000000..f3b9ccf --- /dev/null +++ b/CONFIGURATION.md @@ -0,0 +1,318 @@ +# Configuration + +All the configuration for building image variants is defined by multiple +config files; the base configs for official Alpine Linux cloud images are in +the [`configs/`](configs/) directory. + +We use [HOCON](https://github.com/lightbend/config/blob/main/HOCON.md) for +configuration -- this primarily facilitates importing deeper configs from +other files, but also allows the extension/concatenation of arrays and maps +(which can be a useful feature for customization), and inline comments. + +---- +## Resolving Work Environment Configs and Scripts + +If `work/configs/` and `work/scripts/` don't exist, the `build` script will +install the contents of the base [`configs/`](configs/) and [`scripts/`](scripts/) +directories, and overlay additional `configs/` and `scripts/` subdirectories +from `--custom` directories (if any). + +Files cannot be installed over existing files, with one exception -- the +[`configs/images.conf`](configs/images.conf) same-directory symlink. Because +the `build` script _always_ loads `work/configs/images.conf`, this is the hook +for "rolling your own" custom Alpine Linux cloud images. + +The base [`configs/images.conf`](configs/images.conf) symlinks to +[`alpine.conf`](configs/images.conf), but this can be overridden using a +`--custom` directory containing a new `configs/images.conf` same-directory +symlink pointing to its custom top-level config. + +For example, the configs and scripts in the [`overlays/testing/`](overlays/testing/) +directory can be resolved in a _clean_ work environment with... +``` +./build configs --custom overlays/testing +``` +This results in the `work/configs/images.conf` symlink to point to +`work/configs/alpine-testing.conf` instead of `work/configs/alpine.conf`. + +If multiple directories are specified with `--custom`, they are applied in +the order given. + +---- +## Top-Level Config File + +Examples of top-level config files are [`configs/alpine.conf`](configs/alpine.conf) +and [`overlays/testing/configs/alpine-testing.conf`](overlays/testing/configs/alpine-testing.conf). + +There are three main blocks that need to exist (or be `import`ed into) the top +level HOCON configuration, and are merged in this exact order: + +### `Default` + +All image variant configs start with this block's contents as a starting point. +Arrays and maps can be appended by configs in `Dimensions` and `Mandatory` +blocks. + +### `Dimensions` + +The sub-blocks in `Dimensions` define the "dimensions" a variant config is +comprised of, and the different config values possible for that dimension. +The default [`alpine.conf`](configs/alpine.conf) defines the following +dimensional configs: + +* `version` - Alpine Linux _x_._y_ (plus `edge`) versions +* `arch` - machine architectures, `x86_64` or `aarch64` +* `firmware` - supports launching via legacy BIOS or UEFI +* `bootstrap` - the system/scripts responsible for setting up an instance + during its initial launch +* `cloud` - for specific cloud platforms + +The specific dimensional configs for an image variant are merged in the order +that the dimensions are listed. + +### `Mandatory` + +After a variant's dimensional configs have been applied, this is the last block +that's merged to the image variant configuration. This block is the ultimate +enforcer of any non-overrideable configuration across all variants, and can +also provide the last element to array config items. + +---- +## Dimensional Config Directives + +Because a full cross-product across all dimensional configs may produce images +variants that are not viable (i.e. `aarch64` simply does not support legacy +`bios`), or may require further adjustments (i.e. the `aws` `aarch64` images +require an additional kernel module from `3.15` forward, which aren't available +in previous versions), we have two special directives which may appear in +dimensional configs. + +### `EXCLUDE` array + +This directive provides an array of dimensional config keys which are +incompatible with the current dimensional config. For example, +[`configs/arch/aarch64.conf`](configs/arch/aarch64.conf) specifies... +``` +# aarch64 is UEFI only +EXCLUDE = [bios] +``` +...which indicates that any image variant that includes both `aarch64` (the +current dimensional config) and `bios` configuration should be skipped. + +### `WHEN` block + +This directive conditionally merges additional configuration ***IF*** the +image variant also includes a specific dimensional config key (or keys). In +order to handle more complex situations, `WHEN` blocks may be nested. For +example, [`configs/cloud/aws.conf`](configs/cloud/aws.conf) has... +``` +WHEN { + aarch64 { + # new AWS aarch64 default... + kernel_modules.gpio_pl061 = true + initfs_features.gpio_pl061 = true + WHEN { + "3.14 3.13 3.12" { + # ...but not supported for older versions + kernel_modules.gpio_pl061 = false + initfs_features.gpio_pl061 = false + } + } + } +``` +This configures AWS `aarch64` images to use the `gpio_pl061` kernel module in +order to cleanly shutdown/reboot instances from the web console, CLI, or SDK. +However, this module is unavailable on older Alpine versions. + +Spaces in `WHEN` block keys serve as an "OR" operator; nested `WHEN` blocks +function as "AND" operators. + +---- +## Config Settings + +**Scalar** values can be simply overridden in later configs. + +**Array** and **map** settings in later configs are merged with the previous +values, _or entirely reset if it's first set to `null`_, for example... +``` +some_array = [ thing ] +# [...] +some_array = null +some_array = [ other_thing ] +``` + +Mostly in order of appearance, as we walk through +[`configs/alpine.conf`](configs/alpine.conf) and the deeper configs it +imports... + +### `project` string + +This is a unique identifier for the whole collection of images being built. +For the official Alpine Linux cloud images, this is set to +`https://alpinelinux.org/cloud`. + +When building custom images, you **MUST** override **AT LEAST** this setting to +avoid image import and publishing collisions. + +### `name` array + +The ultimate contents of this array contribute to the overall naming of the +resultant image. Almost all dimensional configs will add to the `name` array, +with two notable exceptions: **version** configs' contribution to this array is +determined when `work/images.yaml` is resolved, and is set to the current +Alpine Linux release (_x.y.z_ or _YYYYMMDD_ for edge); also because +**cloud** images are isolated from each other, it's redundant to include that +in the image name. + +### `description` array + +Similar to the `name` array, the elements of this array contribute to the final +image description. However, for the official Alpine configs, only the +**version** dimension adds to this array, via the same mechanism that sets the +revision for the `name` array. + +### `motd` map + +This setting controls the contents of what ultimately gets written into the +variant image's `/etc/motd` file. Later configs can add additional messages, +replace existing contents, or remove them entirely (by setting the value to +`null`). + +The `motd.version_notes` and `motd.release_notes` settings have slightly +different behavior: +* if the Alpine release (_x.y.z_) ends with `.0`, `release_notes` is dropped + to avoid redundancy +* edge versions are technically not released, so both of these notes are + dropped from `/etc/motd` +* otherwise, `version_notes` and `release_notes` are concatenated together as + `release_notes` to avoid a blank line between them + +### `scripts` array + +These are the scripts that will be executed by Packer, in order, to do various +setup tasks inside a variant's image. The `work/scripts/` directory contains +all scripts, including those that may have been added via `build --custom`. + +### `script_dirs` array + +Directories (under `work/scripts/`) that contain additional data that the +`scripts` will need. Packer will copy these to the VM responsible for setting +up the variant image. + +### `size` string + +The size of the image disk, by default we use `1G` (1 GiB). This disk may (or +may not) be further partitioned, based on other factors. + +### `login` string + +The image's primary login user, set to `alpine`. + +### `local_format` string + +The local VM's disk image format, set to `qcow2`. + +### `repos` map + +Defines the contents of the image's `/etc/apk/repositories` file. The map's +key is the URL of the repo, and the value determines how that URL will be +represented in the `repositories` file... +| value | result | +|-|-| +| `null` | make no reference to this repo | +| `false` | this repo is commented out (disabled) | +| `true` | this repo is enabled for use | +| _tag_ | enable this repo with `@`_`tag`_ | + +### `packages` map + +Defines what APK packages to add/delete. The map's key is the package +name, and the value determines whether (or not) to install/uninstall the +package... +| value | result | +|-|-| +| `null` | don't add or delete | +| `false` | explicitly delete | +| `true` | add from default repos | +| _tag_ | add from `@`_`tag`_ repo | +| `--no-scripts` | add with `--no-scripts` option | +| `--no-scripts` _tag_ | add from `@`_`tag`_ repo, with `--no-scripts` option | + +### `services` map of maps + +Defines what services are enabled/disabled at various runlevels. The first +map's key is the runlevel, the second key is the service. The service value +determines whether (or not) to enable/disable the service at that runlevel... +| value | result | +|-|-| +| `null` | don't enable or disable | +| `false` | explicitly disable | +| `true` | explicitly enable | + +### `kernel_modules` map + +Defines what kernel modules are specified in the boot loader. The key is the +kernel module, and the value determines whether or not it's in the final +list... +| value | result | +|-|-| +| `null` | skip | +| `false` | skip | +| `true` | include | + +### `kernel_options` map + +Defines what kernel options are specified on the kernel command line. The keys +are the kernel options, the value determines whether or not it's in the final +list... +| value | result | +|-|-| +| `null` | skip | +| `false` | skip | +| `true` | include | + +### `initfs_features` map + +Defines what initfs features are included when making the image's initramfs +file. The keys are the initfs features, and the values determine whether or +not they're included in the final list... +| value | result | +|-|-| +| `null` | skip | +| `false` | skip | +| `true` | include | + +### `builder` string + +The Packer builder that's used to build images. This is set to `qemu`. + +### `qemu.machine_type` string + +The QEMU machine type to use when building local images. For x86_64, this is +set to `null`, for aarch64, we use `virt`. + +### `qemu.args` list of lists + +Additional QEMU arguments. For x86_64, this is set to `null`; but aarch64 +requires several additional arguments to start an operational VM. + +### `qemu.firmware` string + +The path to the QEMU firmware (installed in `work/firmware/`). This is only +used when creating UEFI images. + +### `bootloader` string + +The bootloader to use, currently `extlinux` or `grub-efi`. + +### `access` map + +When images are published, this determines who has access to those images. +The key is the cloud account (or `PUBLIC`), and the value is whether or not +access is granted, `true` or `false`/`null`. + +### `regions` map + +Determines where images should be published. The key is the region +identifier (or `ALL`), and the value is whether or not to publish to that +region, `true` or `false`/`null`. \ No newline at end of file diff --git a/README.md b/README.md index a2ac172..d9dd491 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,3 @@ -## _**NOTE: This is a Work-in-Progress**_ - -_It is intended that this will eventually replace -https://gitlab.alpinelinux.org/alpine/cloud/alpine-ec2-ami -as the offical multi-cloud image builder for Alpine Linux._ - ----- - # Alpine Linux Cloud Image Builder This repository contains the code and and configs for the build system used to @@ -18,26 +10,64 @@ own customized images. To get started with offical pre-built Alpine Linux cloud images, visit https://alpinelinux.org/cloud. Currently, we build official images for the -following providers: +following cloud platforms... * AWS -You should also be able to find the most recently published Alpine Linux -images via your cloud provider's web console, or programatically query their -API with a CLI tool or library. +...we are working on also publishing offical images to other major cloud +providers. -_(TODO: examples)_ +Each published image's name contains the Alpine version release, architecture, firmware, bootstrap, and image revision. These details (and more) are also +tagged on the images... + +| Tag | Description / Values | +|-----|----------------------| +| name | `alpine-`_`release-arch-firmware-bootstrap-`_`r`_`revision`_ | +| project | `https://alpinelinux.org/cloud` | +| image_key | _`release-arch-firmware-bootstrap-cloud`_ | +| version | Alpine version (_`x.y`_ or `edge`) | +| release | Alpine release (_`x.y.z`_ or _`YYYYMMDD`_ for edge) | +| arch | architecture (`aarch64` or `x86_64`) | +| firmware | boot mode (`bios` or `uefi`) | +| bootstrap | initial bootstrap system (`tiny` = tiny-ec2-bootstrap) | +| cloud | provider short name (`aws`) | +| revision | image revision number | +| imported | image import timestamp | +| import_id | imported image id | +| import_region | imported image region | +| published | image publication timestamp | +| description | image description | + +Although AWS does not allow cross-account filtering by tags, the image name can +still be used to filter images. For example, to get a list of available Alpine +3.x aarch64 images in AWS eu-west-2... +``` +aws ec2 describe-images \ + --region eu-west-2 \ + --owners 538276064493 \ + --filters \ + Name=name,Values='alpine-3.*-aarch64-*' \ + Name=state,Values=available \ + --output text \ + --query 'reverse(sort_by(Images, &CreationDate))[].[ImageId,Name,CreationDate]' +``` +To get just the most recent matching image, use... +``` + --query 'max_by(Image, &CreationDate).[ImageId,Name,CreationDate]' +``` ---- ## Build System The build system consists of a number of components: -* the primary `build` script and related cloud-specific helpers -* a directory of `configs/` defining the set of images to be built -* a Packer `alpine.pkr.hcl` orchestrating the images' local build, as well as - importing them to cloud providers and publishing them to desitnation regions -* a directory of `scripts/` which set up the images' contents during - provisioning +* the primary `build` script +* the `configs/` directory, defining the set of images to be built +* the `scripts/` directory, containing scripts and related data used to set up + image contents during provisioning +* the Packer `alpine.pkr.hcl`, which orchestrates build, import, and publishing + of images +* the `cloud_helper.py` script that Packer runs in order to do cloud-specific + import and publish operations ### Build Requirements * [Python](https://python.org) (3.9.7 is known to work) @@ -47,182 +77,105 @@ The build system consists of a number of components: ### Cloud Credentials -This build system relies on the cloud providers' Python API libraries to find -and use the necessary credentials -- via configuration in the user's home -directory (i.e. `~/.aws/...`, `~/.oci/...`, etc.) or with special environment -variables (i.e. `AWS_...`, `OCI_...`, etc.) +By default, the build system relies on the cloud providers' Python API +libraries to find and use the necessary credentials, usually via configuration +under the user's home directory (i.e. `~/.aws/`, `~/.oci/`, etc.) or or via +environment variables (i.e. `AWS_...`, `OCI_...`, etc.) -It is expected that each cloud provider's user/role will have been set up with -sufficient permission in order to accomplish the operations necessary to query, -import, and publish images; _it is highly recommended that no permissions are -granted beyond what is absolutely necessary_. +The credentials' user/role needs sufficient permission to query, import, and +publish images -- the exact details will vary from cloud to cloud. _It is +recommended that only the minimum required permissions are granted._ + +_We manage the credentials for publishing official Alpine images with an +"identity broker" service, and retrieve those credentials via the +`--use-broker` argument of the `build` script._ ### The `build` Script ``` -usage: build [-h] [--debug] [--clean] [--revise] {configs,local,import,publish} - [--custom DIR [DIR ...]] [--skip KEY [KEY ...]] [--only KEY [KEY ...]] +usage: build [-h] [--debug] [--clean] [--custom DIR [DIR ...]] + [--skip KEY [KEY ...]] [--only KEY [KEY ...]] [--revise] [--use-broker] [--no-color] [--parallel N] [--vars FILE [FILE ...]] + {configs,state,local,import,publish} -build steps: - configs resolve build configuration - local build local images - import import to cloud providers - publish set permissions and publish to cloud regions +positional arguments: (build up to and including this step) + configs resolve image build configuration + state refresh current image build state + local build images locally + import import local images to cloud provider default region + publish set image permissions and publish to cloud regions optional arguments: -h, --help show this help message and exit - --debug enable debug output (False) - --clean start with a clean work environment (False) - --revise bump revisions if images already published (False) + --debug enable debug output + --clean start with a clean work environment --custom DIR [DIR ...] overlay custom directory in work environment --skip KEY [KEY ...] skip variants with dimension key(s) --only KEY [KEY ...] only variants with dimension key(s) - --no-color turn off Packer color output (False) - --parallel N build N images in parallel (1) - --vars FILE [FILE ...] supply Packer with additional -vars-file(s) + --revise remove existing local/imported image, or bump + revision and rebuild if published + --use-broker use the identity broker to get credentials + --no-color turn off Packer color output + --parallel N build N images in parallel (default: 1) + --vars FILE [FILE ...] supply Packer with -vars-file(s) ``` -A `work/` directory will be created for its Python virtual environment, any -necessary Python libraries will be `pip install`ed, and `build` will execute -itself to ensure that it's running in the work environment. +The `build` script will automatically create a `work/` directory containing a +Python virtual environment if one does not already exist. This directory also +hosts other data related to building images. The `--clean` argument will +remove everything in the `work/` directory except for things related to the +Python virtual environment. -This directory also contains `configs/` and `scripts/` subdirs (with custom -overlays), UEFI firmware for QEMU, Packer cache, the generated `configs.yaml` -and `actions.yaml` configs, and the `images/` tree for local image builds. +If `work/configs/` or `work/scripts/` directories do not yet exist, they will +be populated with the base configuration and scripts from `configs/` and/or +`scripts/` directories. If any custom overlay directories are specified with +the `--custom` argument, their `configs/` and `scripts/` subdirectories are +also added to `work/configs/` and `work/scripts/`. -Use `--clean` if you want to re-overlay, re-download, re-generate, or rebuild -anything in the `work/` directory. To redo the Python virtual environment, -simply remove the `work/` directory and its contents, and it will be recreated -the next time `build` is run. +The "build step" positional argument deterimines the last step the `build` +script should execute -- all steps before this targeted step may also be +executed. That is, `build local` will first execute the `configs` step (if +necessary) and then the `state` step (always) before proceeding to the `local` +step. -### Build Steps +The `configs` step resolves configuration for all buildable images, and writes +it to `work/images.yaml`, if it does not already exist. -When executing `build` you also provide the target step you wish to reach. For -example, if you only want to build local images, use `build local`. Any -predecessor steps which haven't been done will also be executed -- that is, -`build local` also implies `build configs` if that step hasn't completed yet. +The `state` step always checks the current state of the image builds, +determines what actions need to be taken, and updates `work/images.yaml`. A +subset of image builds can be targeted by using the `--skip` and `--only` +arguments. The `--revise` argument indicates that any _unpublished_ local +or imported images should be removed and rebuilt; as _published_ images can't +be removed, `--revise` instead increments the _`revision`_ value to rebuild +new images. -The **configs** step determines the latest stable Alpine Linux release, and -ensures that the `configs/` and `scripts/` overlays, UEFI firmware, and -`configs.yaml` exist. This allows you to validate the generated build variant -configuration before attempting to build any images locally. +`local`, `import`, and `publish` steps are orchestrated by Packer. By default, +each image will be processed serially; providing the `--parallel` argument with +a value greater than 1 will parallelize operations. The degree to which you +can parallelze `local` image builds will depend on the local build hardware -- +as QEMU virtual machines are launched for each image being built. Image +`import` and `publish` steps are much more lightweight, and can support higher parallelism. -If `build` is moving on past **configs** to other steps, it will determine which -image variants to work on (based on `--skip` and `--only` values) and what -actions will be taken, based on existence of local/imported/published images, and -generate the `actions.yaml` file. Providing the `--revise` flag allows you to -rebuild local images that were previously built, reimport unpublished images to -cloud providers, and bump the "revision" value of previously published images -- -this is useful if published images require fixes but the Alpine release itself -isn't changing; published images are not removed (though they may be pruned once -their "end-of-life" date has passed). +The `local` step builds local images with QEMU, for those that are not already +built locally or have already been imported. -At this point, `build` executes Packer, which is responsible for the remaining -**local**, **import**, and **publish** steps -- and also for parallelization, if -the `--parallel` argument is given. Because build hardware varies, it is also -possible to tune a number of QEMU timeouts and memory requirements by providing -an HCL2 Packer Vars file and specifying `--vars ` to override the -defaults in `alpine.pkr.hcl`. +The `import` step imports the local images into the cloud providers' default +regions, unless they've already been imported. At this point the images are +not available publicly, allowing for additional testing prior to publishing. -### Packer and `alpine.pkr.hcl` - -Packer loads and merges `actions.yaml` and `configs.yaml`, and iterates the -resulting object in order to determine what it should do with each image -variant configuration. - -`alpine.pkr.hcl` defines two base `source` blocks -- `null` is used when an -image variant is already built locally and/or already imported to the -destination cloud provider; otherwise, the `qemu` source is used. - -The `qemu` builder spins up a QEMU virtual machine with a blank virtual disk -attached, using the latest stable Alpine Linux Virtual ISO, brings up the VM's -network, enables the SSH daemon, and sets a random password for root. - -If an image variant is to be **built locally**, the two dynamic provisioners copy -the required data for the setup scripts to the VM's `/tmp/` directory, and then -run those setup scripts. It's these scripts that are ultimately responsible for -installing and configuring the desired image on the attached virtual disk. -When the setup scripts are complete, the virtual machine is shut down, and the -resulting local disk image can be found at -`work/images///image.qcow2`. - -The dynamic post-processor uses the `cloud_helper.py` script to **import** a -local image to the cloud provider, and/or **publish** an imported image to the -cloud provider's destination regions, based on what actions are applicable for -that image variant. When the **publish** step is reapplied to an -already-published image, the script ensures that images have been copied to all -destination regions (for example, if the cloud provider recently added a new -region), and that all launch permissions are set as expected. +The `publish` step copies the image from the default region to other regions, +if they haven't already been copied there. This step will always update +image permissions, descriptions, tags, and deprecation date (if applicable) +in all regions where the image has been published. ### The `cloud_helper.py` Script -This script is only meant to be imported by `build` and called from Packer, and -provides a normalized cloud-agnostic way of doing common cloud operations -- -getting details about a variant's latest imported image, importing new local -image to the cloud, removing a previouly imported (but unpublished) image so it -can be replaced, or publishing an imported image to destination regions. +This script is meant to be called only by Packer from its `post-processor` +block for image `import` and `publish` steps. ---- ## Build Configuration -The `build` script generates `work/configs.yaml` based on the contents of the -top-level config file, `work/configs/configs.conf`; normally this is a symlink to -`alpine.conf`, but can be overridden for custom builds. All configs are -declared in [HOCON](https://github.com/lightbend/config/blob/master/HOCON.md) -format, which allows importing from other files, simple variable interpolation, -and easy merging of objects. This flexibility helps keep configuration -[DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself). - -The top-level `build.conf` has three main blocks, `Default` (default/starting -values), `Dimensions` (with configs that apply in different circumstances), and -`Mandatory` (required/final values). The configuration for these blocks are -merged in this exact order. - -### Dimensions and Build Variants - -Build variants _(I was watching Loki™ at the time...)_ are the sets of -dimensional "features" and related configuration details produced from a -Cartesian product across each of the dimensional keys. Dimensional configs are -merged together in the order they appear in `build.conf`. - -If two dimensional keys are incompatible (for example, **version/3.11** did not -yet support **arch/aarch64**), an `EXCLUDE` directive indicates that such a -variant is non-viable, and will be skipped. - -Likewise, if one dimension's configuration depends on the value of a different -dimensional key, the `WHEN` directive will supply the conditional config -details when that other dimensional key is part of the variant. - -Currently the base set of dimensions (and dimension keys) are... - -**version** - current "release" value for each is autodetected, and always a - component of an image's name -* **edge** ("release" value is the current UTC date) -* all *non-EOL* Alpine Linux versions - - **arch** - machine architecture - * **x86_64** (aka "amd64") - * **aarch64** (aka "arm64") - -**firmware** - machine boot firmware -* **bios** (legacy BIOS) -* **uefi** - -**bootstrap** - image instantiation bootstrap is provided by... -* **tiny** (tiny-cloud-boostrap) -* **cloudinit** (cloud-init) - -**cloud** - cloud provider or platform -* **aws** - Amazone Web Services / EC2 -* **oci** - Oracle Cloud Infrastructure _(WiP)_ -* **gcp** - Google Cloud Platform _(WiP)_ -* **azure** - Microsoft Azure _(WiP)_ - -...each dimension may (or may not) contribute to the image name or description, -if the dimensional key's config contributes to the `name` or `description` -array values. - -### Customized Builds - -_(TODO)_ \ No newline at end of file +For more in-depth information about how the build system configuration works, +how to create custom config overlays, and details about individual config +settings, see [CONFIGURATION.md](CONFIGURATION.md). diff --git a/alpine.pkr.hcl b/alpine.pkr.hcl index e55f28b..4956903 100644 --- a/alpine.pkr.hcl +++ b/alpine.pkr.hcl @@ -1,13 +1,19 @@ -# Enable script debug output, set via 'packer build -var DEBUG=1' +# Alpine Cloud Images Packer Configuration + +### Variables + +# include debug output from provisioning/post-processing scripts variable "DEBUG" { default = 0 } +# indicates cloud_helper.py should be run with --use-broker variable "USE_BROKER" { default = 0 } -# Tuneable based on perfomance of whatever Packer's running on, -# override with './build --vars ' +# tuneable QEMU VM parameters, based on perfomance of the local machine; +# overrideable via build script --vars parameter referencing a Packer +# ".vars.hcl" file containing alternate settings variable "qemu" { default = { boot_wait = { @@ -20,6 +26,7 @@ variable "qemu" { } } +### Local Data locals { debug_arg = var.DEBUG == 0 ? "" : "--debug" diff --git a/build b/build index 913911e..00c5b91 100755 --- a/build +++ b/build @@ -37,6 +37,7 @@ import logging import shutil import time +from glob import glob from subprocess import Popen, PIPE from urllib.request import urlopen @@ -47,7 +48,7 @@ from image_configs import ImageConfigManager ### Constants & Variables -STEPS = ['configs', 'actions', 'local', 'import', 'publish'] +STEPS = ['configs', 'state', 'local', 'import', 'publish'] LOGFORMAT = '%(asctime)s - %(name)s - %(levelname)s - %(message)s' WORK_CLEAN = {'bin', 'include', 'lib', 'pyvenv.cfg', '__pycache__'} WORK_OVERLAYS = ['configs', 'scripts'] @@ -100,46 +101,54 @@ def clean_work(): os.unlink(x) -def is_same_dir_symlink(x): - if not os.path.islink(x): +def is_images_conf(o, x): + if not all([ + o == 'configs', + x.endswith('/images.conf'), + os.path.islink(x), + ]): return False + # must also link to file in the same directory x_link = os.path.normpath(os.readlink(x)) return x_link == os.path.basename(x_link) -# TODO: revisit this to improve --custom overlay implementation def install_overlay(overlay): log.info("Installing '%s' overlay in work environment", overlay) dest_dir = os.path.join('work', overlay) os.makedirs(dest_dir, exist_ok=True) for src in unique_list(['.'] + args.custom): src_dir = os.path.join(src, overlay) - for x in os.listdir(src_dir): + for x in glob(os.path.join(src_dir, '**'), recursive=True): + x = x.removeprefix(src_dir + '/') src_x = os.path.join(src_dir, x) dest_x = os.path.join(dest_dir, x) - if is_same_dir_symlink(src_x): - rel_x = os.readlink(src_x) - else: - rel_x = os.path.relpath(src_x, dest_dir) - # TODO: only images.conf symlink can be overridden, in reality - if os.path.islink(dest_x): - # only same-dir symlinks can be overridden - if not is_same_dir_symlink(dest_x): - log.warning("Can't override %s with %s", dest_x, src_x) + if is_images_conf(overlay, src_x): + rel_x = os.readlink(src_x) + if os.path.islink(dest_x): + print(f"\toverriding {dest_x}") + os.unlink(dest_x) + + print(f"\tln -s {rel_x} {dest_x}") + os.symlink(rel_x, dest_x) + continue + + if os.path.isdir(src_x): + if not os.path.exists(dest_x): + log.debug('makedirs %s', dest_x) + os.makedirs(dest_x) + + if os.path.isdir(dest_x): continue - log.debug('overriding %s with %s', dest_x, src_x) - os.unlink(dest_x) - - elif os.path.exists(dest_x): - # we expect only symlnks in the overlay destination! - log.critical('Config overlay non-symlink detected: %s', dest_x) + if os.path.exists(dest_x): + log.critical('Unallowable destination overwirte detected: %s', dest_x) sys.exit(1) - log.debug('ln -sf %s %s', rel_x, dest_x) - os.symlink(rel_x, dest_x) + log.debug('cp -p %s %s', src_x, dest_x) + shutil.copy(src_x, dest_x) def install_overlays(): @@ -188,22 +197,24 @@ parser.add_argument( '--debug', action='store_true', help='enable debug output') parser.add_argument( '--clean', action='store_true', help='start with a clean work environment') -parser.add_argument( - '--revise', action='store_true', - help='bump revisions if images already published') -# positional argument -parser.add_argument( - 'step', choices=STEPS, help='build up to and including this step') # config options parser.add_argument( '--custom', metavar='DIR', nargs='+', action=are_args_valid(os.path.isdir), default=[], help='overlay custom directory in work environment') +# state options parser.add_argument( '--skip', metavar='KEY', nargs='+', action=remove_dupe_args(), default=[], help='skip variants with dimension key(s)') parser.add_argument( '--only', metavar='KEY', nargs='+', action=remove_dupe_args(), default=[], help='only variants with dimension key(s)') +parser.add_argument( + '--revise', action='store_true', + help='remove existing local/imported image, or bump revision and rebuild' + 'if published') +parser.add_argument( + '--use-broker', action='store_true', + help='use the identity broker to get credentials') # packer options parser.add_argument( '--no-color', action='store_true', help='turn off Packer color output') @@ -213,10 +224,9 @@ parser.add_argument( parser.add_argument( '--vars', metavar='FILE', nargs='+', action=are_args_valid(os.path.isfile), default=[], help='supply Packer with -vars-file(s)') +# positional argument parser.add_argument( - '--use-broker', action='store_true', - help='use the identity broker to get credentials') -# perhaps others? + 'step', choices=STEPS, help='build up to and including this step') args = parser.parse_args() log = logging.getLogger('build') @@ -261,7 +271,7 @@ if not image_configs.refresh_state( log.info('No pending actions to take at this time.') sys.exit(0) -if args.step == 'actions': +if args.step == 'state': sys.exit(0) # install firmware if missing diff --git a/configs/alpine.conf b/configs/alpine.conf index 1dd7f32..e3aa9b4 100644 --- a/configs/alpine.conf +++ b/configs/alpine.conf @@ -5,7 +5,7 @@ # *AT LEAST* the 'project' setting with a unique identifier string value # via a "config overlay" to avoid image import and publishing collisions. -project = https://alpinelinux.org/cloud +project = "https://alpinelinux.org/cloud" # all build configs start with these Default { @@ -17,13 +17,14 @@ Default { motd { welcome = "Welcome to Alpine!" - wiki = \ - "The Alpine Wiki contains a large amount of how-to guides and general\n"\ - "information about administrating Alpine systems.\n"\ - "See ." - version_notes = "Release Notes:\n"\ - "* " - release_notes = "*