Commit Graph

57 Commits

Author SHA1 Message Date
4afeeef91d Pull upstream for 3.16 2022-06-20 12:46:52 +02:00
Jake Buchholz Göktürk
7ac2029267 Minor Fixes
* switch 3.12 back to tiny-ec2-bootstrap (tiny-cloud requires ifupdown-ng)
* restrict cloud-init to 3.15+
* pad UEFI firmware so QEMU works with aarch64 again
* kinda fix motd release_notes (more to do yet)
2022-03-19 14:24:26 -07:00
5e8967002e always apk upgrade builder VM, fix --clean of symlinked dirs
Fix for softlinks to directories during --clean / always use latest packages within a release at build time to pick up latest security fixes etc.
2022-03-12 17:56:14 +00:00
Jake Buchholz Göktürk
c9665f68dc Add "cloudinit" to Bootstrap Dimension
* cloudinit bootstrap is functional now
* remove cloudinit from testing overlay
* add e2fsprogs to all images

Resolves #100
2022-02-24 20:10:24 -08:00
Jake Buchholz Göktürk
63a522149d Tiny Cloud / set default NTP server
* switch to tiny-cloud instead of tiny-ec2-bootstrap
* set default NTP server, if configured
* add default /etc/network/interfaces
* add urlopen() timeout to mitigate ipv6 issues connecting to alpinelinux.org
2022-01-30 19:18:09 +00:00
Jake Buchholz Göktürk
274d883acb alpine-cloud-images, part three 2021-11-28 23:04:28 +00:00
Jake Buchholz Göktürk
6674286b46 alpine-cloud-images, part two 2021-11-23 06:09:18 +00:00
Jake Buchholz Göktürk
e01e56bfa3 alpine-cloud-images, part one
This is the first MR to replace !125, and contains everything except the new python stuff -- which is part two.
2021-11-07 12:37:56 -08:00
tomalok
b8ac181435
Update for Version 3.14 (#122)
Update for Version 3.14

* remove 3.10 build
* fix aarch64 AMI's /etc/default/grub (resolves #121)
* bump revision for aarch64 3.13 & 3.12 (to rebuild with fix)
* fix comment on 3.13
* fix comment on 3.14
* new 3.14.0 AMIs released
2021-06-15 12:34:38 -07:00
Mike Crute
b578a39eb7 Remove backport hack for tiny-ec2-bootstrap 2021-05-03 09:49:16 -07:00
tomalok
88f3f1374e
Autodetect Current Revision of Alpine Version (#113)
* continue to use provided 'release' value if specified
* continue to use 'edge' for edge versions
* deduce 'release' value from the version on the alpine-base APK in https://dl-cdn.alpinelinux.org/alpine/v<version/main/<arch>/ 
* update test profile with 3.13
2021-02-02 20:13:33 -08:00
tomalok
2bf6727f67
Release Alpine 3.13.0 (#108)
3.13.0 release
* add version 3.13 profile
* 3.13 end-of-life is 2022-11-01
* note we're an official Alpine project now!
* releases/alpine.yaml has been "--trim release"'d
* updated releases/README.md

scripts/builder.py
* fix ReleaseReadme() to not die when release contains "_rc"
* add --trim to releases-yaml
* tweak some --help for release-yaml and prune

scripts/setup-ami
* explicitly lock AMI root account
2021-01-14 23:53:40 -08:00
tomalok
53fd1d27b1
Builder Overhaul (#106)
Subcommands
* merge 'resolve-profiles' and 'make-amis' into 'amis'
* rename 'update-releases' to 'release-yaml'
* rename 'gen-release-readme' to 'release-readme'
* rename 'prune-amis' to 'prune'
* reorder to match the usual workflow
* use argparse mutually-exclusive group where appropriate
* use argparse 'metavar' and 'nargs' for more salient help

release
* can now specify multiple AMIs on command line
* add explicit '--private' argument
* if no '--private', '--public', or '--allow-account' is specified, default to propagate the source AMI's permissions to its copies
* move 'iter_regions' and 'get*image' methods out of ReleaseAMIs class because they're also used elsewhere
* 'update_image_permissions' resets perms before adding new perms
* pending_copy loop, reports on everything in progress, waits 3m before reporting on everything again, and then waits 30s between reports
* pending_copy also notes when a copy has completed (and only queues for pending_perms if they need adjustment)

Releases class
* used by release-yaml and prune subcommands
* caches region client objects for later use (by prune)
* loads images from region - either from a profile or "unknown" (no profile tag)
* builds the releases object - now structured release -> build (instead of build -> release)

ReleasesReadme
* works with new releases object format
* improve sorting and selection of latest per version per-build AMIs
* empty cell if a region doesn't happen to have a build AMI there

PruneAMIs
* rename 'version' level to 'end-of-life'
* add 'UNKNOWN' pruning level
* works, even if you don't want to --use-broker
* --keep N - keeps an additional N AMIs that would otherwise have been purged per build
* --defer-eol DAYS - give EOL AMIs a grace period past their official EOL date
* no AMI deletion happens unless --no-pretend arg is provided
* improve pruning criteria scan and candidate selection

Co-authored-by: Jake Buchholz <jake@jakesys.net>
2021-01-04 17:36:15 -08:00
Jake Buchholz
e5b574f48f Post-Build Cleanup, etc.
scripts/builder.py...

GenReleaseReadme:
* combine with ReleaseReadmeUpdater
* generates README_<profile>.md
* README_alpine.md is a symlink to README.md
* don't crash when README doesn't preexist
* append image list to README if no list found to replace

MakeAMIs:
* collect all artifact IDs and report after all builds
* don't update releases/readme

PruneAMIs:
* defaults to pretend mode, unless --no-pretend
* improve readability

UpdateReleases:
* replace code with what was RefreshReleases
2020-12-22 15:42:20 -08:00
tomalok
4494aa4463
Release the latest versions (#101)
Release the latest versions
* v3.12.3 (x86_64 & aarch64)
* v3.11.7 (x86_64)
* v3.10.5-r1 (x86_64)
* today's edge (x86_64 & aarch64)
* sort the release AMIs by region

builder.py timings, roughly
* amis - 23m
* release (serial) - 1h38m
* refresh-releases - 4m
* gen-release-readme - instantaneous
2020-12-19 11:16:48 -08:00
tomalok
a530e331f3
Add refresh-releases subcommand, etc. (#97)
* Add refresh-releases subcommand, etc.

* builder.py
  + gen-release-readme
    - convert `build_time` to int
  + release
    - add `source_region` to copied AMI tags
    - check source AMI's permissions, queue for fixing, if necessary
  + refresh-releases
    - update releases/<profile>.yaml based on AMIs that exist in regions
  + explicitly call out `python-dateutil` dependency and `pip install` it into the venv

* Release Alpine 3.12.2 & today's edge
2020-12-14 22:24:29 -08:00
Mike Crute
3b4e395850
New Release Tool (#83)
* Add EC2 data types
* Add release command
2020-12-11 18:02:13 -08:00
tomalok
20ee5f5bc1
Define Bootloader in Profiles, etc. (#94)
* make it easier to switch between bootloaders
* experimental (non-working) EFI_STUB bootloader
* remove apk_tools & alpine_keys from profiles
* determine & install appropriate apk_toosl & alpine_keys in setup-ami based on version and arch.
2020-12-11 17:43:27 -08:00
tomalok
6e252ce9de
Fix 3.12.1 aarch64 Root Resize (#93)
release alpine-ami-3.12.1-aarch64-r1

Also...
* release new edge builds
* Alpine 3.9 is EOL
* build_instance_type set in profiles/arch/
* a couple comment fixes
2020-11-22 16:16:45 -08:00
Jake Buchholz
24bf01621f Fix assembly of /etc/network/interfaces
Include the other interface configs from /etc/network/interfaces.d/ after lo
and existing eth*, as was initially intended.

Also separate out the assembly code into its own script, as this is done both
on boot via eth-eni-setup and on hotplug events by etc-eni-hotplug.

Resolves #91
2020-11-15 20:31:36 -08:00
tomalok
e42c833553
eth-eni-setup init script (#87)
* eth-eni-setup init script

before networking starts up, makes sure eth interfaces match attached ENIs

also fixes a permissions problem with eth-eni-hotplug mdev config

* fix aarch64 build
2020-09-21 19:43:33 -07:00
Jake Buchholz
1cce13e722 remove leftover set -x from setup-ami testing 2020-09-15 22:17:16 -07:00
Jake Buchholz
c6f5325873 ENI Hotplugging, etc.
ENI Hotplug / udhcpc script
* works with all Alpine versions back to 3.9
* udhcpc handles ENI's primary IPv4
* post-bound/post-renews eth-eni-hook handles secondary IPv4 & IPv6 addresses, route tables, and rules

setup-ami tweaks
* move scripts to be installed into setup-ami.d/
* move config snippets into setup-ami.d/etc/ (previously embedded in setup-ami)
2020-09-15 22:17:16 -07:00
Jake Buchholz
a9ba2532df udhcpc hooks for ENI IPv6 & secondary IPv4
Automatically sets up any IPv6 and secondary IPv4 on instance ENIs when DHCP leases are bound or renewed on that interface.

Resolves #70
2020-09-03 15:04:26 -07:00
Mike Crute
27491bcb20 Add argument checking for commands 2020-08-22 20:56:18 +00:00
Mike Crute
4df71cdc07 Use logging instead of print 2020-08-22 20:56:18 +00:00
Mike Crute
62262b6630 Fix rate-limiting error 2020-08-22 20:56:18 +00:00
tomalok
bbd08c72fe
Fix nvme-ebs-links Installation (#81)
ensure that ownership and permissions are set properly
2020-08-17 19:06:08 -07:00
tomalok
10058c1113
Fix 'revision' and 'end_of_life' (#80)
Also...
* update alpine.conf with 3.12
* update apk-tools and alpine-keys
* use test profile to test fixes and newer features
2020-08-17 11:11:50 -07:00
Jake Buchholz
2b76c6ebf6 support modification of default AMI user 2020-08-12 18:12:22 -07:00
Jake Buchholz
d593de3833 Optional Additional Setup
Profiles can specify 'setup_script' to do additional things.  If additional files/dirs are required, a 'setup_copy' map will copy them to the build instance so that 'setup_script' can use/install them.

TBD: docs.
2020-08-12 18:12:22 -07:00
Mike Crute
9d672fbd6a Support ARM instance types 2020-05-30 15:07:45 -07:00
Mike Crute
8a09fdda0e Add identity broker docs 2020-05-30 15:07:45 -07:00
Mike Crute
90f7408fc7 Flip broker usage flag 2020-05-30 15:07:45 -07:00
Mike Crute
b804661ffd Remove python3.8 dependency 2020-05-30 15:07:45 -07:00
Mike Crute
83d07e4b9a Initfs features are in profiles 2020-05-30 13:17:56 -07:00
Mike Crute
b1da6a47d6 Embed nvme config 2020-05-30 13:17:56 -07:00
Mike Crute
812eba9597 Remove release vars from packer 2020-05-30 13:17:56 -07:00
Mike Crute
b804d174b3 Allow building without broker 2020-05-30 12:14:48 -07:00
Mike Crute
df53323de9 Add some more python docs 2020-05-29 21:11:25 -07:00
Mike Crute
b53492723d Migrate full ami build to builder script 2020-05-29 20:58:53 -07:00
Mike Crute
5b2f32c9c8 Allow builder to be run by itself 2020-05-29 20:58:11 -07:00
Mike Crute
41f127d77b Authenticate per-region for make-amis 2020-05-28 17:57:13 -07:00
Mike Crute
bfc4bf99bf Convert packer.json builder to python 2020-05-28 17:55:46 -07:00
Mike Crute
7e60c7fb6a Combine all commands into a meta-command
There is an increasing need to share components among the various
commands, especially with the introduction of the identity broker API.
Rather than trying to assemble an importable python library of code for
the build process I think we can just combine everything into one file
and use argparse sub-commands to integrate them into a set of
individually callable scripts. This change does that integration.
2020-05-28 17:55:36 -07:00
Mike Crute
a36d0616bf Convert python scripts to argparse
This removes the manual command line handling and reformats the scripts
into main methods. This is paving the way for a more unified build tool.
2020-05-26 18:10:03 -07:00
Mike Crute
d63409acce Convert make-amis to python
This is paving the way for identity broker improvements for opt-in
regions. Eventually we'll need to hook some region logic into these
scripts so having them written in python will be helpful.
2020-05-22 18:23:32 -07:00
Mike Crute
1fd42af98d Refactor resolve-profile script
This is paving the way for identity broker improvements for opt-in
regions. The output is functionally identical between the two scripts
modulo the svcs change. Hopefully this makes the transformation process
a little more clear.
2020-05-21 16:45:17 -07:00
tomalok
8a46e41b77
fix regression on nvme mdev script (#55)
* sh doesn't allow nesting of prefix strip
* also update some minor test profile bits
* new AMI revisions (fixed nvme)
* edge AMI release should remain a timestamp
2019-07-27 03:02:20 +02:00
Jake Buchholz
0f1cc5c4b2 Latest AMIs per Version in releases/README.md
Only keep the latest AMIs per version in releases/README.md instead of per release.
2019-07-14 09:07:01 -07:00