Spent all weekend working on a simple Ansible script to generate and construct new FreeBSD images from a specific AMI. It was something of a cross work/qbsd support project. It was a lot of head-to-wall banging until I went back and carefully read the ec2_configinit rc script, and figured out how silly a mistake I’d made. To explain, I’ll describe the steps that configinit takes on boot; what I imagined was happening due to some package madness and uncontrolled tests previously; and finally what I had to do to fix it.
Configinit from @cperciva is just one of a short collection of rc scripts, designed to do stuff on FreeBSD machine images booting up for the first time. What /usr/local/etc/rc.d/ec2_configinit does is relatively simple: if it finds a tar archive, it extracts the files, and runs on them all in turn (not sure if there’s a way to enforce order of execution); if it finds a file with a correct header line, it runs on that file alone. When running, configinit either places the file on the file system (where the header is something like
>>/tmp/foo), or given a shebang, it will execute the file as a shell script.
Where does the trouble begin? When starting up an ec2 image with Ansible’s ec2 module, the parameter for passing data to configinit,
user_data: interprets the value as a string, only. Hence files have to be read in (easy enough to do with the lookup() function), but then it attempts to decode and reencode the string as utf8. Which is fine for the content of single files, but completely borks the issue of tar files to easily pass in config files on demand. But in any case, as I mentioned, the order of the file execution isn’t clearly defined, so it’s unclear how much help it would be anyway.
Anyway, moving on. Following configinit’s execution, another group of scripts runs, named
firstboot_*. One of the more useful ones is firstboot-pkgs, which attempts to install all packages listed in the rc.conf variable
firstboot_pkgs_list. What’s important to note here is that thanks to the lack of enforced ordering in BSD init, it’s possible for configinit to run after the listed packages are installed, but in practice this doesn’t actually happen. Essentially, if you want to have configinit do all the work, you’ll need it to run
pkg bootstrap on its own and install anything it needs at the time; moreover, if you want to edit the config files of packages rather than replacing them, you’ll need to do that here, or use an entirely different method.
And here the rub: as a final step, ec2_configinit reloads rc.conf before moving on with its execution. Which makes sense, as a primary use is to control the startup of the image you’re launching, but makes it effectively impossible to rely on the firstboot scripts and use configinit to return the image to readiness for creating a new, configurable image off of.
Because —as was obvious, once I read the damn thing— configinit executes, touches /firstboot (which still exists at this point), and then the firstboot scripts fire, eating the /firstboot trigger file.
So much for my grand schemes of using configinit for everything. In the end though, I was happily enough able to go back to Ansible at this point and devise a solution: the majority of the configuration could be done on the newly launched “imagebuilder” instance, by using add_hosts to set up a second play against the launched instances. The only caveat is that the “cleanup” stage of things removes all user access and unwanted packages —including python and sudo, rendering Ansible useless after that. So it must be left as the last step, and as a full execution, which I do via the simple expedient of templating a cleanup script out to the instance and executing it.
What this leaves me with, though, is a simple and easy-to-use procedure for prototyping AMIs that I can distribute as I like, on the basis of any FreeBSD AMI, and build my way step-by-step towards provisioning useful, configurable pre-configured images for work or for qbsd.
Now if only I remember to read before I try when I start on my next project....