diff --git a/Payload_img_design.md b/Payload_img_design.md new file mode 100644 index 00000000..d998ff2b --- /dev/null +++ b/Payload_img_design.md @@ -0,0 +1,108 @@ +# live-bootstrap + +This repository uses [`README.rst`](./README.rst) as the canonical main documentation. + +## Kernel-bootstrap raw `external.img` + +`external.img` is a raw container disk used in kernel-bootstrap mode when +`--external-sources` is set and `--repo` is unset. + +### Why not put everything in the initial image? + +In kernel-bootstrap mode, the first boot image is consumed by very early +runtime code before the system reaches the normal bash-based build stage. +That early stage has tight assumptions about memory layout and file table usage. + +When too many distfiles are packed into the initial image, those assumptions can +be exceeded, which leads to unstable handoff behavior (for example, failures +around the Fiwix transition in QEMU or on bare metal). + +So the design is intentionally split: + +- Initial image: only what is required to reach `improve: import_payload` +- `external.img`: the rest of distfiles + +This is not a patch-style workaround. It is a two-phase transport design that +keeps early boot deterministic and moves bulk data import to a stage where the +runtime is robust enough to process it safely. + +### Why import from an external image and copy into main filesystem? + +Because the bootstrap still expects distfiles to end up under the normal local +path (`/external/distfiles`) for later steps. `external.img` is used as a +transport medium only. + +The flow is: + +1. Boot minimal initial image. +2. Reach `improve: import_payload`. +3. Detect the external container disk by magic (`LBPAYLD1`) across detected block devices. +4. Copy payload files into `/external/distfiles`. +5. Continue the build exactly as if files had been present locally all along. + +### Format + +- Magic: `LBPAYLD1` (8 bytes) +- Then: little-endian `u64` file count +- Repeated entries: + - little-endian `u64` name length + - little-endian `u64` file size + - file name string, encoded as UTF-8 bytes (no terminator) + - file bytes + +`name length` is the number of bytes in the UTF-8 encoded file name (not the number of Unicode code points). + +The importer probes detected block devices and selects the one with magic `LBPAYLD1`. + +### Manual creation without Python + +Prepare `external.list` as: + +```text + +``` + +Then: + +```sh +cat > make-payload.sh <<'SH' +#!/bin/sh +set -e +out="${1:-external.img}" +list="${2:-external.list}" + +write_u64le() { + v="$1" + printf '%016x' "$v" | sed -E 's/(..)(..)(..)(..)(..)(..)(..)(..)/\8\7\6\5\4\3\2\1/' | xxd -r -p +} + +count="$(wc -l < "${list}" | tr -d ' ')" +: > "${out}" +printf 'LBPAYLD1' >> "${out}" +write_u64le "${count}" >> "${out}" + +while read -r name path; do + [ -n "${name}" ] || continue + size="$(wc -c < "${path}" | tr -d ' ')" + name_len="$(printf '%s' "${name}" | wc -c | tr -d ' ')" + write_u64le "${name_len}" >> "${out}" + write_u64le "${size}" >> "${out}" + printf '%s' "${name}" >> "${out}" + cat "${path}" >> "${out}" +done < "${list}" +SH +chmod +x make-payload.sh +./make-payload.sh external.img external.list +``` + +Attach `external.img` as an extra raw disk in QEMU, or as the second disk on bare metal. + +### When it is used + +- Used in kernel-bootstrap with `--external-sources` and without `--repo`. +- Not used with `--repo` (that path still uses an ext filesystem disk). +- Without `--external-sources` and without `--repo`, there is no second disk: + the initial image only includes distfiles needed before `improve: get_network`, + and later distfiles are downloaded from mirrors. +- `--extra-builds=guix` increases container contents (includes post-early `steps-guix` + sources), but does not change the mechanism. diff --git a/README.rst b/README.rst index 154c860c..b2379a23 100644 --- a/README.rst +++ b/README.rst @@ -63,17 +63,86 @@ Without using Python: * *Only* copy distfiles listed in ``sources`` files for ``build:`` steps manifested before ``improve: get_network`` into this disk. - * Optionally (if you don't do this, distfiles will be network downloaded): + * In kernel-bootstrap mode with ``--external-sources`` (and no ``--repo``), + use the second image as ``external.img``. + ``external.img`` is a raw container (not a filesystem) used to carry the + distfiles that are not needed before ``improve: import_payload``. + In other words, the first image only carries the minimal set needed to + reach the importer; the rest of the distfiles live in ``external.img``. - * On the second image, create an MSDOS partition table and one ext3 - partition. - * Copy ``distfiles/`` into this disk. - * Run QEMU, with 4+G RAM, optionally SMP (multicore), both drives (in the - order introduced above), a NIC with model E1000 + * Header magic: ``LBPAYLD1`` (8 bytes). + * Then: little-endian ``u64`` file count. + * Repeated for each file: little-endian ``u64`` name length, + little-endian ``u64`` file size, UTF-8 encoded file name bytes + (no terminator), raw file bytes. + * ``name length`` is the number of UTF-8 bytes (not Unicode code points). + + * With ``--repo``, the second disk remains an ext3 distfiles/repo disk. + * Without ``--external-sources`` and without ``--repo``, no second disk is + used: the initial image includes only pre-network distfiles, and later + distfiles are downloaded from configured mirrors after networking starts. + * Run QEMU, with 4+G RAM, optionally SMP (multicore), both drives (main + builder image plus external image, when a second image is used), a NIC with model E1000 (``-nic user,model=e1000``), and ``-machine kernel-irqchip=split``. c. **Bare metal:** Follow the same steps as QEMU, but the disks need to be two different *physical* disks, and boot from the first disk. +Manual raw ``external.img`` preparation +--------------------------------------- + +The following script creates a raw ``external.img`` from a manually prepared +file list. This is equivalent to what ``rootfs.py`` does for kernel-bootstrap +with ``--external-sources`` (and no ``--repo``). + +1. Prepare an ``external.list`` with one file per line, formatted as: + `` ``. +2. Run: + + :: + + cat > make-payload.sh <<'EOF' + #!/bin/sh + set -e + out="${1:-external.img}" + list="${2:-external.list}" + + write_u64le() { + v="$1" + printf '%016x' "$v" | sed -E 's/(..)(..)(..)(..)(..)(..)(..)(..)/\8\7\6\5\4\3\2\1/' | xxd -r -p + } + + count="$(wc -l < "${list}" | tr -d ' ')" + : > "${out}" + printf 'LBPAYLD1' >> "${out}" + write_u64le "${count}" >> "${out}" + + while read -r name path; do + [ -n "${name}" ] || continue + size="$(wc -c < "${path}" | tr -d ' ')" + name_len="$(printf '%s' "${name}" | wc -c | tr -d ' ')" + write_u64le "${name_len}" >> "${out}" + write_u64le "${size}" >> "${out}" + printf '%s' "${name}" >> "${out}" + cat "${path}" >> "${out}" + done < "${list}" + EOF + chmod +x make-payload.sh + ./make-payload.sh external.img external.list + +3. Attach ``external.img`` as an additional raw disk when booting in QEMU, or + as the second physical disk on bare metal. + +Notes: + +* ``external.img`` raw container mode is used with ``--external-sources`` (and + no ``--repo``). With ``--extra-builds=guix``, the container content is larger + because it also includes post-early sources from ``steps-guix``. +* Without ``--external-sources`` and without ``--repo``, there is no second + image. The initial image only includes distfiles needed before + ``improve: get_network``; later distfiles are downloaded from mirrors. +* The runtime importer identifies the correct disk by checking the magic + ``LBPAYLD1`` on each detected block device, not by assuming a device name. + Mirrors ------- diff --git a/lib/generator.py b/lib/generator.py index 9b1f8bce..187a1790 100755 --- a/lib/generator.py +++ b/lib/generator.py @@ -12,6 +12,7 @@ import hashlib import os import random import shutil +import struct import tarfile import traceback @@ -25,16 +26,34 @@ class Generator(): git_dir = os.path.join(os.path.dirname(os.path.join(__file__)), '..') distfiles_dir = os.path.join(git_dir, 'distfiles') + raw_container_magic = b'LBPAYLD1' # pylint: disable=too-many-arguments,too-many-positional-arguments - def __init__(self, arch, external_sources, early_preseed, repo_path, mirrors): + def __init__(self, arch, external_sources, early_preseed, repo_path, mirrors, + extra_builds=None): self.arch = arch self.early_preseed = early_preseed self.external_sources = external_sources self.repo_path = repo_path self.mirrors = mirrors - self.source_manifest = self.get_source_manifest(not self.external_sources) - self.early_source_manifest = self.get_source_manifest(True) + self.extra_builds = list(extra_builds or []) + self.pre_network_source_manifest = self.get_source_manifest( + stop_before_improve="get_network", + extra_builds=[], + ) + self.pre_import_source_manifest = self.get_source_manifest( + stop_before_improve="import_payload", + extra_builds=[], + ) + # Only raw-external mode needs full upfront availability for container generation. + if self.external_sources and not self.repo_path: + self.source_manifest = self.get_source_manifest(extra_builds=self.extra_builds) + else: + self.source_manifest = self.pre_network_source_manifest + self.bootstrap_source_manifest = self.source_manifest + self.external_source_manifest = [] + self.external_image = None + self.kernel_bootstrap_mode = None self.target_dir = None self.external_dir = None @@ -46,6 +65,115 @@ class Generator(): self.external_dir = os.path.join(self.target_dir, 'external') self.distfiles() + def _select_kernel_bootstrap_mode(self): + """ + Select how kernel-bootstrap should transport distfiles. + """ + if self.repo_path: + # Keep second-disk staging outside init image for ext3 repo mode. + self.external_dir = os.path.join(os.path.dirname(self.target_dir), 'external') + self.kernel_bootstrap_mode = "repo" + self.external_source_manifest = [] + return + + if self.external_sources: + # Raw external container mode keeps early distfiles inside init image. + self.external_dir = os.path.join(self.target_dir, 'external') + self.kernel_bootstrap_mode = "raw_external" + self._prepare_kernel_bootstrap_external_manifests() + return + + # Network-only mode keeps pre-network distfiles inside init image. + self.external_dir = os.path.join(self.target_dir, 'external') + self.kernel_bootstrap_mode = "network_only" + self.bootstrap_source_manifest = self.pre_network_source_manifest + self.external_source_manifest = [] + + def _prepare_kernel_bootstrap_external_manifests(self): + """ + Split distfiles between init image and external raw container. + """ + # Keep the early builder image small: include only sources needed + # before improve: import_payload runs, so external.img is the primary + # carrier for the remaining distfiles. + self.bootstrap_source_manifest = self.pre_import_source_manifest + + full_manifest = self.get_source_manifest(extra_builds=self.extra_builds) + if self.bootstrap_source_manifest == full_manifest: + raise ValueError("steps/manifest must include `improve: import_payload` in kernel-bootstrap mode.") + bootstrap_set = set(self.bootstrap_source_manifest) + self.external_source_manifest = [entry for entry in full_manifest if entry not in bootstrap_set] + + def _kernel_bootstrap_init_manifest(self): + """ + Return the exact manifest that is allowed inside init image. + """ + mode_to_manifest = { + "network_only": self.pre_network_source_manifest, # up to get_network + "raw_external": self.bootstrap_source_manifest, # up to import_payload + "repo": self.pre_network_source_manifest, # up to get_network + } + manifest = mode_to_manifest.get(self.kernel_bootstrap_mode) + if manifest is None: + raise ValueError(f"Unexpected kernel bootstrap mode: {self.kernel_bootstrap_mode}") + return manifest + + def _copy_manifest_distfiles(self, out_dir, manifest): + os.makedirs(out_dir, exist_ok=True) + for entry in manifest: + file_name = entry[3].strip() + shutil.copy2(os.path.join(self.distfiles_dir, file_name), + os.path.join(out_dir, file_name)) + + def _ensure_manifest_distfiles(self, manifest): + for entry in manifest: + checksum, directory, url, file_name = entry + distfile_path = os.path.join(directory, file_name) + if not os.path.isfile(distfile_path): + self.download_file(url, directory, file_name) + self.check_file(distfile_path, checksum) + + def _create_raw_container_image(self, target_path, manifest, image_name="external.img"): + if manifest is None: + manifest = [] + + if manifest: + # Guarantee all payload distfiles exist and match checksums. + self._ensure_manifest_distfiles(manifest) + + files_by_name = {} + for checksum, _, _, file_name in manifest: + if file_name in files_by_name and files_by_name[file_name] != checksum: + raise ValueError( + f"Conflicting container file with same name but different hash: {file_name}" + ) + files_by_name[file_name] = checksum + + container_path = os.path.join(target_path, image_name) + ordered_names = sorted(files_by_name.keys()) + with open(container_path, "wb") as container: + container.write(self.raw_container_magic) + if len(ordered_names) > 0xFFFFFFFFFFFFFFFF: + raise ValueError("Too many files for raw container format.") + container.write(struct.pack(" 0xFFFFFFFFFFFFFFFF: + raise ValueError(f"Container file name too long: {file_name}") + + src_path = os.path.join(self.distfiles_dir, file_name) + file_size = os.path.getsize(src_path) + if file_size > 0xFFFFFFFFFFFFFFFF: + raise ValueError(f"Container file too large for raw container format: {file_name}") + + container.write(struct.pack(" 3: - file_name = source[3] - else: - # Automatically determine file name based on URL. - file_name = os.path.basename(source[1]) + step = line.split(" ")[1].split("#")[0].strip() + sourcef = os.path.join(steps_dir, step, "sources") + if os.path.exists(sourcef): + # Read sources from the source file + with open(sourcef, "r", encoding="utf_8") as sources: + for source in sources.readlines(): + source = source.strip().split(" ") - entry = (source[2], directory, source[1], file_name) - if entry not in entries: - entries.append(entry) + if source[0] == "g" or source[0] == "git": + source[1:] = source[2:] + + if len(source) > 3: + file_name = source[3] + else: + # Automatically determine file name based on URL. + file_name = os.path.basename(source[1]) + + entry = (source[2], directory, source[1], file_name) + if entry not in entries: + entries.append(entry) return entries diff --git a/lib/simple_mirror.py b/lib/simple_mirror.py index 1c9f39d4..a0199282 100644 --- a/lib/simple_mirror.py +++ b/lib/simple_mirror.py @@ -12,7 +12,7 @@ class SimpleMirror(socketserver.TCPServer): """Simple HTTP mirror from a directory""" def __init__(self, directory: str): self.directory = directory - super().__init__(("localhost", 0), self._handler) + super().__init__(("0.0.0.0", 0), self._handler) @property def port(self): diff --git a/lib/target.py b/lib/target.py index 50868fc6..4e4c04ab 100644 --- a/lib/target.py +++ b/lib/target.py @@ -59,3 +59,7 @@ class Target: def get_disk(self, name): """Get the path to a device of a disk""" return self._disks.get(name) + + def add_existing_disk(self, name, path): + """Register an existing disk image path.""" + self._disks[name] = os.path.abspath(path) diff --git a/mirror.sh b/mirror.sh index 00d0d6d3..cb7c5c9f 100755 --- a/mirror.sh +++ b/mirror.sh @@ -186,7 +186,8 @@ do_file() { esac } -for src in steps/*/sources; do +for src in steps/*/sources steps-*/*/sources; do + [ -f "${src}" ] || continue while read -r line; do # shellcheck disable=SC2086 do_file ${line} diff --git a/rootfs.py b/rootfs.py index 7cbb4bc9..c36b23b6 100755 --- a/rootfs.py +++ b/rootfs.py @@ -17,6 +17,8 @@ you can run bootstap inside chroot. import argparse import os import signal +import shutil +import tempfile import threading from lib.generator import Generator, stage0_arch_map @@ -24,6 +26,503 @@ from lib.simple_mirror import SimpleMirror from lib.target import Target from lib.utils import run, run_as_root +def parse_internal_ci_break_after(value): + """ + Parse INTERNAL_CI break directive in the form ':'. + """ + if not value: + return None, None + scope, separator, step_name = value.partition(":") + scope = scope.strip() + step_name = step_name.strip() + if ( + separator != ":" + or not _is_valid_manifest_scope(scope) + or not step_name + ): + raise ValueError( + "--internal-ci-break-after must be in the form " + "'steps:' or 'steps-:'." + ) + return scope, step_name + + +def _is_valid_extra_build_name(name): + if not name: + return False + return all(ch.isalnum() or ch in ("-", "_") for ch in name) + + +def parse_extra_builds(value): + """ + Parse comma-separated extra build namespaces (e.g. 'guix,gentoo'). + """ + if not value: + return [] + + builds = [] + for raw_name in value.split(","): + name = raw_name.strip() + if not name: + continue + if not _is_valid_extra_build_name(name): + raise ValueError( + "--extra-builds must be a comma-separated list of names using " + "letters, digits, '-' or '_'." + ) + if name not in builds: + builds.append(name) + return builds + + +def _scope_for_extra_build(extra_build): + return f"steps-{extra_build}" + + +def _is_valid_manifest_scope(scope): + if scope == "steps": + return True + if not scope.startswith("steps-"): + return False + return _is_valid_extra_build_name(scope[len("steps-"):]) + + +def _parse_build_step_name(line): + """ + Extract the package name from a manifest build directive. + """ + stripped = line.strip() + if not stripped.startswith("build: "): + return None + payload = stripped[len("build: "):].split("#", 1)[0].strip() + if not payload: + return None + return payload.split()[0] + +_INIT_MOUNT_MARKER = "# LB_STAGE0_EARLY_MOUNTS" +_INIT_REGEN_MARKER = "# LB_STAGE0_REGENERATE_SCRIPTS" +_RESUME_NEXT_PATH = "/steps/.lb-resume-next" +_RESUME_NEXT_SCOPE_VAR = "LB_RESUME_NEXT_SCOPE" +_RESUME_NEXT_PACKAGE_VAR = "LB_RESUME_NEXT_PACKAGE" +_INIT_LEGACY_NETWORK_BLOCK = ( + 'if [ "${CHROOT}" = False ] && command -v dhcpcd >/dev/null 2>&1; then\n' + 'dhcpcd --waitip=4 || true\n' + 'fi\n' +) +_INIT_GATED_NETWORK_BLOCK = ( + 'if [ -f /steps/env ]; then\n' + '. /steps/env\n' + 'fi\n' + 'if [ "${CHROOT}" = False ] && [ "${NETWORK_READY}" = True ] && command -v dhcpcd >/dev/null 2>&1; then\n' + 'dhcpcd --waitip=4 || true\n' + 'fi\n' +) +_INIT_MOUNT_BLOCK = ( + _INIT_MOUNT_MARKER + "\n" + + "mount | grep ' on /dev ' >/dev/null 2>&1 || (mkdir -p /dev; mount -t devtmpfs devtmpfs /dev)\n" + + "mount | grep ' on /proc ' >/dev/null 2>&1 || (mkdir -p /proc; mount -t proc proc /proc)\n" +) +_INIT_REGEN_BLOCK = ( + _INIT_REGEN_MARKER + "\n" + + "# Regenerate scripts from current manifests so resume runs stay deterministic.\n" + + 'resume_entry=""\n' + + 'resume_root=""\n' + + 'resume_pkg=""\n' + + 'resume_next_scope=""\n' + + 'resume_next_pkg=""\n' + + 'if [ -f "$0" ]; then\n' + + 'resume_entry="$(sed -n "s/.*bash \\(\\/steps[^ ]*\\/[0-9][0-9]*\\.sh\\).*/\\1/p" "$0" | head -n1)"\n' + + 'fi\n' + + 'if [ -n "${resume_entry}" ]; then\n' + + 'resume_root="$(dirname "${resume_entry}")"\n' + + 'if [ -f "${resume_entry}" ]; then\n' + + 'resume_pkg="$(sed -n "s/^build \\([^ ]*\\) .*/\\1/p" "${resume_entry}" | head -n1)"\n' + + 'fi\n' + + 'fi\n' + + f'if [ -f "{_RESUME_NEXT_PATH}" ]; then\n' + + f'resume_next_scope="$(sed -n "s/^scope=//p" "{_RESUME_NEXT_PATH}" | head -n1)"\n' + + f'resume_next_pkg="$(sed -n "s/^package=//p" "{_RESUME_NEXT_PATH}" | head -n1)"\n' + + 'if [ "${resume_next_scope}" = "steps" ]; then\n' + + 'resume_root="/steps"\n' + + 'elif echo "${resume_next_scope}" | grep -Eq "^steps-[A-Za-z0-9_-]+$"; then\n' + + 'resume_root="/${resume_next_scope}"\n' + + 'fi\n' + + 'if [ -n "${resume_next_pkg}" ]; then\n' + + 'resume_pkg="${resume_next_pkg}"\n' + + 'fi\n' + + f'rm -f "{_RESUME_NEXT_PATH}"\n' + + 'fi\n' + + 'if [ -x /script-generator ] && [ -f /steps/manifest ]; then\n' + + '/script-generator /steps/manifest\n' + + 'fi\n' + + 'for extra_manifest in /steps-*/manifest; do\n' + + 'if [ ! -f "${extra_manifest}" ]; then\n' + + 'continue\n' + + 'fi\n' + + 'if [ -x /script-generator ]; then\n' + + '/script-generator "${extra_manifest}" /steps\n' + + 'fi\n' + + 'done\n' + + 'if [ -n "${resume_pkg}" ] && [ -d "${resume_root}" ]; then\n' + + 'mapped_entry="$(grep -F -l "build ${resume_pkg} " "${resume_root}"/[0-9]*.sh 2>/dev/null | head -n1 || true)"\n' + + 'if [ -n "${mapped_entry}" ]; then\n' + + 'resume_entry="${mapped_entry}"\n' + + 'fi\n' + + 'fi\n' + + 'if [ -n "${resume_entry}" ] && [ -f "${resume_entry}" ] && [ "${resume_entry}" != "$0" ]; then\n' + + 'bash "${resume_entry}"\n' + + 'exit "$?"\n' + + 'fi\n' +) + + +def _find_next_build_after(manifest_lines, break_step, context_label): + """ + Return the next build step after break_step within the same manifest. + """ + found_break = False + for line in manifest_lines: + step_name = _parse_build_step_name(line) + if step_name is None: + continue + if found_break: + return step_name + if step_name == break_step: + found_break = True + + if not found_break: + raise ValueError( + "INTERNAL_CI_BREAK_AFTER target not found in " + + context_label + + ": " + + break_step + ) + return None + + +def _insert_internal_ci_break(manifest_lines, break_step, internal_ci, context_label): + inserted = False + break_line = f"jump: break ( INTERNAL_CI == {internal_ci} )\n" + output_lines = [] + for line in manifest_lines: + output_lines.append(line) + if inserted: + continue + + step_name = _parse_build_step_name(line) + if step_name is None: + continue + if step_name == break_step: + output_lines.append(break_line) + inserted = True + + if not inserted: + raise ValueError( + "INTERNAL_CI_BREAK_AFTER target not found in " + + context_label + + ": " + + break_step + ) + return output_lines + + +def _copytree_replace(src, dst): + if os.path.isdir(dst): + shutil.copytree(src, dst, dirs_exist_ok=True) + else: + shutil.copytree(src, dst) + + +def _inject_block_after_shebang(content, marker, block): + if marker in content: + return content + first_newline = content.find("\n") + if first_newline == -1: + return content + return content[: first_newline + 1] + block + content[first_newline + 1 :] + + +def _patch_resume_init_scripts(mountpoint): + for init_name in os.listdir(mountpoint): + if init_name != "init" and not init_name.startswith("init."): + continue + init_path = os.path.join(mountpoint, init_name) + if not os.path.isfile(init_path): + continue + + with open(init_path, "r", encoding="utf-8", errors="ignore") as init_file: + init_content = init_file.read() + + init_content = _inject_block_after_shebang( + init_content, + _INIT_MOUNT_MARKER, + _INIT_MOUNT_BLOCK, + ) + if ( + _INIT_LEGACY_NETWORK_BLOCK in init_content + and _INIT_GATED_NETWORK_BLOCK not in init_content + ): + init_content = init_content.replace( + _INIT_LEGACY_NETWORK_BLOCK, + _INIT_GATED_NETWORK_BLOCK, + ) + init_content = _inject_block_after_shebang( + init_content, + _INIT_REGEN_MARKER, + _INIT_REGEN_BLOCK, + ) + + with open(init_path, "w", encoding="utf-8") as init_file: + init_file.write(init_content) + + +def _update_stage0_tree(mountpoint, + steps_dir, + extra_builds, + mirrors, + internal_ci, + break_scope, + break_step): + """ + Apply all resume updates directly to a mounted stage0 filesystem tree. + """ + old_config_path = os.path.join(mountpoint, "steps", "bootstrap.cfg") + if not os.path.isfile(old_config_path): + raise SystemExit(f"Missing config in stage0 image: {old_config_path}") + old_env_path = os.path.join(mountpoint, "steps", "env") + old_env_content = None + break_output_lines = None + break_manifest_relpath = None + next_step = None + if os.path.isfile(old_env_path): + with open(old_env_path, "rb") as env_file: + old_env_content = env_file.read() + + with open(old_config_path, "r", encoding="utf-8") as cfg: + lines = [ + line for line in cfg + if not line.startswith("EXTRA_BUILDS=") + and not line.startswith("INTERNAL_CI=") + and not line.startswith(_RESUME_NEXT_SCOPE_VAR + "=") + and not line.startswith(_RESUME_NEXT_PACKAGE_VAR + "=") + and not line.startswith("MIRRORS=") + and not line.startswith("MIRRORS_LEN=") + and not line.startswith("PAYLOAD_REQUIRED=") + ] + + dest_steps = os.path.join(mountpoint, "steps") + _copytree_replace(steps_dir, dest_steps) + + env_path = os.path.join(dest_steps, "env") + if old_env_content is not None: + with open(env_path, "wb") as env_file: + env_file.write(old_env_content) + + if os.path.isfile(env_path): + with open(env_path, "r", encoding="utf-8", errors="ignore") as env_file: + env_content = env_file.read() + else: + env_content = "" + if "NETWORK_READY=" not in env_content: + with open(env_path, "a", encoding="utf-8") as env_file: + if env_content and not env_content.endswith("\n"): + env_file.write("\n") + env_file.write("NETWORK_READY=False\n") + + if break_scope and break_step: + if internal_ci in ("", "False", None): + raise SystemExit("INTERNAL_CI must be set when INTERNAL_CI_BREAK_AFTER is used.") + if break_scope == "steps": + source_manifest_path = os.path.join(steps_dir, "manifest") + else: + allowed_scopes = {_scope_for_extra_build(extra) for extra in extra_builds} + if break_scope not in allowed_scopes: + raise SystemExit( + f"INTERNAL_CI_BREAK_AFTER scope '{break_scope}' is not enabled by EXTRA_BUILDS." + ) + source_manifest_path = os.path.join( + os.path.dirname(steps_dir), + break_scope, + "manifest", + ) + if not os.path.isfile(source_manifest_path): + raise SystemExit(f"Missing manifest for INTERNAL_CI_BREAK_AFTER: {source_manifest_path}") + with open(source_manifest_path, "r", encoding="utf-8") as manifest_file: + manifest_lines = manifest_file.readlines() + next_step = _find_next_build_after( + manifest_lines, + break_step, + f"{break_scope}/manifest", + ) + break_output_lines = _insert_internal_ci_break( + manifest_lines, + break_step, + internal_ci, + f"{break_scope}/manifest", + ) + break_manifest_relpath = os.path.join(break_scope, "manifest") + if next_step: + lines.append(f"{_RESUME_NEXT_SCOPE_VAR}={break_scope}\n") + lines.append(f"{_RESUME_NEXT_PACKAGE_VAR}={next_step}\n") + + config_path = os.path.join(dest_steps, "bootstrap.cfg") + if extra_builds: + lines.append("EXTRA_BUILDS=" + ",".join(extra_builds) + "\n") + if mirrors: + lines.append(f'MIRRORS="{" ".join(mirrors)}"\n') + lines.append(f"MIRRORS_LEN={len(mirrors)}\n") + lines.append(f"INTERNAL_CI={internal_ci}\n") + lines.append("PAYLOAD_REQUIRED=False\n") + with open(config_path, "w", encoding="utf-8") as cfg: + cfg.writelines(lines) + + _patch_resume_init_scripts(mountpoint) + + for extra_build in extra_builds: + extra_scope = _scope_for_extra_build(extra_build) + source_steps_extra = os.path.join(os.path.dirname(steps_dir), extra_scope) + dest_steps_extra = os.path.join(mountpoint, extra_scope) + _copytree_replace(source_steps_extra, dest_steps_extra) + + if break_output_lines is not None and break_manifest_relpath is not None: + manifest_path = os.path.join(mountpoint, break_manifest_relpath) + if not os.path.isfile(manifest_path): + raise SystemExit(f"Missing manifest for INTERNAL_CI_BREAK_AFTER: {manifest_path}") + with open(manifest_path, "w", encoding="utf-8") as manifest_file: + manifest_file.writelines(break_output_lines) + + +def _stage0_update_cli(argv): + """ + Internal entrypoint executed as root for mutating mounted stage0 trees. + """ + if len(argv) < 6: + raise SystemExit("stage0 update cli expects at least 6 arguments") + mountpoint = argv[0] + steps_dir = argv[1] + extra_builds = parse_extra_builds(argv[2]) + internal_ci = argv[3] + break_scope = argv[4] + break_step = argv[5] + mirrors = argv[6:] + _update_stage0_tree( + mountpoint, + steps_dir, + extra_builds, + mirrors, + internal_ci, + break_scope, + break_step, + ) + + +def apply_internal_ci_break_to_tree(tree_root, break_scope, break_step, internal_ci): + """ + Inject INTERNAL_CI break jump after a build step in a prepared steps tree. + """ + if not break_scope or not break_step: + return + if internal_ci in ("", "False", None): + raise ValueError("INTERNAL_CI must be set when INTERNAL_CI_BREAK_AFTER is used.") + + manifest_path = os.path.join(tree_root, break_scope, "manifest") + if not os.path.isfile(manifest_path): + raise ValueError(f"Missing manifest for INTERNAL_CI_BREAK_AFTER: {manifest_path}") + + with open(manifest_path, "r", encoding="utf-8") as manifest_file: + manifest_lines = manifest_file.readlines() + output_lines = _insert_internal_ci_break( + manifest_lines, + break_step, + internal_ci, + f"{break_scope}/manifest", + ) + with open(manifest_path, "w", encoding="utf-8") as manifest_file: + manifest_file.writelines(output_lines) + +def update_stage0_image(image_path, + extra_builds=None, + mirrors=None, + internal_ci=False, + internal_ci_break_after=None): + """ + Update an existing stage0 image by refreshing step sources from the working + tree and patching bootstrap config / optional extra build handoff bits. + """ + if extra_builds is None: + extra_builds = [] + if mirrors is None: + mirrors = [] + + steps_dir = os.path.abspath("steps") + steps_manifest = os.path.join(steps_dir, "manifest") + if not os.path.isdir(steps_dir) or not os.path.isfile(steps_manifest): + raise ValueError("steps/manifest does not exist.") + + for extra_build in extra_builds: + extra_scope = _scope_for_extra_build(extra_build) + extra_steps_dir = os.path.abspath(extra_scope) + manifest = os.path.join(extra_steps_dir, "manifest") + if not os.path.isdir(extra_steps_dir) or not os.path.isfile(manifest): + raise ValueError( + f"{extra_scope}/manifest does not exist while --extra-builds includes {extra_build}." + ) + break_scope, break_step = parse_internal_ci_break_after(internal_ci_break_after) + + mountpoint = tempfile.mkdtemp(prefix="lb-stage0-", dir="/tmp") + mounted = False + try: + run_as_root( + "mount", + "-t", "ext4", + "-o", "loop,offset=1073741824", + image_path, + mountpoint, + ) + mounted = True + loader = ( + "import importlib.util, sys;" + "spec=importlib.util.spec_from_file_location('lb_rootfs', sys.argv[1]);" + "mod=importlib.util.module_from_spec(spec);" + "spec.loader.exec_module(mod);" + "mod._stage0_update_cli(sys.argv[2:])" + ) + run_as_root( + "python3", + "-c", + loader, + os.path.abspath(__file__), + mountpoint, + steps_dir, + ",".join(extra_builds), + str(internal_ci) if internal_ci else "False", + break_scope or "", + break_step or "", + *mirrors, + ) + finally: + if mounted: + run_as_root("umount", mountpoint) + os.rmdir(mountpoint) + +def prepare_stage0_work_image(base_image, + output_dir, + extra_builds, + mirrors=None, + internal_ci=False, + internal_ci_break_after=None): + """ + Copy stage0 base image to a disposable work image and refresh steps/config. + """ + work_image = os.path.join(output_dir, "stage0-work.img") + shutil.copy2(base_image, work_image) + update_stage0_image(work_image, + extra_builds=extra_builds, + mirrors=mirrors, + internal_ci=internal_ci, + internal_ci_break_after=internal_ci_break_after) + return work_image + def create_configuration_file(args): """ Creates bootstrap.cfg file which would contain options used to @@ -31,6 +530,8 @@ def create_configuration_file(args): """ config_path = os.path.join('steps', 'bootstrap.cfg') with open(config_path, "w", encoding="utf_8") as config: + kernel_bootstrap = ((args.bare_metal or args.qemu) and not args.kernel) + payload_required = kernel_bootstrap and args.external_sources and not args.repo config.write(f"ARCH={args.arch}\n") config.write(f"ARCH_DIR={stage0_arch_map.get(args.arch, args.arch)}\n") config.write(f"FORCE_TIMESTAMPS={args.force_timestamps}\n") @@ -41,13 +542,13 @@ def create_configuration_file(args): config.write(f"FINAL_JOBS={args.cores}\n") config.write(f"INTERNAL_CI={args.internal_ci or False}\n") config.write(f"INTERACTIVE={args.interactive}\n") + config.write(f"PAYLOAD_REQUIRED={payload_required}\n") config.write(f"QEMU={args.qemu}\n") config.write(f"BARE_METAL={args.bare_metal or (args.qemu and args.interactive)}\n") - if (args.bare_metal or args.qemu) and not args.kernel: - if args.repo or args.external_sources: - config.write("DISK=sdb1\n") - else: - config.write("DISK=sda\n") + if args.extra_builds: + config.write("EXTRA_BUILDS=" + ",".join(args.extra_builds) + "\n") + if kernel_bootstrap: + config.write("DISK=sdb1\n" if args.repo else "DISK=sda\n") config.write("KERNEL_BOOTSTRAP=True\n") else: config.write("DISK=sda1\n") @@ -95,6 +596,9 @@ def main(): parser.add_argument("--build-kernels", help="Also build kernels in chroot and bwrap builds", action="store_true") + parser.add_argument("--extra-builds", + help="Comma-separated extra build namespaces to run after main steps " + "(e.g. guix or guix,gentoo,azl3).") parser.add_argument("--no-create-config", help="Do not automatically create config file", action="store_true") @@ -112,6 +616,10 @@ def main(): parser.add_argument("--early-preseed", help="Skip early stages of live-bootstrap", nargs=None) parser.add_argument("--internal-ci", help="INTERNAL for github CI") + parser.add_argument("--internal-ci-break-after", + help="Insert a temporary jump: break after a build step using " + "'steps:' or 'steps-:' " + "for --stage0-image resume runs and fresh --qemu kernel-bootstrap runs.") parser.add_argument("-s", "--swap", help="Swap space to allocate in Linux", default=0) @@ -125,11 +633,14 @@ def main(): parser.add_argument("-qs", "--target-size", help="Size of the target image (for QEMU only)", default="16G") parser.add_argument("-qk", "--kernel", help="Custom early kernel to use") + parser.add_argument("--stage0-image", + help="Boot an existing stage0 image (target/init.img) directly in QEMU") parser.add_argument("-b", "--bare-metal", help="Build images for bare metal", action="store_true") args = parser.parse_args() + args.extra_builds = parse_extra_builds(args.extra_builds) # Mode validation def check_types(): @@ -179,9 +690,34 @@ def main(): args.swap = 0 # Validate mirrors - if not args.mirrors: + if not args.mirrors and not args.stage0_image: raise ValueError("At least one mirror must be provided.") + if args.internal_ci_break_after: + if not args.internal_ci: + raise ValueError("--internal-ci-break-after requires --internal-ci (e.g. pass2).") + break_scope, _ = parse_internal_ci_break_after(args.internal_ci_break_after) + if break_scope != "steps": + extra_build = break_scope[len("steps-"):] + if extra_build not in args.extra_builds: + raise ValueError( + f"--internal-ci-break-after {break_scope}:* requires --extra-builds include " + f"{extra_build}." + ) + if not args.qemu: + raise ValueError("--internal-ci-break-after currently requires --qemu.") + if args.kernel: + raise ValueError("--internal-ci-break-after cannot be used with --kernel.") + + if args.stage0_image: + if not args.qemu: + raise ValueError("--stage0-image can only be used with --qemu.") + if args.kernel: + raise ValueError("--stage0-image cannot be combined with --kernel.") + if not os.path.isfile(args.stage0_image): + raise ValueError(f"Stage0 image does not exist: {args.stage0_image}") + args.stage0_image = os.path.abspath(args.stage0_image) + # Set constant umask os.umask(0o022) @@ -199,15 +735,16 @@ def main(): mirror_servers.append(server) # bootstrap.cfg - try: - os.remove(os.path.join('steps', 'bootstrap.cfg')) - except FileNotFoundError: - pass - if not args.no_create_config: - create_configuration_file(args) - else: - with open(os.path.join('steps', 'bootstrap.cfg'), 'a', encoding='UTF-8'): + if not args.stage0_image: + try: + os.remove(os.path.join('steps', 'bootstrap.cfg')) + except FileNotFoundError: pass + if not args.no_create_config: + create_configuration_file(args) + else: + with open(os.path.join('steps', 'bootstrap.cfg'), 'a', encoding='UTF-8'): + pass # target target = Target(path=args.target) @@ -223,120 +760,185 @@ def main(): server.shutdown() signal.signal(signal.SIGINT, cleanup) - generator = Generator(arch=args.arch, - external_sources=args.external_sources, - repo_path=args.repo, - early_preseed=args.early_preseed, - mirrors=args.mirrors) + generator = None + if not args.stage0_image: + generator = Generator(arch=args.arch, + external_sources=args.external_sources, + repo_path=args.repo, + early_preseed=args.early_preseed, + mirrors=args.mirrors, + extra_builds=args.extra_builds) bootstrap(args, generator, target, args.target_size, cleanup) cleanup() +def _bootstrap_chroot(args, generator, target, cleanup): + find_chroot = """ +import shutil +print(shutil.which('chroot')) +""" + chroot_binary = run_as_root( + 'python3', + '-c', + find_chroot, + capture_output=True, + ).stdout.decode().strip() + + generator.prepare(target, using_kernel=False) + arch = stage0_arch_map.get(args.arch, args.arch) + init = os.path.join(os.sep, 'bootstrap-seeds', 'POSIX', arch, 'kaem-optional-seed') + run_as_root('env', '-i', 'PATH=/bin', chroot_binary, generator.target_dir, init, cleanup=cleanup) + + +def _bootstrap_bwrap(args, generator, target, cleanup): + init = '/init' + if not args.internal_ci or args.internal_ci == "pass1": + generator.prepare(target, using_kernel=False) + arch = stage0_arch_map.get(args.arch, args.arch) + init = os.path.join(os.sep, 'bootstrap-seeds', 'POSIX', arch, 'kaem-optional-seed') + else: + generator.reuse(target) + + run('env', '-i', 'bwrap', '--unshare-user', + '--uid', '0', + '--gid', '0', + '--unshare-net' if args.external_sources else None, + '--setenv', 'PATH', '/usr/bin', + '--bind', generator.target_dir, '/', + '--dir', '/dev', + '--dev-bind', '/dev/null', '/dev/null', + '--dev-bind', '/dev/zero', '/dev/zero', + '--dev-bind', '/dev/random', '/dev/random', + '--dev-bind', '/dev/urandom', '/dev/urandom', + '--dev-bind', '/dev/ptmx', '/dev/ptmx', + '--dev-bind', '/dev/tty', '/dev/tty', + '--tmpfs', '/dev/shm', + '--proc', '/proc', + '--bind', '/sys', '/sys', + '--tmpfs', '/tmp', + init, + cleanup=cleanup) + + +def _bootstrap_bare_metal(args, generator, target, size): + if args.kernel: + generator.prepare(target, using_kernel=True, target_size=size) + path = os.path.join(args.target, os.path.relpath(generator.target_dir, args.target)) + print("Please:") + print(f" 1. Take {path}/initramfs and your kernel, boot using this.") + print(f" 2. Take {path}/disk.img and put this on a writable storage medium.") + return + + generator.prepare(target, kernel_bootstrap=True, target_size=size) + path = os.path.join(args.target, os.path.relpath(generator.target_dir, args.target)) + print("Please:") + print(f" 1. Take {path}.img and write it to a boot drive and then boot it.") + external_disk = target.get_disk("external") + if external_disk is None: + return + external_path = os.path.join(args.target, os.path.relpath(external_disk, args.target)) + if args.repo: + print(" 2. Take " + f"{external_path} and attach it as a second disk (/dev/sdb preferred).") + else: + print(" 2. Take " + f"{external_path} and attach it as a second raw container disk (/dev/sdb preferred).") + + +def _qemu_arg_list_for_stage0_image(args, target): + work_image = prepare_stage0_work_image( + args.stage0_image, + target.path, + args.extra_builds, + mirrors=args.mirrors, + internal_ci=args.internal_ci, + internal_ci_break_after=args.internal_ci_break_after, + ) + return [ + '-enable-kvm', + '-m', str(args.qemu_ram) + 'M', + '-smp', str(args.cores), + '-drive', 'file=' + work_image + ',format=raw', + '-machine', 'kernel-irqchip=split', + '-nic', 'user,ipv6=off,model=e1000', + ] + + +def _qemu_arg_list_for_kernel(args, generator, target, size): + generator.prepare(target, using_kernel=True, target_size=size) + + arg_list = [ + '-enable-kvm', + '-m', str(args.qemu_ram) + 'M', + '-smp', str(args.cores), + '-drive', 'file=' + target.get_disk("disk") + ',format=raw', + ] + if target.get_disk("external") is not None: + arg_list += [ + '-drive', 'file=' + target.get_disk("external") + ',format=raw', + ] + arg_list += [ + '-nic', 'user,ipv6=off,model=e1000', + '-kernel', args.kernel, + '-append', + ] + if args.interactive: + arg_list += ['consoleblank=0 earlyprintk=vga root=/dev/sda1 init=/init rw'] + else: + arg_list += ['console=ttyS0 earlycon=uart8250,io,0x3f8,115200n8 root=/dev/sda1 init=/init rw'] + return arg_list + + +def _qemu_arg_list_for_kernel_bootstrap(args, generator, target, size): + generator.prepare(target, kernel_bootstrap=True, target_size=size) + if args.internal_ci_break_after: + break_scope, break_step = parse_internal_ci_break_after(args.internal_ci_break_after) + apply_internal_ci_break_to_tree( + generator.target_dir, + break_scope, + break_step, + args.internal_ci, + ) + os.remove(generator.target_dir + '.img') + generator.create_builder_hex0_disk_image(generator.target_dir + '.img', size) + + arg_list = [ + '-enable-kvm', + '-m', str(args.qemu_ram) + 'M', + '-smp', str(args.cores), + '-drive', 'file=' + generator.target_dir + '.img' + ',format=raw', + ] + if target.get_disk("external") is not None: + arg_list += [ + '-drive', 'file=' + target.get_disk("external") + ',format=raw', + ] + arg_list += [ + '-machine', 'kernel-irqchip=split', + '-nic', 'user,ipv6=off,model=e1000', + ] + return arg_list + + def bootstrap(args, generator, target, size, cleanup): """Kick off bootstrap process.""" print(f"Bootstrapping {args.arch}", flush=True) if args.chroot: - find_chroot = """ -import shutil -print(shutil.which('chroot')) -""" - chroot_binary = run_as_root('python3', '-c', find_chroot, - capture_output=True).stdout.decode().strip() - - generator.prepare(target, using_kernel=False) - - arch = stage0_arch_map.get(args.arch, args.arch) - init = os.path.join(os.sep, 'bootstrap-seeds', 'POSIX', arch, 'kaem-optional-seed') - run_as_root('env', '-i', 'PATH=/bin', chroot_binary, generator.target_dir, init, - cleanup=cleanup) - - elif args.bwrap: - init = '/init' - if not args.internal_ci or args.internal_ci == "pass1": - generator.prepare(target, using_kernel=False) - - arch = stage0_arch_map.get(args.arch, args.arch) - init = os.path.join(os.sep, 'bootstrap-seeds', 'POSIX', arch, 'kaem-optional-seed') - else: - generator.reuse(target) - - run('env', '-i', 'bwrap', '--unshare-user', - '--uid', '0', - '--gid', '0', - '--unshare-net' if args.external_sources else None, - '--setenv', 'PATH', '/usr/bin', - '--bind', generator.target_dir, '/', - '--dir', '/dev', - '--dev-bind', '/dev/null', '/dev/null', - '--dev-bind', '/dev/zero', '/dev/zero', - '--dev-bind', '/dev/random', '/dev/random', - '--dev-bind', '/dev/urandom', '/dev/urandom', - '--dev-bind', '/dev/ptmx', '/dev/ptmx', - '--dev-bind', '/dev/tty', '/dev/tty', - '--tmpfs', '/dev/shm', - '--proc', '/proc', - '--bind', '/sys', '/sys', - '--tmpfs', '/tmp', - init, - cleanup=cleanup) - - elif args.bare_metal: - if args.kernel: - generator.prepare(target, using_kernel=True, target_size=size) - path = os.path.join(args.target, os.path.relpath(generator.target_dir, args.target)) - print("Please:") - print(f" 1. Take {path}/initramfs and your kernel, boot using this.") - print(f" 2. Take {path}/disk.img and put this on a writable storage medium.") - else: - generator.prepare(target, kernel_bootstrap=True, target_size=size) - path = os.path.join(args.target, os.path.relpath(generator.target_dir, args.target)) - print("Please:") - print(f" 1. Take {path}.img and write it to a boot drive and then boot it.") + _bootstrap_chroot(args, generator, target, cleanup) + return + if args.bwrap: + _bootstrap_bwrap(args, generator, target, cleanup) + return + if args.bare_metal: + _bootstrap_bare_metal(args, generator, target, size) + return + if args.stage0_image: + arg_list = _qemu_arg_list_for_stage0_image(args, target) + elif args.kernel: + arg_list = _qemu_arg_list_for_kernel(args, generator, target, size) else: - if args.kernel: - generator.prepare(target, using_kernel=True, target_size=size) - - arg_list = [ - '-enable-kvm', - '-m', str(args.qemu_ram) + 'M', - '-smp', str(args.cores), - '-drive', 'file=' + target.get_disk("disk") + ',format=raw' - ] - if target.get_disk("external") is not None: - arg_list += [ - '-drive', 'file=' + target.get_disk("external") + ',format=raw', - ] - arg_list += [ - '-nic', 'user,ipv6=off,model=e1000', - '-kernel', args.kernel, - '-append', - ] - if args.interactive: - arg_list += ['consoleblank=0 earlyprintk=vga root=/dev/sda1 ' - 'rootfstype=ext3 init=/init rw'] - else: - arg_list += ['console=ttyS0 earlycon=uart8250,io,0x3f8,115200n8 ' - 'root=/dev/sda1 rootfstype=ext3 init=/init rw'] - else: - generator.prepare(target, kernel_bootstrap=True, target_size=size) - arg_list = [ - '-enable-kvm', - '-m', str(args.qemu_ram) + 'M', - '-smp', str(args.cores), - '-drive', 'file=' + generator.target_dir + '.img' + ',format=raw' - ] - if target.get_disk("external") is not None: - arg_list += [ - '-drive', 'file=' + target.get_disk("external") + ',format=raw', - ] - arg_list += [ - '-machine', 'kernel-irqchip=split', - '-nic', 'user,ipv6=off,model=e1000' - ] - if not args.interactive: - arg_list += ['-no-reboot', '-nographic'] - run(args.qemu_cmd, *arg_list, cleanup=cleanup) + arg_list = _qemu_arg_list_for_kernel_bootstrap(args, generator, target, size) + if not args.interactive: + arg_list += ['-no-reboot', '-nographic'] + run(args.qemu_cmd, *arg_list, cleanup=cleanup) if __name__ == "__main__": main() diff --git a/seed/script-generator.c b/seed/script-generator.c index faedd1d7..66f97ad9 100644 --- a/seed/script-generator.c +++ b/seed/script-generator.c @@ -32,6 +32,48 @@ struct Directive { }; typedef struct Directive Directive; +char *steps_root = "/steps"; +char *config_root = "/steps"; + +char *join_path(const char *base, const char *suffix) { + char *out = calloc(MAX_STRING, sizeof(char)); + strcpy(out, base); + if (strlen(out) > 0 && out[strlen(out) - 1] != '/') { + strcat(out, "/"); + } + while (*suffix == '/') { + suffix += 1; + } + strcat(out, suffix); + return out; +} + +char *config_path(const char *suffix) { + return join_path(config_root, suffix); +} + +char *dirname_from_path(const char *path) { + char *slash = strrchr(path, '/'); + char *out = calloc(MAX_STRING, sizeof(char)); + if (slash == NULL) { + strcpy(out, "."); + return out; + } + if (slash == path) { + strcpy(out, "/"); + return out; + } + strncpy(out, path, slash - path); + return out; +} + +void write_steps_prefix(FILE *out) { + fputs(steps_root, out); + if (steps_root[strlen(steps_root) - 1] != '/') { + fputs("/", out); + } +} + /* Tokenizer. */ /* Skip over a comment. */ @@ -120,10 +162,11 @@ typedef struct Variable Variable; Variable *variables; Variable *load_config() { - FILE *config = fopen("/steps/bootstrap.cfg", "r"); - /* File does not exist check. */ + char *config_file = config_path("bootstrap.cfg"); + FILE *config = fopen(config_file, "r"); if (config == NULL) { - return NULL; + fputs("Unable to open bootstrap.cfg\n", stderr); + exit(1); } char *line = calloc(MAX_STRING, sizeof(char)); @@ -392,9 +435,12 @@ Directive *interpreter(Directive *directives) { /* Script generator. */ FILE *start_script(int id, int bash_build) { - /* Create the file /steps/$id.sh */ + /* Create the file ${steps_root}/$id.sh */ char *filename = calloc(MAX_STRING, sizeof(char)); - strcpy(filename, "/steps/"); + strcpy(filename, steps_root); + if (filename[strlen(filename) - 1] != '/') { + strcat(filename, "/"); + } strcat(filename, int2str(id, 10, 0)); strcat(filename, ".sh"); @@ -407,28 +453,47 @@ FILE *start_script(int id, int bash_build) { } if (bash_build) { + char *bootstrap_file = config_path("bootstrap.cfg"); + char *env_file = config_path("env"); + char *helpers_file = join_path(steps_root, "helpers.sh"); fputs("#!/bin/bash\n", out); if (strcmp(get_var("INTERACTIVE"), "True") == 0) { if (bash_build != 1) { fputs("set -eEo pipefail\ntrap 'env PS1=\"[TRAP] \\w # \" bash -i' ERR\n", out); } else { /* FIXME early bash has buggy ERR trap handling */ - fputs("set -e\ntrap 'bash -c '\"'\"'while true; do printf \"" - "[TRAP - use Ctrl+D] $(pwd) # \"; eval \"$(cat)\"; done'\"'\"'' EXIT\n", - out); + fputs("set -e\ntrap 'status=$?; if [ \"${status}\" -ne 0 ]; then env PS1=\"[TRAP] \\w # \" bash -i; fi' EXIT\n", out); } } else { fputs("set -e\n", out); } - fputs("cd /steps\n", out); - fputs(". ./bootstrap.cfg\n", out); - fputs(". ./env\n", out); - fputs(". ./helpers.sh\n", out); + fputs("cd ", out); + fputs(steps_root, out); + fputs("\n", out); + fputs(". ", out); + fputs(bootstrap_file, out); + fputs("\n", out); + fputs(". ", out); + fputs(env_file, out); + fputs("\n", out); + fputs("SRCDIR=", out); + fputs(steps_root, out); + fputs("\n", out); + fputs(". ", out); + fputs(helpers_file, out); + fputs("\n", out); } else { + char *env_file = config_path("env"); fputs("set -ex\n", out); - fputs("cd /steps\n", out); + fputs("cd ", out); + fputs(steps_root, out); + fputs("\n", out); output_config(out); - FILE *env = fopen("/steps/env", "r"); + FILE *env = fopen(env_file, "r"); + if (env == NULL) { + fputs("Unable to open env\n", stderr); + exit(1); + } char *line = calloc(MAX_STRING, sizeof(char)); while (fgets(line, MAX_STRING, env) != 0) { /* Weird M2-Planet behaviour. */ @@ -454,7 +519,7 @@ void output_call_script(FILE *out, char *type, char *name, int bash_build, int s } else { fputs("kaem --file ", out); } - fputs("/steps/", out); + write_steps_prefix(out); if (strlen(type) != 0) { fputs(type, out); fputs("/", out); @@ -463,6 +528,23 @@ void output_call_script(FILE *out, char *type, char *name, int bash_build, int s fputs(".sh\n", out); } +void output_resume_network_init(FILE *out) { + fputs("if [ -f ", out); + write_steps_prefix(out); + fputs("helpers.sh ]; then\n", out); + fputs(". ", out); + write_steps_prefix(out); + fputs("helpers.sh\n", out); + fputs("resume_network_init \"", out); + fputs(config_root, out); + fputs("\"\n", out); + fputs("fi\n", out); +} + +void output_init_call_script(FILE *out, char *name, int bash_build) { + output_call_script(out, "", name, bash_build, 0); +} + void output_build(FILE *out, Directive *directive, int pass_no, int bash_build) { if (bash_build) { fputs("build ", out); @@ -486,7 +568,8 @@ void generate_preseed_jump(int id) { FILE *out = fopen("/preseed-jump.kaem", "w"); fputs("set -ex\n", out); fputs("PATH=/usr/bin\n", out); - fputs("bash /steps/", out); + fputs("bash ", out); + write_steps_prefix(out); fputs(int2str(id, 10, 0), out); fputs(".sh\n", out); fclose(out); @@ -503,8 +586,14 @@ void generate(Directive *directives) { int counter = 0; - /* Initially, we use kaem, not bash. */ + /* + * Default /steps manifests start in kaem. + * Alternative step roots (for example /steps-guix) start in bash. + */ int bash_build = 0; + if (strcmp(steps_root, "/steps") != 0) { + bash_build = 2; + } FILE *out = start_script(counter, bash_build); counter += 1; @@ -576,6 +665,7 @@ void generate(Directive *directives) { exit(1); } fputs("#!/bin/bash\n", out); + output_resume_network_init(out); } else { out = fopen(filename, "w"); if (out == NULL) { @@ -584,7 +674,7 @@ void generate(Directive *directives) { } fputs("set -ex\n", out); } - output_call_script(out, "", int2str(counter, 10, 0), bash_build, 0); + output_init_call_script(out, int2str(counter, 10, 0), bash_build); fclose(out); out = start_script(counter, bash_build); counter += 1; @@ -598,8 +688,8 @@ void generate(Directive *directives) { } void main(int argc, char **argv) { - if (argc != 2) { - fputs("Usage: script-generator