feat(arc): copy capability payloads once per node into a capability store

.github/workflows/capabilities.yml

@@ -1,6 +1,6 @@
 name: Capabilities
 
-# Runs the ARC role's scale-set profile capabilities in a kind cluster (tests/capabilities/run.sh): the reference Node.js tool image and libpq system-library image built from this checkout, values rendered through the role's own tasks, the scale-set chart rendered with them and checked against the ARC CRDs, and a pod from the chart's runner pod template whose init containers fill the tool cache and the sysroot, whose job-started hook puts the tool on PATH and the library on its search paths, and whose step finds the tool through @actions/tool-cache and runs the library's programs. Only on changes that can affect them. Builds nothing it publishes.
+# Runs the ARC role's scale-set profile capabilities in a kind cluster (tests/capabilities/run.sh): the reference Node.js tool image and libpq system-library image built from this checkout, values rendered through the role's own tasks, the scale-set chart rendered with them and checked against the ARC CRDs, and a pod from the chart's runner pod template whose init containers fill the tool cache and the sysroot, whose job-started hook puts the tool on PATH and the library on its search paths, and whose step finds the tool through @actions/tool-cache and runs the library's programs; then the same images pushed to a registry container in the test, two pods of a profile with the capability store on against the node's store, and the node cache pruner run once in a pod from the role's rendered manifests. Only on changes that can affect them. Builds nothing it publishes.
 #
 # GitHub-hosted ubuntu-latest has Docker, kind, kubectl and helm already.
 
@@ -9,8 +9,11 @@ on:
     paths:
       - roles/github_runner_arc/**
       - plugins/filter/arc.py
+      - scripts/node-cache-pruner.sh
+      - autoscaler/Dockerfile
       - tool-images/**
       - tests/capabilities/**
+      - tests/node_cache_pruner/render.yml
       - .github/workflows/capabilities.yml
   workflow_dispatch:
 

roles/github_runner_arc/defaults/main.yml

@@ -25,7 +25,14 @@ github_runner_arc_node_cache_path: "/node-cache"
 # Where on the node the cache lives, as the directory every profile's own directory (<namespace>/<release>) sits under, so profiles never share a cache. It must be a dedicated directory the role may create and empty as root: written without a repeated or trailing slash (every path of the cache is built from this one string), at least three levels deep, with a last component ending in node-cache, and not inside a system directory such as /etc or /var/lib/kubelet. A symbolic link at or above the path defeats this validation. Each profile's directory is a hostPath volume created when a pod first needs it. Its lifetime is the node's: it survives pods, and is emptied by the pruner below when it outgrows its limit.
 github_runner_arc_node_cache_host_path: "/var/lib/github-runner/node-cache"
 
-# The most gibibytes one profile's cache may hold on a node before the pruner empties it. The limit is per profile, so a node's worst case is the sum of the limits of the profiles with a cache on it, times github_runner_arc_node_cache_hard_limit_factor. The pruner (scripts/node-cache-pruner.sh, a DaemonSet on the nodes runner pods may use) empties a cache only while no runner pod of its profile runs on the node, since a package manager's store is in use by every job that linked from it. 0 installs no pruner and leaves the cache to grow; a negative value, a leading zero, a fraction or a value above 1073741824 is an error (the pruner's own arithmetic bounds it, and the role refuses what the pruner would).
+# Where on the node the capability store lives, for profiles with capability_store on: one entry per image a profile's capabilities (and the dind sidecar's externals) copy from, named for the image's digest and the kind of payload copied from it, shared by the pods of a node and mounted read-only by them. A dedicated directory the role creates and the pruner removes entries from as root, with the same rules as the node cache's root, whose last component is capability-store or ends in -capability-store.
+github_runner_arc_capability_store_host_path: "/var/lib/github-runner/capability-store"
+# Where the store's root is mounted in the init containers, and under which the runner container mounts, read-only, each entry its pod uses as <path>/<key> (the tool cache's links point into those): the runner container never mounts the root, so a job cannot read another profile's entries.
+github_runner_arc_capability_store_path: "/opt/capability-store"
+# The image the store's own init containers run (creating the root, flushing and publishing a copy with sync and mv -T, merging the sysroot with cp -al and linking the tool cache), pinned by digest since one of them runs as root on a hostPath. Its sh, mkdir, chmod, chown, sync, mv, cp, find and rm must be busybox's or GNU's: mv -T refuses to move onto a directory that is not empty, which publishing relies on.
+github_runner_arc_capability_store_image: "docker.io/library/busybox:1.37.0@sha256:bdf57e528e45e4433820e045b29b4597825a1c9e38353532d90a01445013f82e"
+
+# The most gibibytes one profile's cache may hold on a node before the pruner empties it. The limit is per profile, so a node's worst case is the sum of the limits of the profiles with a cache on it, times github_runner_arc_node_cache_hard_limit_factor. The pruner (scripts/node-cache-pruner.sh, a DaemonSet on the nodes runner pods may use) empties a cache only while no runner pod of its profile runs on the node, since a package manager's store is in use by every job that linked from it. 0 installs no pruner and leaves the cache to grow, which a profile with capability_store refuses (the pruner removes the store's unused entries); a negative value, a leading zero, a fraction or a value above 1073741824 is an error (the pruner's own arithmetic bounds it, and the role refuses what the pruner would).
 github_runner_arc_node_cache_size_limit_gib: 20
 
 # The hard limit, as a multiple of the size limit. A cache that a runner pod keeps busy cannot be emptied, and one over the hard limit is reported on every round by a log line starting NODE_CACHE_OVER_HARD_LIMIT, for whatever collects the pruner's logs to alert on. The pruner changes nothing about the node to force the emptying: draining the node or lowering its runner count lets the next round do it, and kubelet's disk-pressure eviction still protects the node itself.

roles/github_runner_arc/meta/argument_specs.yml

@@ -12,7 +12,8 @@ argument_specs:
         default: []
         description:
           - Orgs whose runner scale sets this host installs. Each entry has name, app_id (needed only when the role writes the App Secret; otherwise the existing Secret supplies it), image, private_key or private_key_op_reference, and optionally installation_id and app_secret_name (the App Secret's name, default <org>-github-app).
-          - Each entry's scale_set_profiles is a non-empty list of profiles, each with suffix and optionally namespace (default arc-runners-<org><suffix>), release_name (default <org>-runners<suffix>), values_file, max_runners, min_runners, container_mode, resources, node_selector, runs_on_label, scale_set_labels, capabilities, tolerations, autoscale, sizing and burst.
+          - Each entry's scale_set_profiles is a non-empty list of profiles, each with suffix and optionally namespace (default arc-runners-<org><suffix>), release_name (default <org>-runners<suffix>), values_file, max_runners, min_runners, container_mode, resources, node_selector, runs_on_label, scale_set_labels, capabilities, capability_store, tolerations, autoscale, sizing and burst.
+          - capability_store, a boolean (default false), makes the profile's runner pods copy each tool-cache and system-library capability's payload, and the runner's externals for a Docker-in-Docker sidecar the values write out, into the capability store (github_runner_arc_capability_store_host_path), which the pods of a node share, instead of into every pod's own volumes, and mount the result read-only. Every image those copies come from must be pinned by digest, and the store needs the node cache pruner, so a profile that sets it is refused while github_runner_arc_node_cache_size_limit_gib is 0 or github_runner_arc_node_label_key is empty, since the pruner runs only on the nodes carrying that label. A job that can reach a privileged Docker daemon can write to the store, so give it only to a profile whose workflows are all of one trust level.
           - max_runners, when set, is the release's maxRunners and takes precedence over its values file and its sizing.
           - scale_set_labels replaces runs_on_label with the whole list of scaleSetLabels; an empty list sets none, so jobs target the release name.
           - tolerations is a list of Kubernetes tolerations for the profile's runner pods, for nodes the eligibility label selects that also carry a taint to keep other workloads off. Measured sizing counts a node whose only repelling taints these match, as the pods can be placed there; without them it leaves such a node out. With burst they come before burst.tolerations.
@@ -44,10 +45,22 @@ argument_specs:
         type: str
         default: /var/lib/github-runner/node-cache
         description: The directory on each node under which every profile's node cache lives, in a directory of its own named <namespace>/<release>. It must be an absolute path with no colon, no repeated slash and no trailing slash, at least three levels deep, whose last component ends with node-cache, and not a system directory or inside one. A symbolic link at or above it defeats the validation.
+      github_runner_arc_capability_store_host_path:
+        type: str
+        default: /var/lib/github-runner/capability-store
+        description: The directory on each node that holds the capability store of the profiles with capability_store, one entry per image payload, shared by the pods of a node. It must be an absolute path with no colon or line break, no repeated slash and no trailing slash, at least three levels deep, whose last component is capability-store or ends with -capability-store, not a system directory or inside one, and neither equal to, inside nor containing github_runner_arc_node_cache_host_path. A symbolic link at or above it defeats the validation. The node cache pruner removes the entries no scale set or pod uses, as root, only while some profile has capability_store; once none has, the entries stay until the operator deletes the directory on each node.
+      github_runner_arc_capability_store_path:
+        type: str
+        default: /opt/capability-store
+        description: Where the store's root is mounted in the profiles' init containers, and under which the runner container mounts, read-only, each entry its pod uses as <path>/<key>. An absolute directory other than / with no colon or line break, which for a profile whose capabilities have a payload in the store may not equal, contain or sit inside the tool cache path, the sysroot path, the job-started hooks directory or the node cache path.
+      github_runner_arc_capability_store_image:
+        type: str
+        default: docker.io/library/busybox:1.37.0@sha256:bdf57e528e45e4433820e045b29b4597825a1c9e38353532d90a01445013f82e
+        description: The image the store's own init containers run (creating the root, flushing and publishing a copy, merging the sysroot, linking the tool cache). It must be pinned by digest (name@sha256:<64 hex digits>), since one of them runs as root on a hostPath, and its sh, mkdir, chmod, chown, sync, mv, cp, find and rm must be busybox's or GNU's, mv -T refusing to move onto a directory that is not empty.
       github_runner_arc_node_cache_size_limit_gib:
         type: int
         default: 20
-        description: The most gibibytes one profile's node cache may hold on a node before the pruner empties it, which it does only while no runner pod of the profile runs on the node. The limit is per profile, so a node's worst case is the sum over the profiles with a cache on it, times the hard limit factor. 0 installs no pruner; a negative value, a leading zero, a fraction or more than 1073741824 is an error.
+        description: The most gibibytes one profile's node cache may hold on a node before the pruner empties it, which it does only while no runner pod of the profile runs on the node. The limit is per profile, so a node's worst case is the sum over the profiles with a cache on it, times the hard limit factor. 0 installs no pruner, which a profile with capability_store cannot do without, since the pruner removes the store's unused entries; a negative value, a leading zero, a fraction or more than 1073741824 is an error.
       github_runner_arc_node_cache_hard_limit_factor:
         type: int
         default: 2

roles/github_runner_arc/tasks/decide_node_cache_pruner.yml

@@ -1,10 +1,15 @@
 ---
-# Decides which profiles have a node cache and whether the pruner runs, from the fleet and the capability catalogue. The pruner runs while the size limit is above zero and some profile has a node cache, and is removed otherwise, so it follows the fleet. A directory a removed capability leaves is emptied only while another profile keeps the pruner installed; once the pruner is gone, delete <github_runner_arc_node_cache_host_path> on each node by hand.
+# Decides which profiles have a node cache and whether the pruner runs, from the fleet and the capability catalogue. The pruner runs while the size limit is above zero and some profile has a node cache or uses the capability store, and is removed otherwise, so it follows the fleet. A directory a removed capability leaves is emptied only while another profile keeps the pruner installed; once the pruner is gone, delete <github_runner_arc_node_cache_host_path> on each node by hand.
 
 - name: "Node cache: find the profiles that carry a node-cache capability"
   ansible.builtin.set_fact:
     github_runner_arc_node_cache_profiles: "{{ github_runner_arc_fleet.profiles | exadev.github_runner.arc_node_cache_profiles(github_runner_arc_capabilities) }}"
 
+# The pruner also removes the capability store's entries that no scale set or pod uses, so it runs for a fleet with a store profile too. Which entries are used is read from the scale sets and pods of every namespace a pod made from an earlier template may still run in, not only those of the profiles that use the store now (see arc_capability_store_namespaces); the list is empty, and the store left alone, while no profile uses it. It needs the pruner's Roles as the cluster has them (github_runner_arc_node_cache_pruner_access) and the namespaces that are Terminating (github_runner_arc_terminating_namespaces), which it leaves out.
+- name: "Node cache: find the namespaces the capability store's references are read from"
+  ansible.builtin.set_fact:
+    github_runner_arc_capability_store_namespaces: "{{ github_runner_arc_fleet.profiles | exadev.github_runner.arc_capability_store_namespaces(github_runner_arc_node_cache_pruner_access.results | map(attribute='resources') | flatten, github_runner_arc_node_cache_pruner_labels, github_runner_arc_capability_store_pruner_role, github_runner_arc_terminating_namespaces) }}"
+
 - name: "Node cache: decide whether the pruner is wanted"
   ansible.builtin.set_fact:
-    github_runner_arc_node_cache_pruner_wanted: "{{ github_runner_arc_node_cache_size_limit_gib | int > 0 and github_runner_arc_node_cache_profiles | length > 0 }}"
+    github_runner_arc_node_cache_pruner_wanted: "{{ github_runner_arc_node_cache_size_limit_gib | int > 0 and (github_runner_arc_node_cache_profiles | length > 0 or github_runner_arc_capability_store_namespaces | length > 0) }}"