Continuous integration

Doctools has a continuous deployment integration pipeline that works as follows:

                   ┌──────────────────┐
                ┌─►│Build Doc Latest  ├─┐
                │  └──────────────────┘ │
                │                       │
┌─────────────┐ │  ┌────────────────┐   │  ┌──────────────┐
│Build Package├─┼─►│Build Doc on Min├───┼─►│Deploy Package│
└─────────────┘ │  └────────────────┘   │  └──────────────┘
                │                       │
                │  ┌──────────┐         │
                ├─►│Custom Doc├─────────┤
                │  └──────────┘         │
                │                       │
                │  ┌─────┐              │
                └─►│Tests├──────────────┘
                   └─────┘

The Build Package step “compiles” JavaScript and SASS, fetches third-party assets and licenses and generates the Python package.

Then, in the middle-stage, two parallel runs are launched:

• Build Doc Latest: uses the latest stable dependencies releases to generate this documentation, and store as an artifact.

• Build Doc on Min: uses the minimum requirements dependencies to generate this documentation, but the output is discarded.

• Custom Doc: calls Custom Doc to check if the CLI tool succeeds in generating a full custom PDF document.

• Tests: run tests using pytest, in special, methods that are not called during the Build Doc * pipelines.

Both of them are set to fail-on-warning during the documentation generation.

Finally, the Deploy Package:

• Grabs the version and checks if the tag version already exists:

  • If so, set to update the symbolic pre-release release.

  • If not, set to update the symbolic latest and pre-relase release.

• Still if a new version:

  • Create the git tag and push to origin.

  • Create the tagged release.

  • Upload the artifact to the tagged release.

• Upload the artifact to the symbolic release (pre-release, latest).

• Finally, the Build Doc Latest artifact is downloaded and deployed to the branch gh-pages

By design, the live page on github.io follows the pre-release/latest commit-ish; properly versioned live documentation should be managed by an external system that watches the git tags (e.g. readthedocs).

This approach allows having a single defined version on adi_doctools/__init__.py, and have the tags created and releases created/updated without much fuzz.

The philosophy is to have latest updated on tag increment and first successful run, and pre-release updated on successful run without tag change. These releases exist to provide a pointer to the latest/pre-release packages, e.g. releases/download/latest/adi-doctools.tar.gz.

Non-handled corner-cases mitigations:

• Release pre-release and latest must exist prior the first run.

• Branch gh-pages must exist with at least one commit.

Configure podman

Below are suggested instructions for setting up podman on a Linux environment, if you wish to use it as your container engine. If you already use something else like docker, keep it and skip this section.

Adjust to your preference as needed, and skip the steps marked in green if not using WSL2.

Install podman from your package manager.

Ensure cgroup v2 on wsl2’s .wslconfig:

[wsl2]
kernelCommandLine = cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1

Restart wsl2.

Enable podman service for your user.

~$

systemctl enable --now --user podman.socket

~$

systemctl start --user podman.socket

Set the DOCKER_HOST variable on your ~/.bashrc:

export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock

Network users & partitions

Podman default configuration expects a local user to be able to create a user namespace where multiple IDs are mapped and a compatible partition to use as the storage location graphRoot.

Note

The ideal solution is to create a local non-root user and storage location. Podman processes should then be started under this user UID.

Network systems using solutions such as SSSD do not append the user to the system (is not listed in /etc/subuid), so automatic user namespace is not possible. To be compatible with this configuration, a single UID within a user space needs to be used, achieved with the ignore_chown_errors parameter.

Normally these systems also mount an network file system (nfs) as the home folder, which is also not supported. In this case, the graphRoot location needs to be set to somewhere else (an easy test location is /tmp).

This is an example of ~/.config/containers/storage.conf to support such environments:

[storage]
driver = "overlay"
# Set to a path in a non-nfs partition
graphRoot = "/tmp"

[storage.options.overlay]
# Single UID
ignore_chown_errors = "true"

Ensure apply with podman system migrate and see the changed settings with podman info.

An alternative mitigation for nfs is to create an xfs disk image and mount, but since mount requires root permission it is unlikely to be helpful for most users:

truncate -s 100g ~/.local/share/containers-xfs.img
mkfs.xfs -m reflink=1  ~/.local/share/containers-xfs.img -m bigtime=1,inobtcount=1 -i nrext64=0
sudo mount ~/.local/share/containers-xfs.img ~/.local/share/containers

Build the container image

To build the container image, use your favorite container engine:

~$

cd ~/doctools

# or docker, ...

~/doctools$

podman build --tag adi/doctools:latest ci

You may want to build the container in a host, where you have all your tools installed, and then deploy to a server. In this case, export the image and then import on the server:

user@host:~/doctools$

podman save -o adi-doctools.tar adi/doctools:latest

user@host:~/doctools$

scp adi-doctools.tar server:/tmp/

admin@server:/tmp$

podman load -i adi-doctools.tar

Or if you are feeling adventurous:

user@host:~/doctools$

podman save adi/doctools:latest | ssh server "cat - | podman load"

Interactive run

At its core, the workflows are straight forward, roughly they do:

The Tests step:

~$

cd tests ; pytest

Build Doc *:

~$

cd docs ; make html

But at a specific minimum and maximum supported environment version.

Custom Doc:

~$

mkdir /tmp/test-pdf ; cd $_

/tmp/test-pdf$

adoc custom-doc ; adoc custom-doc

Doing the relevant step on host covers most issues that the CI would catch.

You can use the container image with the container-run.sh script. to interactive login into an image, mounting the provided path, to run the steps on the container, for example:

~/doctools$

container-run adi/doctools .

~/doctools$

python3 -m venv .venv

~/doctools$

source .venv/bin/activate ; \
     pip3 install -e . ; \
     pip3 install pytest

~/doctools$

cd tests ; pytest

~/doctools/tests$

exit

You can also use just cr adi/doctools. One liner to install:

~$

grep "/container-run.sh" ~/.bashrc || \
   { curl "https://raw.githubusercontent.com/analogdevicesinc/doctools/refs/heads/main/ci/scripts/container-run.sh" \
     -o ~/.local/bin/container-run.sh && \
   echo "source ~/.local/bin/container-run.sh" >> ~/.bashrc ; }

Self-hosted runner

To host your GitHub Actions Runner, set up your secrets:

# e.g. MyVerYSecRunnerToken

~$

printf RUNNER_TOKEN | podman secret create public_doctools_runner_token -

The runner token is obtained from the GUI at github.com/<org>/<repository>/settings/actions/runners/new.

If github_token from Self-hosted cluster is set, the runner_token is ignored and a new one is requested.

~/doctools$

podman run \
   --secret public_doctools_runner_token,type=env,target=runner_token \
   --env org_repository=analogdevicesinc/doctools \
   --env runner_labels=repo-only,big_cpu \
   adi/doctools

Docker alternative

docker does not have a built-in keyring, instead you pass directly to run command. Consider hardening strategies to mitigate risks, like using another keyring as below.

~/doctools$

docker run \
   --env public_doctools_runner_token=$(gpg --quiet --batch --decrypt /run/secrets/public_doctools_runner_token.gpg) \
   --env org_repository=analogdevicesinc/doctools \
   --env runner_labels=repo-only,big_cpu \
   localhost/adi/doctools

The environment variable runner_labels (comma-separated), set the runner labels. If not provided on the Containerfile as ENV runner_labels=<labels,> or as argument --env runner_labels=<labels,>, it defaults to repo-only. version, also an environment variable, also is concatanated to runner_labels and name_label, so the effective final runner_labels is <version>,<labels,> Version should always be set in the Containerfile. Most of the time, you want to use the Containerfile-set environment variable.

The repo-only label ensures that the jobs run only in the repository-scoped runners, avoiding them to assigned to organization-level runners.

If you are in an environment as described in Network users & partitions, append these flags to every podman run command:

• --user root: due to ignore_chown_errors allowing a single user mapping, this user is root (0). Please note that this the container’s root user and in most images is the only available user.

• --env RUNNER_ALLOW_RUNASROOT=1: suppresses the GitHub Action runner “Must not run with sudo”. Again, is the container’s root.

Self-hosted cluster

Systemd can run the runner process directly in an isolated environment using Linux kernel namespaces. This eliminates the conmon intermediate process, gives systemd direct ownership of the runner PID, and integrates resource limits into the unit, instead of using Podman/Docker equivalents.

Export from Podman the OCI image tarball and extract it:

~$

out=~/.local/share/container/images/adi-doctools-latest

~$

mkdir -p $out

~$

podman export $(podman create adi/doctools:latest) | tar -xC $out

Note

To clean-up use unshare:

~$

podman unshare rm -rf ~/.local/share/container/images/adi-doctools-latest

Below is a suggested systemd service at ~/.config/systemd/user/container-public-doctools@.service:

[Unit]
Description=container public doctools ci %i
Wants=network-online.target
Requires=container-init@adi-doctools-latest.%i.service
After=container-init@adi-doctools-latest.%i.service

[Service]
Restart=on-success
ExecStart=/usr/local/bin/entrypoint.sh
Environment="version=latest"
Environment="name_label=%i"
Environment="org_repository=analogdevicesinc/doctools"
Environment="HOME=/home/runner"
Environment="USER=runner"
Environment="LOGNAME=runner"
UnsetEnvironment=XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_STATE_HOME XDG_RUNTIME_DIR XDG_DATA_DIRS XDG_CONFIG_DIRS DBUS_SESSION_BUS_ADDRESS machine_name SHELL
LoadCredential=github_token:%h/.local/share/container/secrets/public_github_token
RootDirectory=%h/.local/share/container/images/adi-doctools-latest
BindPaths=%h/.local/share/container/runner/adi-doctools-latest/%i:/home/runner
BindReadOnlyPaths=/etc/resolv.conf
PrivateUsers=yes
PrivateTmp=yes
PrivateIPC=yes
PrivateMounts=yes
ProtectHostname=yes
MemoryMax=16G
MemorySwapMax=20G
CPUQuota=400%
TasksMax=512
TimeoutStopSec=600
Type=exec

[Install]
WantedBy=multi-user.target

Note

HOME=/home/runner must be set if host user is not runner. Also the image runner UID is 1000.

To identify the runner host machine, an alias-hostname is used via ~/.config/systemd/user/container-public-doctools@.service.d/machine.conf (indentical for the same host, but duplicated per container scope):

[Service]
Environment="name_label=big-server-%i"

The script ~/.local/bin/container-init.sh provisions an ephemeral copy:

#!/bin/sh
name="${1%%.*}"
idx="${1##*.}"
mkdir -p "$HOME/.local/share/container/runner/$name"
rm -rf "$HOME/.local/share/container/runner/$name/$idx"
cp -a "$HOME/.local/share/container/images/$name/home/runner" \
      "$HOME/.local/share/container/runner/$name/$idx"

And the init service at ~/.config/systemd/user/container-init@.service:

[Unit]
Description=container init home %i

[Service]
Type=oneshot
ExecStart=%h/.local/bin/container-init.sh %i

Remember to systemctl --user daemon-reload after modifying.

The secrets are plain text files readable only by the runner user:

~$

cd ~/.local/share/container/secrets

~/.local/share/container/secrets$

install -m 600 -D /dev/stdin public_github_token <<< "github_pat_MyToken"

If a compatible TPM2 is available, use LoadCredentialEncrypted= instead of LoadCredential= and store AES256-GCM ciphertext to it:

~$

cd ~/.local/share/container/secrets

~/.local/share/container/secrets$

systemd-creds encrypt --name=github_token - public_github_token.cred

Then in the unit replace LoadCredential= with:

LoadCredentialEncrypted=github_token:%h/.local/share/container/secrets/public_github_token.cred

Enable and start the service:

systemctl --user enable container-public-doctools@0.service
systemctl --user start container-public-doctools@0.service

Attention

User services are terminated on logout, unless you define loginctl enable-linger <your-user> first.

See systemd’s container interface and systemd’s credentials for more information.

Resource quotas

MemoryMax= and CPUQuota= require cgroup v2 controllers to be delegated to the user slice:

$ sudo mkdir -p /etc/systemd/system/user@.service.d
$ cat <<EOF | sudo tee /etc/systemd/system/user@.service.d/delegate.conf
$ [Service]
$ Delegate=cpu cpuset io memory pids
$ EOF
$ sudo systemctl daemon-reload

Tune the limit flags for your needs. The cpu config requires a kernel with CONFIG_CFS_BANDWIDTH enabled. You can check with zgrep CONFIG_CFS_BANDWIDTH= /proc/config.gz.

Runner token and GitHub token

Instead of passing runner_token, you can also pass a github_token to generate the runner_token on demand. Using the github_token is the recommended approach because during clean-up the original runner_token may have expired already.

Alternatively, you can mount a FIFO to /var/run/secrets/runner_token to generate a token just in time, without ever passing the github_token to the container (scripts not provided).

However, please note, just like the GitHub Actions generated GITHUB_TOKEN, the path /run/secrets/runner_token can be read by workflows, while the previous option is removed from the environment prior executing the GitHub Actions runtime.

The order of precedence for authentication token is:

1. github_token: environment variable.

2. runner_token: plain text or FIFO at /run/secrets/runner_token.

3. runner_token: environment variable.

Please understand the security implications and ensure the token secrecy, by for example, require manual approval for running workflows PRs from third party sources and don’t relax runner user permissions.

The required GitHub Fine-Grained token permission should be set as follows:

For repository runner:

• administration:write: “Administration” repository permissions (write).

For org runner:

• organization_self_hosted_runners:write: “Self-hosted runners” organization permissions (write).

• The user needs to be an org-level admin.