How NixOS is built

We can use dig to figure out where the nixos.org nameservers are hosted.

$ dig NS nixos.org +short
ns-61-b.gandi.net.
ns-161-c.gandi.net.
ns-177-a.gandi.net.

Turns out they're hosted on gandi.net. Let's move on to see where this is defined.

On NixOS/infra there's a dns directory with a flake-module.nix in it, where we see that DNSControl is used to configure domains. Looking at creds.json there's an entry to setup Gandi credentials which matches what we discovered earlier. Furthermore, at dnsconfig.js we include four more files, one for each of the following domains:

• nixcon.org
• nix.dev
• nixos.org
• ofborg.org

Following this breadcrumb, we arrive at dns/nixos.org where channels.nixos.org is defined as a CNAME to a Fastly domain. The exact same setup is used for cache.nixos.org, the NixOS binary cache from where users download prebuilt binaries.

In infra/terraform we find the Terraform files for managing the S3 buckets and the Fastly CDN setup. For channels in particular, we define a S3 bucket with a website endpoint enabled which we can view in nix-channels.s3-website-us-east-1.amazonaws.com.

Further down the file, there's a definition for a Fastly Service which uses the public website endpoint of the bucket as a backend. Again, we have a similar setup for cache.nixos.org.

To recap, channels.nixos.org points to Fastly which caches responses and acts as a global CDN, which in turn uses an AWS S3 bucket as the source for channels and packages.

There are two main actions that run on a schedule. The first one is periodic merges which merges the master branch into staging every few hours. The other one is labels which runs every ten minutes and automatically applies labels to PRs and issues via this script.

Every time someone opens a new pull request or a pull request is updated, some actions run. In particular these actions revolve around different areas of the PR, from linting or building the code changes to applying labels and pinging the maintainers of a package to get a review.

The lint workflow lints the changes and also calls nixpkgs-vet which is a software specifically made to enforce NixOS RFC 140.

Eval, as the name suggests, evaluates the derivations (in all supported systems), calculates some statistics for later use and verifies things still evaluate. These statistics might include what paths were added or changed in a PR. Using PR #429664 as an example, we get the following artifact out of the diff between the PR and the previous master branch for x86_64.

{
  "added": [],
  "changed": [
    "python312Packages.llm-ollama.x86_64-linux",
    "python313Packages.llm-ollama.x86_64-linux",
    "release-checks"
  ],
  "removed": []
}

This information is later used by the previously mentioned labels workflow to add related labels to a PR.

Build, as the name suggests, builds docs, shell, lib, tarball but doesn't actually build the package. A point of interest here is the fact that the build action uses Cachix, a hosted Nix binary cache. This cache is only meant to be used by the Nixpkgs CI and shouldn't be trusted.

Reviewers figures out which contributors should get a ping to review the PR.

Builds in Hydra are organized in the following structure:

Project -> Jobset -> Job

To get back to the initial question of how the ISO is built, let's look at the NixOS project. We have a jobset for each release (release-25.05, staging, unstable-small, etc). Next, if we search for iso there's a job called nixos_iso_minimal. The configuration of the `release-25.05` jobset calls nixos/release-combined.nix from Nixpkgs.

This nix file specifies all the jobs that Hydra should build for this jobset. One of these jobs is nixos.iso_minimal, which is our entry point for building the ISO.

Similarly, for Nixpkgs there's a Nixpkgs project with a jobset called trunk that builds the latest committed version of a package.

Anything that Hydra builds is signed and uploaded to the S3 bucket for cache.nixos.org.

We can verify this by querying the cache. This is the latest build of a package I maintain. On the details tabs we see:

Output store paths
 /nix/store/zi6ypq21r8534cx53bx8rx930k749xs8-python3.13-llm-ollama-0.12.0-dist,
 /nix/store/clz5dsk4cxsakbw9zw634riwhpjwhjk7-python3.13-llm-ollama-0.12.0

Let's query the cache for the narinfo file of this build:

$ curl https://cache.nixos.org/clz5dsk4cxsakbw9zw634riwhpjwhjk7.narinfo

 StorePath: /nix/store/clz5dsk4cxsakbw9zw634riwhpjwhjk7-python3.13-llm-ollama-0.12.0
 URL: nar/0xpnlv3q605ljcid91kn72awipjh58x9f3iaqrj1d662q9m3gvh3.nar.xz
 Compression: xz
 FileHash: sha256:0xpnlv3q605ljcid91kn72awipjh58x9f3iaqrj1d662q9m3gvh3
 FileSize: 22620
 NarHash: sha256:10g4imc7s0yhlx042ly44pfr98z1kjc5j3g7p74n1ani0zp36hb0
 NarSize: 102152
 References: 9yh9ak97gn659bk4d3n411fx6c0ng7s2-python3-3.13.5 clz5dsk4cxsakbw9zw634riwhpjwhjk7-python3.13-llm-ollama-0.12.0 dz5gpk4mxg7hhyxcnb5jg6696an7hlrn-python3.13-ollama-0.5.1 r48mlzgpdzj04lyq17hbn4nw5y85wnzl-python3.13-llm-0.26 wa8b4x0vsvxz1dvgwnm9gdwhvba7ka70-python3.13-pydantic-2.11.7 zbzpxnsm8j0rqakmdvsrf36zh28xick2-python3.13-click-8.1.8
 Deriver: 06ni1ca50ndxh2pgxy0kznbyv07pxmyr-python3.13-llm-ollama-0.12.0.drv
 Sig: cache.nixos.org-1:5FnqydQlPAWi9bVeDoN2MVxzHiez0CV43SIpjdKIo0SEwCmNXoiTq+snEOYwGHOfI/OEqtaMXu2kcHqR4kEaCw==

The signing and uploading to the cache behaviour is configured in nixos/infra.

My understanding is that builds of Hydra started getting signed in June of 2015. Moreover, the key has likely never been rotated before! There is an ongoing effort to support multiple key signatures at the same time and rotate the old key.

I don't mind Perl that much, but I think Hydra shows its age since it's littered with practices that today are considered antipatterns. A simple example, here's a remote endpoint that shells out to git. If that die() line that validates the two revs was missing, it would be a command injection vector. This pattern can be seen all over Hydra.

It took me a couple of hours to find a (minor?) security issue in Hydra when I started looking. I reported it upstream, but it was already reported by someone else. I think it will be out of embargo towards the end of August.