← Home

Setting up a NixOS remote builder for the M1 Mac

If you want to build a Virtual Machine disk image, or Docker container using Nix on an M1 Mac, you’ll need to use a remote builder that’s running Linux.

Virtual machines

We’ll set up two Linux machines, one for ARM64, and another for x86_64, using UTM.

ISO

Download the Minimal NixOS ISO for both ARM64 and x86_64 from [0].

Usage in UTM

Create a new virtual machine in UTM.

  • For the ARM64 machine, select “Virtualize” since the ARM64 VM will be running on the M1 Mac, which is also ARM64.
  • For the x86_64 machine, select “Emulate”. This allows the x86_64 VM to run on the ARM64 Mac.

Next, choose “Other” as the operating system, so that you can select the downloaded ISO files.

Networking

Customise the networking to use “Bridged” mode, so that the VMs can be accessed from the host machine [1].

Note that the interface is explicitly set to en0 on the Mac, which is the WiFi interface. If you’re using Ethernet, you may need to change this to en1.

Installation

When NixOS starts, you’ll be dropped into a shell, but the VM disk won’t be mounted.

You can see the VM disk by running lsblk.

For some reason, in UTM, the disk is /dev/vda for aarch64, while it’s /dev/sda for the emulated x86_64 machine.

NAME  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0   7:0    0 899.1M  1 loop /nix/.ro-store
sr0    11:0    1 984.9M  0 rom  /iso
vda   254:0    0  128G  0 disk

The commands all need to run as root, so run sudo -i to become root.

Use cfdisk to partition the disk. Use /dev/vda for the ARM64 machine, and /dev/sda for the x86_64 machine.

cfdisk /dev/vda

Create EFI partition

  • Choose gpt.
  • Create a new partition.
  • Give it a size of 512M.
  • Change the type to EFI System.

Create swap partition

  • Select “Free Space”.
  • Create a new partition.
  • Give it a size of 5G (1GB more than the system RAM of 4GB).
  • Change the type to Linux swap.

Create root partition

  • Select “Free Space”.
  • Create a new partition.
  • Use the remaining space.
  • The filesystem will already be set to Linux filesystem, so we don’t need to change it.

Results

Select “Write” to write the changes, then quit.

Create filesystems

Create the filesystems for the partitions. Remember to use /dev/vda for the ARM64 machine, and /dev/sda for the x86_64 machine.

mkfs.fat -F 32 -n boot /dev/vda1
mkswap -L swap /dev/vda2
mkfs.ext4 -L nixos /dev/vda3

Mount the filesystems

mount /dev/vda3 /mnt
mkdir -p /mnt/boot
mount /dev/vda1 /mnt/boot

Enable the swap.

swapon /dev/vda2

Create a root SSH user

An SSH key and username are set up in my NixOS configuration in Github, but note that I have two SSH keys, one for my normal user, and one for root, which actually carries out Nix build operations on my Mac.

To generate the SSH key for the root user on your Mac, use ssh-keygen as root, then get the public key to add to your NixOS configuration.

sudo ssh-keygen

The public key is in /var/root/.ssh/id_ed25519.pub, cat that file to get the public key.

Download and build configuration

The configuration at https://github.com/a-h/nixos includes a flake that can be used to build a NixOS system for aarch64 and x86_64.

Although the machines have different mount points for disks, the NixOS configuration looks up the disk by label, so the same configuration can be used for both machines.

fileSystems."/" = {
  device = "/dev/disk/by-label/nixos";
  fsType = "ext4";
};
fileSystems."/boot" = {
  device = "/dev/disk/by-label/boot";
  fsType = "vfat";
};
swapDevices = [
  {
    device = "/dev/disk/by-label/swap";
  }
];

To build the config for aarch64, run the following commands, replace builder-aarch64 with builder-x86_64 for the x86_64 machine.

curl -LO https://github.com/a-h/nixos/archive/refs/heads/main.zip
unzip ./main.zip
cd nixos-main && nixos-install --root /mnt --flake .#builder-aarch64

Shutdown

shutdown now

Remove the ISO, and restart

Remove the boot ISO from the VM, and restart the VM.

SSH into the machines

To find the IP address of the virtual machines, run ip a on them.

I set up a DNS entry in /etc/hosts on the Mac, so that I can easily SSH into the machines.

192.168.0.62 builder-aarch64
192.168.0.63 builder-x86_64

Then I can SSH into the machines using a hostname.

ssh adrian@builder-aarch64

Since the username on my Mac and the VMs is the same, I don’t need to specify the username.

ssh builder-aarch64

Using the remote builder from MacOS

The documentation at [2] explains how to set up a remote builder.

nix store info --store ssh://builder-aarch64

To ensure that Linux builds are done on the corresponding VM for the architecture, set up the builders in the nix.conf file. On my machine it’s in ~/.config/nix/nix.conf.

builders = ssh://adrian@builder-x86_64 x86_64-linux ; ssh://adrian@builder-aarch64 aarch64-darwin

Remember that the build will be done by your Nix daemon on your local machine.

If you see:

1 146 /Users/adrian % nix build --impure \
  --expr '(with import <nixpkgs> { system = "x86_64-linux"; }; runCommand "foo" {} "uname > $out")'
cannot build on 'ssh://builder-x86_64': error: failed to start SSH connection to 'builder-x86_64': Host key verification failed.
Failed to find a machine for remote build!
derivation: y1i89c0934r6v492a12px1p5h44h7481-foo.drv
required (system, features): (x86_64-linux, [])
2 available machines:
(systems, maxjobs, supportedFeatures, mandatoryFeatures)
([x86_64-linux], 1, [], [])
([aarch64-darwin], 1, [], [])
error: a 'x86_64-linux' with features {} is required to build '/nix/store/y1i89c0934r6v492a12px1p5h44h7481-foo.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test}

Note, the Host key verification failed. message.

To get the host keys into the root user, copy the values from your local user’s known hosts file (~/.ssh/known_hosts) to the root user’s known hosts file (/var/root/.ssh/known_hosts), or run sudo ssh adrian@builder-x86_64 to manually add each one.

Then run the build command again.

You may run into the issue that the root user on your machine doesn’t have the same SSH keys as your user. Either create an SSH key for the root user on your Mac (as described earlier), and update the remote builder to support the use of that key, or copy the SSH keys from your normal user to the Mac root user.

sudo cp -r ~/.ssh /root

Testing

To test that the remote builder is working, run the following command.

nix build --impure \
  --expr '(with import <nixpkgs> { system = "x86_64-linux"; }; runCommand "foo" {} "uname > $out")'

The ./result file should contain Linux.

Similarly, for the ARM64 machine.

nix build --impure \
  --expr '(with import <nixpkgs> { system = "aarch64-linux"; }; runCommand "foo" {} "uname > $out")'