I believe the main selling point of this concept of opt-in state is that it makes it dead simple to keep track of ephemeral machine state (everything not explicitly specified by your NixOS configuration) and enforces elimination of Configuration Drift. While the benefits of this are clear for servers, this also works pretty well with workstations and laptops, where you gradually accumulate junk in /etc and /var which you never can be completely confident in deleting.¹

Here are some notes on how to reproduce the setup with an encrypted² btrfs root, along with a few tips for a nicer laptop experience. The instructions for encrypted btrfs root are heavily based on this blog post.

Making a Live USB

The laptop I’m currently using is a Dell XPS-13 2-in-1 (7390) with a fair number of issues running on Linux, some of which interferes with boot. Fortunately, most of these have been fixed in newer kernels, but the default installation ISO ships an older kernel version, so we need a custom ISO. Building an ISO with a custom configuration for NixOS is shockingly simple; following the instructions on the wiki:

# iso.nix
{ config, pkgs, ... }:
{
  imports = [
    # installation-cd-graphical-plasma5-new-kernel.nix uses pkgs.linuxPackages_latest
    # instead of the default kernel.
    <nixpkgs/nixos/modules/installer/cd-dvd/installation-cd-graphical-plasma5-new-kernel.nix>
    <nixpkgs/nixos/modules/installer/cd-dvd/channel.nix>
  ];

  hardware.enableAllFirmware = true;
  nixpkgs.config.allowUnfree = true;

  environment.systemPackages = with pkgs; [
    wget
    vim
    git
    tmux
    gparted
    nix-prefetch-scripts
  ];
}

The image can be built with

nix-build '<nixpkgs/nixos>' -A config.system.build.isoImage -I nixos-config=iso.nix

Then, we write the ISO to a USB stick like so:

sudo dd if=./result/iso/nixos-20.03....-x86_64-linux.iso of=/dev/<usb device> bs=1M status=progress

NixOS Installation

Once we’ve booted into a graphical session, we need to partition the disk. We’ll refer to the whole disk as $DISK (/dev/nvme0n1 in my case), and we need three partitions. The EFI partition, swap³, and the rest of the disk for btrfs to use, which we’ll respecively refer to as "$DISK"p1, "$DISK"p2, and "$DISK"p3.

Btrfs doesn’t natively support encryption, so we’ll be using dm-crypt to transparently encrypt the partition, which would be available at /dev/mapper/enc after running these commands:

cryptsetup --verify-passphrase -v luksFormat "$DISK"p3
cryptsetup open "$DISK"p3 enc

We can then format each partition as needed:

mkfs.vfat -n boot "$DISK"p1
mkswap "$DISK"p2
swapon "$DISK"p2
mkfs.btrfs /dev/mapper/enc

Now we have a btrfs volume, we need to decide on how to structure our subvolumes. We want to split our data into a number of subvolumes to keep track of a few things:

• root: The subvolume for /, which will be cleared on every boot.
• home: The subvolume for /home, which should be backed up.
• nix: The subvolume for /nix, which needs to be persistent but is not worth backing up, as it’s trivial to reconstruct.
• persist: The subvolume for /persist, containing system state which should be persistent across reboots and possibly backed up.
• log: The subvolume for /var/log. I’m not so interested in backing up logs but I want them to be preserved across reboots, so I’m dedicating a subvolume to logs rather than using the persist subvolume.

Somewhat arbitrarily, we’ll go with the “Flat” layout as described in the btrfs wiki, and create our subvolumes accordingly.

mount -t btrfs /dev/mapper/enc /mnt

# We first create the subvolumes outlined above:
btrfs subvolume create /mnt/root
btrfs subvolume create /mnt/home
btrfs subvolume create /mnt/nix
btrfs subvolume create /mnt/persist
btrfs subvolume create /mnt/log

# We then take an empty *readonly* snapshot of the root subvolume,
# which we'll eventually rollback to on every boot.
btrfs subvolume snapshot -r /mnt/root /mnt/root-blank

umount /mnt

Once we’ve created the subvolumes, we mount them with the options that we want. Here, we’re using Zstandard compression along with the noatime option.

mount -o subvol=root,compress=zstd,noatime /dev/mapper/enc /mnt

mkdir /mnt/home
mount -o subvol=home,compress=zstd,noatime /dev/mapper/enc /mnt/home

mkdir /mnt/nix
mount -o subvol=nix,compress=zstd,noatime /dev/mapper/enc /mnt/nix

mkdir /mnt/persist
mount -o subvol=persist,compress=zstd,noatime /dev/mapper/enc /mnt/persist

mkdir -p /mnt/var/log
mount -o subvol=log,compress=zstd,noatime /dev/mapper/enc /mnt/var/log

# don't forget this!
mkdir /mnt/boot
mount "$DISK"p1 /mnt/boot

Then, let NixOS figure out the config.

nixos-generate-config --root /mnt

This should result with /mnt/etc/nixos/hardware-configuration.nix looking something like this:

# Do not modify this file!  It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations.  Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, ... }:

{
  imports =
    [ <nixpkgs/nixos/modules/installer/scan/not-detected.nix>
    ];

  boot.initrd.availableKernelModules = [ "xhci_pci" "nvme" "usb_storage" "sd_mod" "rtsx_pci_sdmmc" ];
  boot.initrd.kernelModules = [ ];
  boot.kernelModules = [ "kvm-intel" ];
  boot.extraModulePackages = [ ];

  fileSystems."/" =
    { device = "/dev/disk/by-uuid/f73c53b7-ae6c-4240-89c3-511ad918edcc";
      fsType = "btrfs";
      options = [ "subvol=root" "compress=zstd" "noatime" ];
    };

  boot.initrd.luks.devices."enc".device = "/dev/disk/by-uuid/050db9bf-0741-4150-8cf8-d6ec12735d4c";

  fileSystems."/home" =
    { device = "/dev/disk/by-uuid/f73c53b7-ae6c-4240-89c3-511ad918edcc";
      fsType = "btrfs";
      options = [ "subvol=home" "compress=zstd" "noatime" ];
    };

  fileSystems."/nix" =
    { device = "/dev/disk/by-uuid/f73c53b7-ae6c-4240-89c3-511ad918edcc";
      fsType = "btrfs";
      options = [ "subvol=nix" "compress=zstd" "noatime" ];
    };

  fileSystems."/var/log" =
    { device = "/dev/disk/by-uuid/f73c53b7-ae6c-4240-89c3-511ad918edcc";
      fsType = "btrfs";
      options = [ "subvol=log" "compress=zstd" "noatime" ];
    };

  fileSystems."/persist" =
    { device = "/dev/disk/by-uuid/f73c53b7-ae6c-4240-89c3-511ad918edcc";
      fsType = "btrfs";
      options = [ "subvol=persist" "compress=zstd" "noatime" ];
    };

  fileSystems."/boot" =
    { device = "/dev/disk/by-uuid/8CE7-3C76";
      fsType = "vfat";
    };

  swapDevices =
    [ { device = "/dev/disk/by-uuid/5b1b6659-14ab-497f-a788-5518c25e7ec8"; }
    ];

  nix.maxJobs = lib.mkDefault 8;
  powerManagement.cpuFreqGovernor = lib.mkDefault "powersave";
  # High-DPI console
  console.font = lib.mkDefault "${pkgs.terminus_font}/share/consolefonts/ter-u28n.psf.gz";
}

Make sure that this is what you want, and adjust options as necessary. Note that in order to correctly persist /var/log, the log subvolume needs to be mounted early enough in the boot process. To do this, we need to add neededForBoot = true; so the entry will look like this:

  fileSystems."/var/log" =
    { device = "/dev/disk/by-uuid/f73c53b7-ae6c-4240-89c3-511ad918edcc";
      fsType = "btrfs";
      options = [ "subvol=log" "compress=zstd" "noatime" ];
      neededForBoot = true;
    };

Although it’s possible to customize /mnt/etc/nixos/configuration.nix at this point to set up all the things you need in one fell swoop, I recommend starting out with a reletively minimal config to make sure everything works ok. I went with something like this, with a user called delta:

{ config, pkgs, ... }:
{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];

  boot.kernelPackages = pkgs.linuxPackages_latest;
  boot.supportedFilesystems = [ "btrfs" ];
  hardware.enableAllFirmware = true;
  nixpkgs.config.allowUnfree = true;

  # Use the systemd-boot EFI boot loader.
  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;

  networking.hostName = "apollo"; # Define your hostname.
  networking.networkmanager.enable = true;

  # Enable the X11 windowing system.
  services.xserver.enable = true;

  # Enable the KDE Desktop Environment.
  services.xserver.displayManager.sddm.enable = true;
  services.xserver.desktopManager.plasma5.enable = true;

  # Define a user account. Don't forget to set a password with ‘passwd’.
  users.users.delta = {
    isNormalUser = true;
    extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
  };

  system.stateVersion = "20.03";
}

Take a deep breath.

nixos-install
reboot

If all goes well, we’ll be prompted for the passphrase for $DISK entered earlier, then we’ll see the greeter for the KDE Desktop Environment. Swith to another tty with Ctrl+Alt+F1, login as root, passwd delta to set your password, and switch back to KDE with Ctrl+Alt+F7. Once you’re logged in, you can continue to tweak your NixOS configuration as you want. However, I generally recommend keeping enabled services at a minimum, and setting up opt-in state first.

Darling Erasure

Now that we’re comfortable in our desktop environment of choice (mine is XMonad), we can move onto the opt-in state setup. First, we need to find out what state exists in the first place. Seeing what has changed since we took the blank snapshot seems like a good way to do this.

Taking a diff between the root subvolume and the root-blank subvolume (in btrfs, snapshots are just subvolumes) can be done with a script based off of the answers to this serverfault question.

#!/usr/bin/env bash
# fs-diff.sh
set -euo pipefail

OLD_TRANSID=$(sudo btrfs subvolume find-new /mnt/root-blank 9999999)
OLD_TRANSID=${OLD_TRANSID#transid marker was }

sudo btrfs subvolume find-new "/mnt/root" "$OLD_TRANSID" |
sed '$d' |
cut -f17- -d' ' |
sort |
uniq |
while read path; do
  path="/$path"
  if [ -L "$path" ]; then
    : # The path is a symbolic link, so is probably handled by NixOS already
  elif [ -d "$path" ]; then
    : # The path is a directory, ignore
  else
    echo "$path"
  fi
done

Then, all it takes to find out which files now exist in the root subvolume is:

sudo mkdir /mnt
sudo mount -o subvol=/ /dev/mapper/enc /mnt
./fs-diff.sh

This may show a surprisingly small list of files, or possible something fairly lengthy, depending on your configuration. We’ll first tackle NetworkManager, so we don’t have to re-type passwords to Wi-Fi access points after every reboot. While grahamc’s original blog post suggests that simply persisting /etc/NetworkManager/system-connections by moving it to somewhere in /persist and creating a symlink is enough, this was not enough to get it to work on my XMonad setup. I ended up with something like this, symlinking a few files in /var/lib/NetworkManager as well.

  environment.etc = {
    "NetworkManager/system-connections".source = "/persist/etc/NetworkManager/system-connections";
  };
  systemd.tmpfiles.rules = [
    "L /var/lib/NetworkManager/secret_key - - - - /persist/var/lib/NetworkManager/secret_key"
    "L /var/lib/NetworkManager/seen-bssids - - - - /persist/var/lib/NetworkManager/seen-bssids"
    "L /var/lib/NetworkManager/timestamps - - - - /persist/var/lib/NetworkManager/timestamps"
  ];

Now you might have noticed that the NixOS configuration itself lives in /etc/nixos/, which will be deleted if left there. After adding a few things, I ended up with a configuration like this.

 environment.etc = {
    nixos.source = "/persist/etc/nixos";
    "NetworkManager/system-connections".source = "/persist/etc/NetworkManager/system-connections";
    adjtime.source = "/persist/etc/adjtime";
    NIXOS.source = "/persist/etc/NIXOS";
    machine-id.source = "/persist/etc/machine-id";
  };
  systemd.tmpfiles.rules = [
    "L /var/lib/NetworkManager/secret_key - - - - /persist/var/lib/NetworkManager/secret_key"
    "L /var/lib/NetworkManager/seen-bssids - - - - /persist/var/lib/NetworkManager/seen-bssids"
    "L /var/lib/NetworkManager/timestamps - - - - /persist/var/lib/NetworkManager/timestamps"
  ];
  security.sudo.extraConfig = ''
    # rollback results in sudo lectures after each reboot
    Defaults lecture = never
  '';

Rolling back the root subvolume is a little bit involved when compared to zfs, but can be achieved with this config.

  # Note `lib.mkBefore` is used instead of `lib.mkAfter` here.
  boot.initrd.postDeviceCommands = pkgs.lib.mkBefore ''
    mkdir -p /mnt

    # We first mount the btrfs root to /mnt
    # so we can manipulate btrfs subvolumes.
    mount -o subvol=/ /dev/mapper/enc /mnt

    # While we're tempted to just delete /root and create
    # a new snapshot from /root-blank, /root is already
    # populated at this point with a number of subvolumes,
    # which makes `btrfs subvolume delete` fail.
    # So, we remove them first.
    #
    # /root contains subvolumes:
    # - /root/var/lib/portables
    # - /root/var/lib/machines
    #
    # I suspect these are related to systemd-nspawn, but
    # since I don't use it I'm not 100% sure.
    # Anyhow, deleting these subvolumes hasn't resulted
    # in any issues so far, except for fairly
    # benign-looking errors from systemd-tmpfiles.
    btrfs subvolume list -o /mnt/root |
    cut -f9 -d' ' |
    while read subvolume; do
      echo "deleting /$subvolume subvolume..."
      btrfs subvolume delete "/mnt/$subvolume"
    done &&
    echo "deleting /root subvolume..." &&
    btrfs subvolume delete /mnt/root

    echo "restoring blank /root subvolume..."
    btrfs subvolume snapshot /mnt/root-blank /mnt/root

    # Once we're done rolling back to a blank snapshot,
    # we can unmount /mnt and continue on the boot process.
    umount /mnt
  '';

While NixOS will take care of creating the specified symlinks, we need to move the relevant file and directories to where the symlinks are pointing at after running sudo nixos-rebuild boot and before rebooting.

sudo nixos-rebuild boot

sudo mkdir -p /persist/etc/NetworkManager
sudo cp -r {,/persist}/etc/NetworkManager/system-connections
sudo mkdir -p /persist/var/lib/NetworkManager
sudo cp /var/lib/NetworkManager/{secret_key,seen-bssids,timestamps} /persist/var/lib/NetworkManager/

sudo cp {,/persist}/etc/nixos
sudo cp {,/persist}/etc/adjtime
sudo cp {,/persist}/etc/NIXOS

Before rebooting, make sure that your user credentials are appropriately handled. Be especially careful⁴ when setting users.mutableUsers to false and using users.extraUsers.<name?>.passwordFile, as these settings are some of the few in NixOS which can lock you out across NixOS configurations and require non-trivial recovery work or a reinstall. If you want declerative user management, I recommend using users.extraUsers.<name?>.hashedPasswords, but this has it’s own downsides as well.⁵

Take another deep breath.

reboot

If something goes wrong and /mnt/root isn’t deleted, btrfs subvolume snapshot /mnt/root-blank /mnt/root will just create a snapshot under /mnt/root, so a quick hack to check if rolling back failed without consulting journalctl -b is to see if /mnt/root/root-blank exists.⁶

Adding NixOS Services Case Study (Docker and LXD)

As much as Nix and NixOS are attractive for everyday use, sometimes the time it takes to get some language or package running on NixOS just doesn’t seem worth it. That’s when container runtimes like Docker and LXD can help. These tools can act as an escape hatch to get some software working quickly on your machine.

Here, we’ll go through the workflow for getting NixOS services to work with opt-in state, with Docker and LXD as examples.

First, let’s get Docker and LXD running to inspect what kind of state they have. Thanks to NixOS, this is a just a few lines of configuration.

  virtualisation = {
     docker.enable = true;
    lxd = {
      enable = true;
      recommendedSysctlSettings = true;
    };
  };

sudo nixos-rebuild switch

This will install, set up, and start both Docker and LXD on our machine. With fs-diff.sh we can see a few relevant files and directories show up.

/etc/docker/key.json
...
/var/lib/docker/...
/var/lib/lxd/...

Some quick googling tells us that /etc/docker/key.json is generated on every boot, so it seems like we don’t need to keep this around. On the other hand, /var/lib/docker and /var/lib/lxd seem important, so let’s adjust our config accordingly.

  systemd.tmpfiles.rules = [
    "L /var/lib/NetworkManager/secret_key - - - - /persist/var/lib/NetworkManager/secret_key"
    "L /var/lib/NetworkManager/seen-bssids - - - - /persist/var/lib/NetworkManager/seen-bssids"
    "L /var/lib/NetworkManager/timestamps - - - - /persist/var/lib/NetworkManager/timestamps"
    "L /var/lib/lxd - - - - /persist/var/lib/lxd"
    "L /var/lib/docker - - - - /persist/var/lib/docker"
  ];

Now, stop the two services and copy over the directories.

sudo mkdir -p /persist/var/lib/

sudo systemctl stop lxd
sudo cp -r {,/persist}/var/lib/lxd

sudo systemctl stop docker
sudo cp -r {,/persist}/var/lib/docker

sudo nixos-rebuild boot
reboot

If all goes well, running the fs-diff.sh after reboot shouldn’t show persisted directories /var/lib/lxd and /var/lib/docker since they should be symlinks which are created during the boot process.

Docker should work without any problems at this point, but we LXD needs some additional configuration. LXD requires a storage pool to operate, so we create a subvolume for LXD, and mount it in /persist.

sudo mount -o subvol=/ /mnt
sudo btrfs subvolume create /mnt/lxd
sudo umount /mnt
sudo mkdir /persist/lxd
sudo mount -o subvol=lxd /dev/mapper/enc /persist/lxd

Once the subvolume is ready, we run lxd init and answer the questions in the following manner.

$ lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: no
Do you want to configure a new storage pool? (yes/no) [default=yes]:
Name of the new storage pool [default=default]:
Name of the storage backend to use (btrfs, dir, lvm) [default=btrfs]:
Would you like to create a new btrfs subvolume under /var/lib/lxd? (yes/no) [default=yes]: no
Create a new BTRFS pool? (yes/no) [default=yes]: no
Name of the existing BTRFS pool or dataset: /persist/lxd
Would you like to connect to a MAAS server? (yes/no) [default=no]:
Would you like to create a new local network bridge? (yes/no) [default=yes]:
What should the new bridge be called? [default=lxdbr0]:
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
Would you like LXD to be available over the network? (yes/no) [default=no]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:

Remember to add the relevant information to /etc/nixos/hardware-configuration.nix so NixOS will mount the subvolume where LXD expects (i.e. /persist/lxd).

  fileSystems."/persist/lxd" =
    { device = "/dev/disk/by-uuid/f73c53b7-ae6c-4240-89c3-511ad918edcc";
      fsType = "btrfs";
      options = [ "subvol=lxd" "compress=zstd" "noatime" ];
    };

EDIT 2020-01-26: Added persistence for /etc/machine-id, which fixes an issue where journalctl fails to find logs from past boots, among various others. Thanks j-hui for pointing this out!

Thanks to cannorin and __pandaman64__ for comments and suggestions.

However, finding out the convoluted installation instructions for software on Ubuntu and turning it into a provisioning script is always a bit of a pain, so I decided to use NixOS this time to quickly try it out. The requirements were fairly involved, though.

• The service needs to run containerized on LXD
• Only the users in the admin LDAP group should be able to
  • ssh in, with automatic home directory creation
  • access a web interface via HTTPS¹

I’ll go through the steps I took to meet each requirement, one by one.

Setting up a NixOS LXD container

(If you’re not interested in LXD, you can just skip to LDAP Authentication)

Thanks to nix-community/nixos-generators, this is pretty straightforward. First, we want to start off with a configuration that’s similar to that found in Ubuntu LXD images:

# configuration.nix
{ config, pkgs, ... }:
{
  # https://github.com/NixOS/nixpkgs/issues/9735#issuecomment-500164017
  systemd.services."console-getty".enable = false;
  systemd.services."getty@".enable = false;

  imports = [
    <nixpkgs/nixos/modules/virtualisation/lxc-container.nix>
  ];
  networking.hostName = "nixos";

  services.openssh = {
    enable = true;
    permitRootLogin = "no";
  };

  users.users.nixos = {
    isNormalUser = true;
    extraGroups = [ "wheel" ];
  };

  security.sudo.wheelNeedsPassword = false;

  environment.systemPackages = with pkgs; [
    vim
    htop
    tmux
    wget
  ];

  networking.useDHCP = true;
}

With this, we can create an LXD image with the following command:

nix-shell -p nixos-generators --run \
  'lxc image import --alias nixos \
    $(nixos-generate --format lxc-metadata --configuration ./configuration.nix) \
    $(nixos-generate --format lxc --configuration ./configuration.nix)'

Now that we’ve created an image under the alias nixos, we can launch a NixOS container with lxc launch nixos nixos-test -c security.nesting=true.

Once an alias is defined like this:

lxc alias add nixos 'exec @ARGS@ --mode interactive -- /run/current-system/sw/bin/login -p -f nixos'

Logging into a running NixOS container is as simple as lxc nixos nixos-test. We can now fill in /etc/nixos/configuration.nix and sudo nixos-rebuild switch as needed.

LDAP Authentication

Setting up LDAP authentication was… not as straightforward as I’d hoped.

First Attempt

Let’s forget about group authorization for a moment, and try to get LDAP-based ssh logins working.

{ lib, config, pkgs, ... }:
{
  users.ldap = {
    enable = true;
    base = "dc=example,dc=com";
    server = "ldap://example.com/";
    useTLS = true;
    extraConfig = ''
      ldap_version 3
      pam_password md5
    '';
  };

  security.pam.services.sshd.makeHomeDir = true;
}

Note that the LDAP server we are using is configured to allow anonymous binding and authentication, which may not fit your use-case.

Unfortunately, this doesn’t work; attempting to login as a LDAP user gives errors like the following:

Jul 24 10:51:23 nixos sshd[11992]: Postponed keyboard-interactive for invalid user nixos-user from a.b.c.d port 59828 ssh2 [preauth]
Jul 24 10:51:25 nixos sshd[11994]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=a.b.c.d  user=nixos-user
Jul 24 10:51:25 nixos sshd[11994]: pam_ldap: error trying to bind as user "uid=nixos-user,ou=Users,dc=example,dc=com" (Invalid credentials)
Jul 24 10:51:26 nixos sshd[11992]: error: PAM: Authentication failure for illegal user nixos-user from a.b.c.d

This is frustratingly misleading since the real culprit is this, slightly preceding, message from sshd:

Jul 24 10:51:23 nixos sshd[11992]: User nixos-user not allowed because shell /bin/bash does not exist

This occurs because in our LDAP server, we had set users’ LDAP loginShell attribute to /bin/bash. In hindsight, I don’t think this was such a bad a thing to do, considering that I’ve never dealt with a single system in which /bin/bash doesn’t exist (with the exception of NixOS, of course :smile: ).

Second Attempt

The standard approach to solving this problem is configuring the LDAP client to override the relevant attribute:

...
{
  users.ldap = {
    ...
    extraConfig = ''
      ...

      nss_override_attribute_value loginShell /run/current-system/sw/bin/bash
    '';
  };
  ...
}

Unfortunately, this doesn’t seem to solve the issue, and sshd continues to complain about /bin/bash missing.²

This seems to leave us with changing the loginShell attributes to /bin/sh, since that’s the only way to accommodate NixOS systems while maintaining compatibility with the non-NixOS systems also using LDAP. But /bin/sh isn’t a nice shell to work in, so do we really want to force this on everyone across all machines? Well, there is one easy trick to make this all go away; just symlink /bin/bash to /run/current-system/sw/bin/bash :gasp:.

{ lib, config, pkgs, ... }:
{
  users.ldap = {
    enable = true;
    base = "dc=example,dc=com";
    server = "ldap://example.com/";
    useTLS = true;
    extraConfig = ''
      ldap_version 3
      pam_password md5

      # TOFIX: this does not work for some reason
      # # https://serverfault.com/a/137996
      # nss_override_attribute_value loginShell /run/current-system/sw/bin/bash
    '';
  };

  # evil, horrifying hack for dysfunctional nss_override_attribute_value
  systemd.tmpfiles.rules = [
    "L /bin/bash - - - - /run/current-system/sw/bin/bash"
  ];
}

Group-based Authorization and Home Directory Creation

LDAP group-based authorization was also not as straightforward as I’d hoped. Adding pam_groupdn cn=admin,ou=Groups,dc=example,dc=com to users.ldap.extraConfig only seems to work when users solely belong to the admin group, which was not the case for us. Instead, using pam_listfile.so got us what we wanted:

...
{
  ...
  security.pam.services.sshd = {
    makeHomeDir = true;

    # see https://stackoverflow.com/a/47041843 for why this is required
    text = lib.mkDefault (
      lib.mkBefore ''
        auth required pam_listfile.so \
          item=group sense=allow onerr=fail file=/etc/allowed_groups
      ''
    );
  };

  environment.etc.allowed_groups = {
    text = "admins";
    mode = "0444";
  };
  ...
}

LDAP Authentication for Nginx

This is actually just a variation on the example for PAM authentication for Nginx on the NixOS Wiki used along with pam_listfile.so:

...
{
  ...
  services.nginx = {
    enable = true;
    package = (pkgs.nginx.override { modules = [ pkgs.nginxModules.pam ]; });
    virtualHosts."www.example.com" = {
      enableACME = true;
      forceSSL = true;
      ...
      extraConfig = ''
        auth_pam "LDAP Authentication Required";
        auth_pam_service_name "nginx";
      '';
    };
  };
  security.pam.services.nginx.text = ''
    auth    required     pam_listfile.so \
                         item=group sense=allow onerr=fail file=/etc/allowed_groups
    auth    required     ${pkgs.pam_ldap}/lib/security/pam_ldap.so
    account required     ${pkgs.pam_ldap}/lib/security/pam_ldap.so
  '';
  ...
}

With this configuration, accesses to www.example.com will be authenticated with HTTP Basic Authentication backed by the same LDAP group-based policy.

Wrapping Up

After working through a fairly involved example of deploying NixOS into conventional infrastructure, I’m left with the impression that there’s a lot more we can do to make the experience working with mundane things like LDAP much better. I would love to see NixOS becoming the sysadmin’s favorite distribution, as it definitely has the potential to become so.

Thanks to __pandaman64__ for comments and suggestions.

The tutorial is a step-by-step guide on how to create a Jepsen test for etcd, a popular key-value based on the the Raft consensus algorithm. Since we’re testing distributed systems, we need multiple systems running, and Jepsen defaults to requiring five machines accessible in a certain way. Getting the configuration exactly right took some time to figure out, so I’ve created a NixOS configuration module that you can import which will take care of setting up etcd in NixOS containers correctly:

{ config, lib, pkgs, ... }:
let
  unstable = import ./unstable.nix;
  addressMap =
    {
      "n1" = { localAddress = "10.233.0.101"; hostAddress = "10.233.1.101"; };
      "n2" = { localAddress = "10.233.0.102"; hostAddress = "10.233.1.102"; };
      "n3" = { localAddress = "10.233.0.103"; hostAddress = "10.233.1.103"; };
      "n4" = { localAddress = "10.233.0.104"; hostAddress = "10.233.1.104"; };
      "n5" = { localAddress = "10.233.0.105"; hostAddress = "10.233.1.105"; };
    };
  toHostsEntry = name: { localAddress, ... }: "${localAddress} ${name}";
  extraHosts =
    builtins.concatStringsSep "\n"
      (lib.attrsets.mapAttrsToList toHostsEntry addressMap);
  nodeConfig = hostName: { localAddress, hostAddress }: {
    inherit localAddress hostAddress;

    ephemeral = true;
    autoStart = true;
    privateNetwork = true;

    config = { config, pkgs, ... }:
      {
        networking = {
          inherit hostName extraHosts;
        };

        services.openssh = {
          enable = true;
          permitRootLogin = "yes";
        };
        users.users.root.initialPassword = "root";

        system.stateVersion = "20.03";

        services.etcd =
          let
            peerUrl = "http://${localAddress}:2380";
            clientUrl = "http://${localAddress}:2379";
            toClusterEntry = name: { localAddress, ... }:
              "${name}=http://${localAddress}:2380";
          in
            {
              enable = true;
              name = hostName;

              initialAdvertisePeerUrls = [ peerUrl ];
              listenPeerUrls = [ peerUrl ];

              advertiseClientUrls = [ clientUrl ];
              listenClientUrls = [ clientUrl "http://127.0.0.1:2379" ];

              initialClusterToken = "etcd-cluster";
              initialCluster =
                lib.attrsets.mapAttrsToList toClusterEntry addressMap;
              initialClusterState = "new";

              # Apparently Jepsen can't read journald logs? Unfortunate.
              extraConf.LOG_OUTPUT = "stderr";
            };

        # Workaround for nixos-container issue
        # (see https://github.com/NixOS/nixpkgs/issues/67265 and
        # https://github.com/NixOS/nixpkgs/pull/81371#issuecomment-605526099).
        # The etcd service is of type "notify", which means that
        # etcd would not be considered started until etcd is fully online;
        # however, since NixOS container networking only works sometime *after*
        # multi-user.target, we forgo etcd's notification entirely.
        systemd.services.etcd.serviceConfig.Type = lib.mkForce "exec";

        systemd.services.etcd.serviceConfig.StandardOutput = "file:/var/log/etcd.log";
        systemd.services.etcd.serviceConfig.StandardError = "file:/var/log/etcd.log";

        networking.firewall.allowedTCPPorts = [ 2379 2380 ];
      };
  };
in
{
  containers = lib.attrsets.mapAttrs nodeConfig addressMap;
  networking = {
    inherit extraHosts;
  };
}

As Jepsen doesn’t support NixOS natively, a few tweaks were required, but there were suprisingly few hiccups along the way. Thank you aphyr for the amazing work! Here’s the code that I ended up with after going through the tutorial here, and encourage you to try it out.

mt-caret/nixos-jepsen.etcdemo

I’ll probably try setting up a Jepsen test for the distributed system that we use at work next, and I’m looking forward to blogging about what I learn along the way.

My NixOS configuration currently depends on multiple channels (<nixos-20.09>, <nixos-unstable>, <nixpkgs-unstable>, and home-manager):

https://channels.nixos.org/nixos-20.09/nixexprs.tar.xz nixos
https://channels.nixos.org/nixos-unstable/nixexprs.tar.xz unstable
https://channels.nixos.org/nixpkgs-unstable/nixexprs.tar.xz actual-unstable
https://github.com/nix-community/home-manager/archive/release-20.09.tar.gz home-manager

What’s painful about this is that on a clean install the various nix channels aren’t present on my machine, and I need to manually set up the required Nix channels for the root user via the nix-channel command or by writing to /root/.nix-channels. I always forget to do this, and get a variant of the following error:

nix-repl> <foo>
error: file 'foo' was not found in the Nix search path (add it using $NIX_PATH or -I), at (string):1:1

The recommended fix I usually hear is to avoid using Nix channels entirely and pin the various nixpkgs in configuration.nix. However, I don’t want to do this. It’s painful to have to update the pin every so often, and my nixpkgs lags behind Nix channels so I get a lot of binary cache misses and end up building many packages from source, making the experience not unlike Gentoo.

On the other hand, doing builtins.fetchTarball every time also isn’t ideal. Who wants to require an Internet connect every time you need to change the font size for your decleratively configured terminal emulator? If we sacrifice just a bit of determinism (but not too much) we can do better.

In the example above, when the import path foo doesn’t exist, evaluation of <foo> results in a plain old exception which can be caught with builtins.tryEval:

let
  resolvedPath = builtins.tryEval <nixos-unstable>;
in
  if resolvedNixPath.success then
    resolvedNixPath.value
  else
    builtins.fetchTarball "https://channels.nixos.org/nixos-unstable/nixexprs.tar.xz"

By checking if evaluation of the Nix import path threw an exception or not, we can determine if a particular import <foo> succeeded or not. We can then import nixpkgs normally if the import succeeds, or fall back to fetching a tarball and importing from there. This can be further extended to automatically create .nix-channels files for root and home-manager users:

let
  resolve = nixPath: defaultUrl:
    let
      resolvedNixPath = builtins.tryEval nixPath;
    in
      if resolvedNixPath.success then
        resolvedNixPath.value
      else
        builtins.trace
          "WARNING: could not resolve path, defaulting to '${defaultUrl}'"
          (builtins.fetchTarball defaultUrl);
  channelUrl = channel: "https://channels.nixos.org/${channel}/nixexprs.tar.xz";
  version = "20.09";
  channels = [
    {
      importPath = <nixos>;
      name = "nixos";
      url = channelUrl "nixos-${version}";
    }
    {
      importPath = <unstable>;
      name = "unstable";
      url = channelUrl "nixos-unstable";
    }
    {
      importPath = <actual-unstable>;
      name = "actual-unstable";
      url = channelUrl "nixpkgs-unstable";
    }
  ];
  allNixpkgs =
    builtins.listToAttrs (
      builtins.map (
        { importPath, name, url }:
          {
            inherit name;
            value = import (resolve importPath url) {};
          }
      ) channels
    );
in
allNixpkgs
// rec {
  nix-channels = allNixpkgs.nixos.writeTextFile {
    name = ".nix-channels";
    text =
      builtins.concatStringsSep "\n"
        (builtins.map ({ name, url, ... }: "${url} ${name}") channels);
  };
  home-manager-config = { pkgs, lib, ... }: {
    home.file.".nix-channels".source = nix-channels;
  };
  nixos-config = { ... }: {
    systemd.tmpfiles.rules = [
      # unlike "L", "L+" will remove the existing file if it exists and replace
      # it a symlink
      "L+ /root/.nix-channels - - - - ${nix-channels}"
    ];
  };
}

where importing home-manager-config for home-manager and nixos-config for root users will just work.

Upgrading the NixOS version just works as well; simply bumping version and running sudo nixos-rebuild switch --upgrade twice will do it, where the first nixos-rebuild switch updates the .nix-channel file and the second nixos-rebuild switch does the actual upgrading.

I’m pretty happy that I no longer have to think (too hard) about this part of the clean install process anymore. Unfortunately, since I use VirtualBox with the extension pack enabled, every time I nixos-rebuild switch --upgrade I still get the Gentoo experience anyway :cry:.

Dune, the build system often used for OCaml projects, is pretty general, and you can use it as a Makefile-on-steroids.

Here’s a simple dune rule for building a markdown file into an html file:

(rule
 (deps build.sh 2024-11-09-static-site-generation-in-dune.md)
 (targets 2024-11-09-static-site-generation-in-dune.html)
 (action
  (run build.sh 2024-11-09-static-site-generation-in-dune.md 2024-11-09-static-site-generation-in-dune.html)))

Where build.sh can do whatever you want to turn the markdown into html. I use pandoc for this site. There’s an immediate problem with this approach. We need to write one rule for every markdown file! What we want is something like recursive Makefiles to generate rules for each markdown file, but for dune.

Fortunately, the dune docs have a tutorial on “rule generation” that walks us through doing just this.

The main dune file in our directory would look like this:

(include dune.inc)

(rule
 (deps
  (source_tree .))
 (action
  (with-stdout-to
   dune.inc.gen
   (run ../gen/gen.exe))))

(rule
 (alias default)
 (action
  (diff dune.inc dune.inc.gen)))

The idea here is that we’ll have a program-generated dune.inc file which this dune file will include. Whenever there is a change in the source directory containing markdown files, ../gen/gen.exe will be invoked to re-generate the dune.inc file, which will then result in a diff between the old and new dune.inc files due to the (diff ...) rule.

dune promote will update the generated dune.inc file to the new version, and dune build will rebuild all the markdown files. Using dune build -w @default --auto-promote will automate the promotion and watch for changes.

The source for gen.exe looks like this:

open! Core

let generate_pragma slug =
  print_endline
    [%string
      {|
(rule
 (deps build.sh %{slug}.md)
 (targets %{slug}.html)
 (action
  (run build.sh %{slug}.md %{slug}.html)))|}]
;;

let () =
  let source_files =
    Sys_unix.ls_dir "."
    |> List.sort ~compare:String.compare
    |> List.filter_map ~f:(fun file -> String.chop_suffix file ~suffix:".md")
  in
  List.iter source_files ~f:generate_pragma;
  let output_files = List.map source_files ~f:(fun file -> [%string "%{file}.html"]) in
  print_endline
    [%string
      {|
(alias
  (name default)
  (deps %{String.concat ~sep:" " output_files}))
  |}]
;;

We also want to generate an index page, so we’ll add an rule for that:

(rule
 (deps ./index.sh (source_tree .))
 (action
  (with-stdout-to
   index.html
   (run ./index.sh))))

You can see the full source for this site on GitHub.

The OCaml manual describes how to call OCaml code from C under the section “Advanced example with callbacks” in the chapter “Interfacing C with OCaml”, using the following invocations to ocamlopt:

$ ocamlopt -output-obj -o modcaml.o mod.ml
$ ocamlopt -c modwrap.c
$ cp `ocamlopt -where`/libasmrun.a mod.a && chmod +w mod.a
$ ar r mod.a modcaml.o modwrap.o
$ cc -o prog -I `ocamlopt -where` main.c mod.a -lm

Reading the section in the OCaml manual about ocamlopt command line options carefully, it looks like we can use -output-complete-obj to simplify above commands:

$ ocamlopt -output-complete-obj -o modcaml.o mod.ml
$ ocamlopt -c modwrap.c
$ cc -o prog_native2 -I `ocamlopt -where` main.c modcaml.o modwrap.o -lm

Translating to dune

We first want to recreate the modcaml.o file. Grepping in the dune codebase for -output-complete-obj, we find this code which matches on Executables.Link_mode.t.

The dune docs mention how to specify linking modes for executables, so we’ll try that.

(executables
 (names mod)
 (modes object))

After some trial and error, I’ve figured out that the target corresponding to this rule is mod.exe.o:

$ dune build --verbose mod.exe.o
...
Actual targets:
- _build/default/mod.exe.o
Running[1]: (cd _build/default && /.../.opam/5.2.1/bin/ocamlc.opt -w @1..3@5..28@31..39@43@46..47@49..57@61..62@67@69-40 -strict-sequence -strict-formats -short-paths -keep-locs -g -bin-annot -bin-annot-occurrences -I .mod.eobjs/byte -no-alias-deps -opaque -o .mod.eobjs/byte/dune__exe__Mod.cmi -c -intf mod.mli)
Running[2]: (cd _build/default && /.../.opam/5.2.1/bin/ocamlopt.opt -w @1..3@5..28@31..39@43@46..47@49..57@61..62@67@69-40 -strict-sequence -strict-formats -short-paths -keep-locs -g -I .mod.eobjs/byte -I .mod.eobjs/native -intf-suffix .ml -no-alias-deps -opaque -o .mod.eobjs/native/dune__exe__Mod.cmx -c -impl mod.ml)
Running[3]: (cd _build/default && /.../.opam/5.2.1/bin/ocamlopt.opt -w @1..3@5..28@31..39@43@46..47@49..57@61..62@67@69-40 -strict-sequence -strict-formats -short-paths -keep-locs -g -o mod.exe.o -output-complete-obj .mod.eobjs/native/dune__exe__Mod.cmx)

I’m not exactly sure if there are better ways to do this, but we manually define rules to run the remaining steps:

(rule
 (targets modwrap.o)
 (deps modwrap.c)
 (action
  (run ocamlopt %{deps})))

(rule
 (targets main.exe)
 (deps main.c mod.exe.o modwrap.o)
 (action
  (run %{cc} -o %{targets} -I %{ocaml_where} %{deps} -lm)))

And voila!

$ dune exec ./main.exe
fib(10) = Result is: 89            

The full code is available in the calling-ocaml-from-c repository.