Tony Finn

Nix from First Principles: Flake Edition

Nix is a tool for managing packages and system configuration that has seen a lot more interest in the last few years. It's one I became interested in in early 2022. Nix in 2022 is at an inflection point - a new generation of tooling is widely adopted among the userbase and considered to be easier to use, but the official documentation still largely explains how to do it in the traditional style.

This guide is a beginner's guide to Nix and related tooling, focusing on the newer nix command, and flake.nix compared to older tools like nix-env and default.nix. It does not require any prior Nix knowledge, and instead builds up the Flake based world from first principles, so that it can serve as an introduction to Nix itself, as well as the concept and uses of Flakes.

Overview (Nix From First Principles: Flake Edition #1)

#Nix

This is part 1 of the Nix from First Principles: Flake Edition series.

One technology that I'd been hearing more and more over the last two years has been Nix, in its various forms. The big draws I've heard about include declarative, portable, cross lang dependency specification, and the ability to have a config file declare the entire state of a system without storing huge docker container artifacts for every permutation.

Actually learning Nix though felt much harder than it needed to be. The biggest problem is that the documentation you need is often fragmented among the different subprojects, and also the transition that Nix is in at the moment from the current standard methods of default.nix, configuration.nix and shell.nix to a new world of flakes.

As an outsider looking in, the flake world felt kind of in limbo at the moment where anyone I talked to indicated that flakes fix so many of the problems with current nix, but also they're still officially experimental so the official documentation declines to mention flakes.

After spending a fair amount of time understanding Nix for myself, I've decided to write my own guide which explains it from scratch in a world using only the new Nix features, with the nix CLI and flakes for the use cases I want from Nix. Credit to ianthehenry for his guide on Nix focused on the "old world" approach, and Xe Iaso for explaining flakes. The information included in this guide is largely from synthesising their blogs and the official documentation.

This first post will describe the major parts of the Nix ecosystem.

Dive into Nix

Installation (Nix From First Principles: Flake Edition #2)

#Nix

This is part 2 of the Nix from First Principles: Flake Edition series.

Nix can be installed on Linux or Mac systems. If you're using the Nix based Linux distribution, NixOS, then you'll already have it installed. Otherwise the official nix website provides the most up to date installation steps, which at the time of writing are

Linux:

sh <(curl -L https://nixos.org/nix/install) --daemon

Mac:

sh <(curl -L https://nixos.org/nix/install)

Windows (run in WSL):

sh <(curl -L https://nixos.org/nix/install) --no-daemon

Read detailed installation/configuration information

Nix Package Manager CLI (Nix From First Principles: Flake Edition #3)

#Nix

This is part 3 of the Nix from First Principles: Flake Edition series.

The foundational part of Nix is the Nix package manager. This is a tool that allows you to install packages. The "first layer" of Nix usage is to use this tool to install packages. The Nix package manager, similar to projects like os-rpmtree (used in Fedora Silverblue) or snapper (used in snap based Linux distros), will allow you to easily revert package installs or updates if something goes wrong.

This post covers the basics of using the nix package manager at the command line - while most users use nix through more advanced workflows, the basics explained here will be useful in understanding those workflows, as well as for users who want to dip their toes into Nix by treating it like a more traditional package manager to start out.

Read about installing packages manually

Just Enough Nixlang (Nix From First Principles: Flake Edition #4)

#Nix

This is part 4 of the Nix from First Principles: Flake Edition series.

This is a crash course in Nix's language, which I'll also sometimes call Nixlang from here on, to avoid confusing it with the Nix package manager. For more detailed instructions see the manual. The Nix language is used heavily within Nix, for defining packages, environments, runnable commands and library functions.

Nix's language is expression oriented. This means everything, including up to the level of an entire program, has an output value. It will be most familiar if you have experimented with functional langages such as Haskell, but it is less complicated than those languages, so you don't need to have mastered one to start.

To start experimenting with it, you can run the nix repl command which will let you type in Nix expressions and see the output.

Learn the basics of the Nix language

Derivations (Nix From First Principles: Flake Edition #5)

#Nix

This is part 5 of the Nix from First Principles: Flake Edition series.

So, time to explain the nix derivation. A Nix derivation is a special type of set created from the derivation function which describes how to obtain and build a package. The Nix package manger will then evaluate this derivation, and use the result to copy the built package into the Nix store, which is effectively a folder with all locally Nix built or installed packages in it. The subfolders are hashes derived from the inputs to the expression that built the stored packages.

Build your first derivation

Nixpkgs - How Not to Reinvent the Wheel (Nix From First Principles: Flake Edition #6)

#Nix

This is part 6 of the Nix from First Principles: Flake Edition series.

In the last section I discussed creating your first derivation, which allows you to make a first nix package. As you might have noticed, the default execution environment is incredibly barebones, to the point that you needed to include such fundamental tools as chmod and cp. If that process had to be repeated by every Nix user, it would be very inconvenient. Luckily, there is nixpkgs, which provides a number of packages that the community has already built, along with the standard environment which includes a number of tools to use in building your own. You may remember installing some of these packages in part 3.

Read how to reduce boilerplate with Nixpkgs

What about flakes then? (Nix From First Principles: Flake Edition #7)

#Nix

This is part 7 of the Nix from First Principles: Flake Edition series.

One of the big new concepts in Nix is the flake, which is a new standard format for nix projects to declare all their outputs. At the beginning of this series, I mentioned them as being one of the major components of modern Nix, and now this series has introduced enough of a foundation it's time to explain flakes themselves.

What is a flake?

A flake is a standard format for describing a collection of Nix resources. These resources can be packages, of the type described in previous posts, intended to be used to install some software to your system. There's also a number of other types of resources that can be exposed by the flake which we'll cover in later posts. Some examples of these are:

  • developer environment descriptions
  • modules for configuring NixOS systems
  • runnable commands

By standardising a format to list all these outputs (compared to the ad-hoc files this series has been using until now), Nix allows a number of higher level tools to operate on these resources, like the nix profile install and nix run commands that have been shown in earlier parts.

Finally, the development of the flake based tooling allowed the Nix team to resolve some pain points with the older generation of tooling, such as making the process of recording versions for reproducibility more ergonomic with lock files, or making better ways to identify newer versions of the same package to allow tools to manage updates.

Start learning about Flakes

Flakes and Developer Environments (Nix From First Principles: Flake Edition #8)

#Nix

This is part 8 of the Nix from First Principles: Flake Edition series.

While packages are the most common type of item you'll find contained in flakes, another type of item is a development shell. This is an environment that you can enter with the nix develop command. By default, each package will provide a development shell which is the build environment used to build it. By running just nix develop without specifying which shell is wanted, nix will put you in a bash shell with the environment from the default package.

However, sometimes you may wish to set up additional tools in a development environment, for example you may have scripts written in Python and want to have Python available as for a developer to use, yet you do not want to include Python as a dependency in your built package. Or you might want to set some environment variables so that logging is set to debug mode in your development environment.

You can customise the nix develop environment with the devShells attribute set in a flake. The helper function pkgs.mkShell is useful here. This is an extended version of the mkDerivation function that has been used in previous examples, except this version is specialised for shells. The following flake shows an example:

flake.nix

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";

  outputs = { self, nixpkgs }:
    let
      pkgs = import nixpkgs { system = "x86_64-linux"; };
    in
    {
      devShells.x86_64-linux.default = pkgs.mkShell {
        packages = [
          # Made available on the CLI
          pkgs.cargo
          pkgs.rustc
          pkgs.python3
        ];
        RUST_LOG = 1; # Set as environment variable
      };

      # You could also define extra shells or packages here
    };
}

Now if you run nix develop in the same directory as this flake, you'll be left at a bash shell with cargo, python3 and rustc available. If you run the command env | grep RUST, you can observe the following output:

$ env | grep RUST
RUST_LOG=1

Or which python3:

/nix/store/xcaaly5shfy227ffs8nipxrd49b56iqq-python3-3.10.8/bin/python3

You could even define multiple shells. For example, lets say you wanted to support Python 3.8 through 3.10 in a single project. nixpkgs provides python artifacts in specific names like python38 if you want specific versions. So you could build a flake which defines your python versions for each one, like so:

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";

  outputs = { self, nixpkgs }:
    let
      pkgs = import nixpkgs { system = "x86_64-linux"; };
      # A list of shell names and their Python versions
      pythonVersions = {
        python38 = pkgs.python38;
        python39 = pkgs.python39;
        python310 = pkgs.python310;
        default = pkgs.python310;
      };
      # A function to make a shell with a python version
      makePythonShell = shellName: pythonPackage: pkgs.mkShell {
        # You could add extra packages you need here too
        packages = [ pythonPackage ]; 
        # You can also add commands that run on shell startup with shellHook
        shellHook = ''
          echo "Now entering ${shellName} environment."
        '';
      };
    in
    {
      # mapAttrs runs the given function (makePythonShell) against every value
      # in the attribute set (pythonVersions) and returns a new set
      devShells.x86_64-linux = builtins.mapAttrs makePythonShell pythonVersions;
    };
}

Now if you run the command nix develop .#python38, you will end up in a shell that contains Python 3.8, something you can confirm with python --version.

$ nix develop .#python38
Now entering python38 environment.
$ python --version
Python 3.8.15

You can also inspect all items that are available in the flake with the command nix flake show:

$ nix flake show
git+file:///home/tony/code/nix-guide?dir=8-dev-envs%2fmanypython
└───devShells
    └───x86_64-linux
        ├───default: development environment 'nix-shell'
        ├───python310: development environment 'nix-shell'
        ├───python38: development environment 'nix-shell'
        └───python39: development environment 'nix-shell'

But what if I don't want bash?

Of course, you might prefer other shells than bash - however, Nix seems hardcoded to use bash. So you might wonder, how can you override the shell? For example, you might prefer the zsh or fish shells.

The first method of doing this requires no external tools, just Nix itself. You install your desired shell in the your devShell, and execute it in the shellHook. Here's an example that starts the fish shell.

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";

  outputs = { self, nixpkgs }:
    let
      pkgs = import nixpkgs { system = "x86_64-linux"; };
    in
    {
      devShells.x86_64-linux.default = pkgs.mkShell {
        packages = [
          pkgs.fish
        ];
        # Note that `shellHook` still uses bash syntax.
        # This starts fish, then exists the bash shell when fish exits.
        shellHook = ''
          fish && exit
        '';
      };
    };
}

While the big advantage here is that you don't need any tools apart from nix, one big disadvantage is that your shell of choice needs to be included in each development environment, and other users of your flakes are forced to use your preferred shell also.

Another option is to run nix develop -c fish. This overrides the command on shell launch to launch your preferred shell. (Thanks to u/Mysteriox7 on reddit for bringing this to my attention). This is an improvement over forcing all users of your flake to use your preferred shell, but does need to be typed each time.

With both approaches however, one thing that can be a pain is they require typing that command every time you enter a project which you want to use nix develop for.

Luckily, there are third party tools to the rescue.

nix-direnv

nix-direnv is a integration between nix and the direnv tool. direnv is a command to run commands whenever you enter a directory with a .envrc file. nix-direnv builds upon this to load a cached nix environment into your currently open shell. Effectively, the first time you open a project with nix-direnv, it will build your development environment, record the environment changes this process makes then apply those to your current shell. On opening the project again in the future, it will load the cached environment, so it doesn't need to execute your nix scripts again if they are unchanged,

To use this, you'll need to install direnv and nix-direnv. Follow their installation instructions at the links provided. Then to set up your flake project with direnv, you just need to create a .envrc file with one command, use flake

.envrc

use flake

Alternatively, you can pass it a flake reference if you want to use a different devShell. For example, my hosted version of the Python 3.8 development shell can be used with

use flake "gitlab:tonyfinn/nix-guide?dir=8-dev-envs/manypython#python38"

That's it for this part of the Nix guide. Next time I'll cover another type of flake output - runnable commands.

Runnable Flakes (Nix From First Principles: Flake Edition #9)

#Nix

This is part 9 of the Nix from First Principles: Flake Edition series.

In the last two parts, I covered using flakes for two different types of output: packages and developer environments. This time I'll cover a third type of output - runnable commands. These are what power the nix run command that you've seen a few times throughout the series, and now you will find out how to get this functionality for your own flakes.

Run Flakes without installation