🗒️ l-lin

Search

SearchSearch

✏️ 2024-07-18 🗓️️ Log

running an external binary on NixOS

NOTE

TLDR: I installed nix-ld, so most binaries should work without any problem. If not, check below on how to configure it to make it work for your binary.

Otherwise, there is also nix-alien and steam-run alternatives.

HA! Welcome to the dark side of NixOS. You want to run a binary you downloaded on the internet on NixOS? Nope, you can’t!

Well, it’s logical as NixOS philosophy is to have reproducible and immutable environments. So it’s logical some programs are not in the same place as the other Linux distribution.

Precompiled binaries that were not created for NixOS usually have a so-called link-loader hardcoded into them. On Linux/x86_64 this is for example /lib64/ld-linux-x86-64.so.2. for glibc. NixOS, on the other hand, usually has its dynamic linker in the glibc package in the Nix store and therefore cannot run these binaries.

NixOS is not FHS compliant

There is something called the Filesystem Hierarchy Standard (FHS) which is a reference describing the conventions used for the layout of Unix-like systems, e.g. have the essential command binaries under /bin, …

NixOS is NOT FHS compliant, and some programs you downloaded on the internet will try to access hard-coded FHS file path like /usr/lib or /opt.

Moreover, most programs are using a hard-coded Executable and Linkable Format (ELF) path to be executed.

This format is a common standard file format for executable files, object code, shared libraries, and core dumps.

Generally, we write most programs in high-level languages such as C or C++. These programs cannot be directly executed on the CPU because the CPU doesn’t understand these instructions. Instead, we use a compiler that compiles the high-level language into object code. Using a linker, we also link the object code with shared libraries to get a binary file.

As a result, the binary file has instructions that the CPU can understand and execute. The binary file can adopt any format that defines the structure it should follow. However, the most common of these structures is the ELF format.

So you may encounter some error like this:

[ERROR][2024-06-13 14:07:39] .../vim/lsp/rpc.lua:734	"rpc"	"/home/l-lin/.local/share/nvim/mason/bin/lua-language-server"	"stderr"	"Could not start dynamically linked executable: /home/l-lin/.local/share/nvim/mason/packages/lua-language-server/libexec/bin/lua-language-server\nNixOS cannot run dynamically linked executables intended for generic\nlinux environments out of the box. For more information, see:\nhttps://nix.dev/permalink/stub-ld\n"

So what can you do?

As it turns out, there are 10 different methods to run a non-nixos executable on Nixos! :scream:

Here are some methods that worked for me:

Patching ELF manually

# You can manually patch them, and if you're lucky, that's enough:
patchelf --set-interpreter $(patchelf --print-interpreter `which find`) lua-language-server
 
# You can now execute it like any other binary:
./lua-language-server

If that’s not enough, e.g. there’re some missing libraries:

$ ldd marksman | grep 'not found'
        libz.so.1 => not found
        libstdc++.so.6 => not found

In that case, you need to:

  1. find which packages provide those libraries
  • you can use nix-index to find the packages
  • or you can use https://pkgs.org
  • libz.so.1 is provided by zlib and libstdc++.so.6 is part of the C/C++ compiler tool chain
  1. find the path to those packages in your Nix store
  • if the package is not present in your Nix store, you will need to install it
$ # First generate the database index of all files in our channel (can be quite slow):
$ nix-index
+ querying available packages
+ generating index: 114066 paths found :: 29905 paths not in binary cache :: 00000 paths in queue
+ wrote index of 70,026,341 bytes
 
$ # We use the power of nix-locate to find the packages which contain the file:
$ nix-locate --minimal --top-level -w lib/libz.so.1
zlib.out
remarkable-toolchain.out
remarkable2-toolchain.out
libz.out
figma-linux.out
$ nix-locate --minimal --top-level -w lib/libstdc++.so.6
robo3t.out
remarkable-toolchain.out
remarkable2-toolchain.out
libgcc.lib
 
$ # You can find the package using the following command:
$ nix eval 'nixpkgs#zlib.outPath' --raw
/nix/store/lv6nackqis28gg7l2ic43f6nk52hb39g-zlib-1.3.1
$ nix eval 'nixpkgs#stdenv.cc.cc.lib.outPath' --raw
/nix/store/xvzz97yk73hw03v5dhhz3j47ggwf1yq1-gcc-13.2.0-lib

Now, patch the Rpath of the binary.

# By patching the RPATH, marksman is now aware of the missing
# libraries and works on NixOS
patchelf \
  --set-rpath "$(nix eval nixpkgs#zlib.outPath --raw)/lib:$(nix eval nixpkgs#stdenv.cc.cc.lib.outPath --raw)/lib" \
  marksman

rpath designates the run-time search path hard-coded in an executable file or library. Dynamic linking loaders use the rpath to find required libraries.

Specifically, it encodes a path to shared libraries into the header of an executable (or another shared library). This rpath header value (so named in the Executable and Linkable Format header standards) may either override or supplement the system default dynamic linking search paths.

As you can see, it’s quite tedious and error-prone. The following options may be better.

Running using nix-alien

nix-alien will help you run unpatched binaries without modifying them by setting the interpreter and linking the dynamic libraries needed.

First add nix-alien in your home-manager configuration.

It should already be present at nix-alien:

{ inputs, systemSettings, ... }: {
  home.packages = with inputs.nix-alien.packages.${systemSettings.system}; [ nix-alien ];
}

Then, you can run it like this:

# It will open an interactive form to choose where the 
nix-alien ./marksman

It also have other options. I still did not explore them.

Using nix-ld

nix-ld provides a shim layer for these binaries. It is installed in the same location where other Linux distributions install their link loader, ie. /lib64/ld-linux-x86-64.so.2 and then loads the actual link loader as specified in the environment variable NIX_LD. In addition, it also accepts a colon-separated path from library lookup paths in NIX_LD_LIBRARY_PATH. This environment variable is rewritten to LD_LIBRARY_PATH before passing execution to the actual ld. This allows you to specify additional libraries that the executable needs to run.

First add nix-ld in your NixOS configuration.

It should already be present at nix-ld:

{ pkgs, ... }: {
  programs.nix-ld = {
    enable = true;
    package = nix-ld-rs;
    libraries = [
      # ...
    ];
  };
}

Now, you will be able to run any binary that only needs to have their interpreter patched! For example, most LSP servers will be able to run!

If not, use nix-index with nix-locate to find the package of the missing library:

$ nix-locate --minimal --top-level -w lib/libgobject-2.0.so.0
remarkable-toolchain.out
remarkable2-toolchain.out
glib.out

Then update unpatched-binaries.nix to include the package, and apply the change with make nixos.

Using steam-run

steam-run is a tool in the Nix package repository that provides an environment mimicking the traditional FHS, primarily intended for running the Steam gaming client on NixOS. However, it can be used for other use cases (like this one).

First add steam-run in your NixOS configuration (should already be present):

{ pkgs, ... }: {
  # Run commands in the same FHS environment that is used for Steam: https://store.steampowered.com/
  environment.systemPackages = with pkgs; [ steam-run ];
}
# Once steam-run is installed system-wide, you can run any program in the FHS environment:
steam-run your-program args
 
# Example running openfortivpn-webclient binary in current folder:
steam-run ./openfortivpn-webclient

References to run binaries in NixOS

Some ways to create a FHS environment:

Recent Notes

Graph View

Backlinks