create python nix derivation

Abstract
Creating a python derivation is not trivial. python has already lots of issue with version conflict. So if you need to create one python nix derivation, you just hope it does not have dependency. Otherwise, there is some mitigation, but it’s not easy to deal it. The easiest one I find is to manually pip install in the shellHook. It does not quite respect nix reproducible philosophy however:
{
  # ...
  devShells.default = pkgs.mkShell {
    packages = with pkgs; [
        python3
        python3Packages.pip
        python3Packages.virtualenv
      ];
     shellHook = ''
       if [ ! -d .venv ]; then
         virtualenv .venv
       fi
 
       # Activate the virtual environment
       source .venv/bin/activate
       pip install presenterm-export
     '';
  };
}

Context

I wanted to install [presenterm-export][] to my local computer, so I could export my markdown slides. Unfortunately, this tool is not present yet in nixpkgs.

I’m using direnv to automatically install the tools when I go to a folder. So I created a flake.nix with the following content:

{
  description = "Flake to install needed dependencies.";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  inputs.flake-utils.url = "github:numtide/flake-utils";
 
  outputs = { nixpkgs, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; config.allowUnfree = true; };
      in {
        devShells.default = pkgs.mkShell {
          packages = with pkgs; [ curl jq presenterm ];
        };
      });
}

Since there’s no [presenterm-export][] in the nixpkgs, I need to install manually.

nix derivation

According to [presenterm-export][], we can install using the following:

pip install presenterm-export

So I checked some tutorials out there, but there were no trivial one where I could just copy paste:

Still, I managed to do something. I created a presenter-export.nix with the following:

{ 
  # Builtin nix function that also calls mkDerivation.
  buildPythonPackage,
  # Fetch the tarball from pypi.
  fetchPypi,
  setuptools,
  # Pypi dependencies to install.
  # Found from code source: https://github.com/mfontanini/presenterm-export/blob/454ca9e73adb38f57ba93f4e4ed51695d04432df/pyproject.toml#L17-L20.
  ansi2html,
  dataclass-wizard,
  libtmux,
  weasyprint,
}: buildPythonPackage rec {
  pname = "presenterm-export";
  version = "0.2.3";
 
  pyproject = true;
  src = fetchPypi {
    # For some reason, the name in pypi is using a underscore...
    pname = "presenterm_export";
    inherit version;
    hash = "sha256-cevPqvW3vX6gAa7+MElIZYbMaJHoP8juC2o/6eci13Y=";
  };
  build-system = [ setuptools ];
 
  dependencies = [
    ansi2html
    libtmux
    weasyprint
    dataclass-wizard
  ];
}

Then update flake.nix:

{
  description = "Flake to install needed dependencies.";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  inputs.flake-utils.url = "github:numtide/flake-utils";
 
  outputs = { nixpkgs, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; config.allowUnfree = true; };
        presentermExport = with pkgs.python3Packages; pkgs.callPackage ./presenterm-export.nix {
          inherit buildPythonPackage fetchPypi setuptools ansi2html dataclass-wizard libtmux weasyprint
        };
      in {
        devShells.default = pkgs.mkShell {
          packages = with pkgs; [ curl jq presenterm ];
        };
      });
}

Then I execute the following:

$ nix develop
error: builder for '/nix/store/q0i5nk2zmmy4kzawknbhk3qqbymccb06-python3.11-presenterm-export-0.2.3.drv' failed with exit code 1;
       last 10 log lines:
       > removing build/bdist.linux-x86_64/wheel
       > Successfully built presenterm_export-0.2.3-py3-none-any.whl
       > Finished creating a wheel...
       > Finished executing pypaBuildPhase
       > Running phase: pythonRuntimeDepsCheckHook
       > Executing pythonRuntimeDepsCheck
       > Checking runtime dependencies for presenterm_export-0.2.3-py3-none-any.whl
       >   - ansi2html==1.8.0 not satisfied by version 1.9.2
       >   - libtmux==0.23.2 not satisfied by version 0.37.0
       >   - weasyprint==62.3.0 not satisfied by version 61.2
       For full logs, run 'nix log /nix/store/q0i5nk2zmmy4kzawknbhk3qqbymccb06-python3.11-presenterm-export-0.2.3.drv'.

It did not work (I hope it would 😔)…

It seems I need to have the specific version of the dependencies.

❌ Find the nixpkgs SHA to download the specific version of a dependency

We can use Nix Package Versions to find the right nix version / HASH to get the specific version of a dependency.

I could find the SHA for ansi2html. However, for some reasons, I could not find anything for libtmux and weasyprint.

Another way is to directly find in the https://github.com/NixOS/nixpkgs repository.

In order to quickly find the right version, I cloned the repository locally and check the git log of the dependency, i.e.:

libtmux: https://github.com/NixOS/nixpkgs/commits/master/pkgs/development/python-modules/libtmux/default.nix ⇒ ebe1cb8c0f4d86db9424abf7b0de5be36ba574ef
weasyprint: https://github.com/NixOS/nixpkgs/commits/master/pkgs/development/python-modules/weasyprint/default.nix
- unfortunately, there’s no version 62.3 for this dependency…

I updated my flake.nix with the following:

{
  description = "Flake to install needed dependencies.";
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/4b8f07321700193fdab64e4a2cc3c4ab59f39eb1";
    ansi2htmlNixpkgs.url = "github:NixOS/nixpkgs/336eda0d07dc5e2be1f923990ad9fdb6bc8e28e3";
    libtmuxNixpkgs.url = "github:NixOS/nixpkgs/ebe1cb8c0f4d86db9424abf7b0de5be36ba574ef";
    flake-utils.url = "github:numtide/flake-utils";
  };
 
  outputs = { nixpkgs, ansi2htmlNixpkgs, libtmuxNixpkgs, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; config.allowUnfree = true; };
        ansi2htmlPkgs = import ansi2htmlNixpkgs { inherit system; };
        libtmuxPkgs = import libtmuxNixpkgs { inherit system; };
        presenterExport = with pkgs.python3Packages; pkgs.callPackage ./presenterm-export.nix  {
          inherit buildPythonPackage fetchPypi setuptools;
          ansi2html = ansi2htmlPkgs.python3Packages.ansi2html;
          inherit dataclass-wizard;
          libtmux = libtmuxPkgs.python310Packages.libtmux;
          inherit weasyprint;
        };
      in {
        devShells.default = pkgs.mkShell {
          packages = with pkgs; [ curl jq presenterm ];
        };
      });
}

Unfortunately, it’s not working either:

         at /nix/store/avci5b2qjsxkcc17fpl1gralpv6b4xsd-source/pkgs/stdenv/generic/make-derivation.nix:376:7:

          375|       depsBuildBuild              = elemAt (elemAt dependencies 0) 0;
          376|       nativeBuildInputs           = elemAt (elemAt dependencies 0) 1;
             |       ^
          377|       depsBuildTarget             = elemAt (elemAt dependencies 0) 2;

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: Python version mismatch in 'python3.12-presenterm-export-0.2.3':

       The Python derivation 'python3.12-presenterm-export-0.2.3' depends on a Python derivation
       named 'python3.11-ansi2html-1.8.0', but the two derivations use different versions
       of Python:

           'python3.12-presenterm-export-0.2.3' uses /nix/store/z7xxy35k7620hs6fn6la5fg2lgklv72l-python3-3.12.4
                   'python3.11-ansi2html-1.8.0' uses /nix/store/sxr2igfkwhxbagri49b8krmcqz168sim-python3-3.11.8

       Possible solutions:

         * If 'python3.11-ansi2html-1.8.0' is a Python library, change the reference to 'python3.11-ansi2html-1.8.0'
           in the propagatedBuildInputs of 'python3.12-presenterm-export-0.2.3' to use a 'python3.11-ansi2html-1.8.0' built from the same
           version of Python

         * If 'python3.11-ansi2html-1.8.0' is used as a tool during the build, move the reference to
           'python3.11-ansi2html-1.8.0' in 'python3.12-presenterm-export-0.2.3' from propagatedBuildInputs to nativeBuildInputs

         * If 'python3.11-ansi2html-1.8.0' provides executables that are called at run time, pass its
           bin path to makeWrapperArgs:

               makeWrapperArgs = [ "--prefix PATH : ${lib.makeBinPath [ ansi2html ] }" ];

        at /nix/store/356lp2h4rvgkvlw83pnf092p3f858ph8-source/presenterm-export.nix:14:3

Fail

Not possible to fetch different versions of the dependencies because they would use a different python version, hence a conflict. Not sure if it’s possible or not to use the same python.

❌ dependencies as `nativeBuildInputs`

I tried to use their one of their option in presenter-export.nix:

  # ...
  nativeBuildInputs = [
    ansi2html
    libtmux
  ];
 
  dependencies = [
    weasyprint
    dataclass-wizard
  ];
}

However, that did not went well. It downloaded 7GB of data to finally failed with the following error:

error: builder for '/nix/store/rn12q26sw4yxj297q17gwf5g0fdwm61n-audit-3.1.2.drv' failed with exit code 1;
       last 10 log lines:
       > checking for python version... 3.12
       > checking for python platform... linux
       > checking for GNU default python prefix... ${prefix}
       > checking for GNU default python exec_prefix... ${exec_prefix}
       > checking for python script directory (pythondir)... ${PYTHON_PREFIX}/lib/python3.12/site-packages
       > checking for python extension module directory (pyexecdir)... ${PYTHON_EXEC_PREFIX}/lib/python3.12/site-packages
       > Traceback (most recent call last):
       >   File "<string>", line 1, in <module>
       > ModuleNotFoundError: No module named 'distutils'
       > configure: error: Python explicitly requested and python headers were not found
       For full logs, run 'nix log /nix/store/rn12q26sw4yxj297q17gwf5g0fdwm61n-audit-3.1.2.drv'.
error: 1 dependencies of derivation '/nix/store/64a12rhqh517x7pdpgy039a0byhql01k-dbus-1.14.10.drv' failed to build
error: 1 dependencies of derivation '/nix/store/vn2bjd4p48d85k4nggb5gddbazjvnrrv-linux-pam-1.6.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/cjfq86dl1pl6bbxdbmnbg8vzljx2xh6s-cups-2.4.8.drv' failed to build
error: 1 dependencies of derivation '/nix/store/762b7ph538mbgy71wa7dnhr2k4wdxqzh-util-linux-minimal-2.39.4.drv' failed to build
error: 1 dependencies of derivation '/nix/store/wfb10mpxsiskh8jyb179nm57v286g0ca-e2fsprogs-1.47.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/fap3yi9x0zndgabcyc3mxamp5l2vk133-ghostscript-with-X-10.03.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/1dkc10jn0y4d0v8gsz5ay0kmhwykzbfc-glib-2.80.3.drv' failed to build
error: 1 dependencies of derivation '/nix/store/41zm6zhiplh8zrnmdakgsfzlicpdqpl8-libarchive-3.7.4.drv' failed to build
error: 1 dependencies of derivation '/nix/store/law0hvlp64scrxmhz6hmscmblz4f5zz6-python3.12-weasyprint-62.2.drv' failed to build
error: 1 dependencies of derivation '/nix/store/mjajks3frz923gfnp9923afv4l346my9-cmake-3.29.6.drv' failed to build
error: 1 dependencies of derivation '/nix/store/rvhjakhyk9xjj14b25xq86rpsm160vdb-python3.12-presenterm-export-0.2.3.drv' failed to build
error: 1 dependencies of derivation '/nix/store/nx8i4d1yqzvsr89iswvfwinj958x41f7-nix-shell-env.drv' failed to build

Fail

Could not make it work when the dependencies were packaged with a different version of python…

Use python virtual environment

src: Python - NixOS Wiki

We could instead use python virtual environment, so updating flake.nix with the following:

{
  description = "Flake to install needed dependencies.";
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };
 
  outputs = { nixpkgs, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; config.allowUnfree = true; };
      in {
 
        devShells.default = pkgs.mkShell {
          packages = with pkgs; [
            curl
            jq
            presenterm
 
            # python3 and virtualenv to install presenterm-export
            python3
            python3Packages.pip
            python3Packages.virtualenv
            # Need to manually install weasyprint.
            # src: https://github.com/mfontanini/presenterm-export/issues/7#issuecomment-2195478609
            python3Packages.weasyprint
          ];
        };
      });
}

Then, install manually:

$ # create virtualenv
$ virtualenv .venv
$ # activate virtualenv
$ source .venv/bin/activate
$ # install presenterm-export
$ pip install presenterm-export

We could also use shellHook so we don’t have to manually perform those steps:

{
  # ...
  
        devShells.default = pkgs.mkShell {
          packages = with pkgs; [
            # ...
          ];
          shellHook = ''
            # Create a virtual environment in the .venv directory
            if [ ! -d .venv ]; then
              virtualenv .venv
            fi
 
            # Activate the virtual environment
            source .venv/bin/activate
 
            if [ ! type presenterm-export >/dev/null 2>&1 ]; then
              pip install presenterm-export
            fi
 
            echo "Virtual environment activated. Use 'deactivate' to exit."
          '';
        };
      });
}

nix derivation for `weasyprint`

The installation went well with the python virtualenv. However, the execution, not so well:

$ # Perform export
$ presenterm --export-pdf SLIDES.md
tmux version: 3.4
Writing temporary files into /tmp/tmp7bl8s1nc
Running presentation to capture slide...
Running '/nix/store/27s9ibvam9qcyv120jaijw43j2hani0c-presenterm-0.7.0/bin/presenterm' '--export' '/home/l-lin/work/welcoming-kubernetes/SLIDES.md'
'presenterm-export' execution failed:
Traceback (most recent call last):
  File "/home/l-lin/work/welcoming-kubernetes/.venv/bin/presenterm-export", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/l-lin/work/welcoming-kubernetes/.venv/lib/python3.12/site-packages/presenterm_export/cli.py", line 112, in main
    run(args, metadata)
  File "/home/l-lin/work/welcoming-kubernetes/.venv/lib/python3.12/site-packages/presenterm_export/cli.py", line 49, in run
    presentation = capture_slides(args.rest, metadata.commands)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/l-lin/work/welcoming-kubernetes/.venv/lib/python3.12/site-packages/presenterm_export/capture.py", line 31, in capture_slides

According to the pyproject.toml of presenterm-export, weasyprint should have the version 62.3.0. However, the one in nixpkgsis 62.2.0:

$ weasyprint --info
System: Linux
Machine: x86_64
Version: #1-NixOS SMP PREEMPT_DYNAMIC Thu Jun 27 11:49:15 UTC 2024
Release: 6.6.36
 
WeasyPrint version: 62.2
Python version: 3.12.4
Pydyf version: 0.11.0
Pango version: 15202

If I tried to install manually using pip install:

shellHook = ''
  # ...
  if [ ! type weasyprint >/dev/null 2>&1 ]; then
    pip install weasyprint
  fi
  # ...
'';

weasyprint is indeed installed, but there are errors:

$ weasyprint --info
-----
 
WeasyPrint could not import some external libraries. Please carefully follow the installation steps before reporting an issue:
https://doc.courtbouillon.org/weasyprint/stable/first_steps.html#installation
https://doc.courtbouillon.org/weasyprint/stable/first_steps.html#troubleshooting
 
-----
 
Traceback (most recent call last):
  File "/home/l-lin/work/welcoming-kubernetes/.venv/bin/weasyprint", line 5, in <module>
    from weasyprint.__main__ import main
  File "/home/l-lin/work/welcoming-kubernetes/.venv/lib/python3.12/site-packages/weasyprint/__init__.py", line 419, in <module>
    from .css import preprocess_stylesheet  # noqa: I001, E402
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/l-lin/work/welcoming-kubernetes/.venv/lib/python3.12/site-packages/weasyprint/css/__init__.py", line 28, in <module>
    from .computed_values import COMPUTER_FUNCTIONS
  File "/home/l-lin/work/welcoming-kubernetes/.venv/lib/python3.12/site-packages/weasyprint/css/computed_values.py", line 9, in <module>
    from ..text.ffi import ffi, pango, units_to_double
  File "/home/l-lin/work/welcoming-kubernetes/.venv/lib/python3.12/site-packages/weasyprint/text/ffi.py", line 431, in <module>
    gobject = _dlopen(
              ^^^^^^^^
  File "/home/l-lin/work/welcoming-kubernetes/.venv/lib/python3.12/site-packages/weasyprint/text/ffi.py", line 420, in _dlopen
    return ffi.dlopen(names[0])  # pragma: no cover
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/l-lin/work/welcoming-kubernetes/.venv/lib/python3.12/site-packages/cffi/api.py", line 150, in dlopen
    lib, function_cache = _make_ffi_library(self, name, flags)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/l-lin/work/welcoming-kubernetes/.venv/lib/python3.12/site-packages/cffi/api.py", line 832, in _make_ffi_library
    backendlib = _load_backend_lib(backend, libname, flags)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/l-lin/work/welcoming-kubernetes/.venv/lib/python3.12/site-packages/cffi/api.py", line 827, in _load_backend_lib
    raise OSError(msg)
OSError: cannot load library 'gobject-2.0-0': gobject-2.0-0: cannot open shared object file: No such file or directory.  Additionally, ctypes.util.find_library() did not manage to locate a library called 'gobject-2.0-0'

It seems there are dependencies that needs to be installed.

So instead, I copied the weasyprint/default.nix from nixpks and its library-paths.patch locally, and just updated the version and removed some non-needed attributes:

buildPythonPackage rec {
  pname = "weasyprint";
-  version = "62.2";
+  version = "62.3";
  format = "pyproject";
 
-  disabled = pythonOlder "3.9";

And in flake.nix:

{
  description = "Flake to install needed dependencies.";
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };
 
  outputs = { nixpkgs, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; config.allowUnfree = true; };
        python3Packages = pkgs.python3Packages;
        weasyprint = pkgs.callPackage ./weasyprint.nix {
          inherit (pkgs) lib stdenv;
          inherit (python3Packages) buildPythonPackage cffi cssselect2;
          inherit (python3Packages) fetchPypi flit-core;
          inherit (pkgs) fontconfig;
          inherit (python3Packages) fonttools;
          inherit (pkgs) ghostscript glib harfbuzz;
          inherit (python3Packages) html5lib;
          inherit (pkgs) pango;
          inherit (python3Packages) pillow pydyf pyphen;
          inherit (python3Packages) pytestCheckHook;
          inherit (pkgs) substituteAll;
          inherit (python3Packages) tinycss2;
        };
      in {
 
        devShells.default = pkgs.mkShell {
          packages = with pkgs; [
            curl
            jq
            presenterm
            
            # python3 and virtualenv to install presenterm-export
            python3
            python3Packages.pip
            python3Packages.virtualenv
            #python3Packages.weasyprint
            weasyprint
          ];
          shellHook = ''
            # Create a virtual environment in the .venv directory
            if [ ! -d .venv ]; then
              virtualenv .venv
            fi
 
            # Activate the virtual environment
            source .venv/bin/activate
 
            # if [ ! type weasyprint >/dev/null 2>&1 ]; then
            #   pip install weasyprint
            # fi
            if [ ! type presenterm-export >/dev/null 2>&1 ]; then
              pip install presenterm-export
            fi
 
            echo "Virtual environment activated. Use 'deactivate' to exit."
          '';
        };
      });
}

Now, weasyprint is in the right version:

$ weasyprint --info
System: Linux
Machine: x86_64
Version: #1-NixOS SMP PREEMPT_DYNAMIC Thu Jun 27 11:49:15 UTC 2024
Release: 6.6.36
 
WeasyPrint version: 62.3
Python version: 3.12.4
Pydyf version: 0.11.0
Pango version: 15202

However, I still got the error when trying to export my slides in PDF:

$ presenterm --export-pdf SLIDES.md
 
tmux version: 3.4
Writing temporary files into /tmp/tmp2jpe7q9i
Running presentation to capture slide...
Running '/nix/store/27s9ibvam9qcyv120jaijw43j2hani0c-presenterm-0.7.0/bin/presenterm' '--export' '/home/l-lin/work/welcoming-kubernetes/SLIDES.md'
'presenterm-export' execution failed:
Traceback (most recent call last):
  File "/home/l-lin/work/welcoming-kubernetes/.venv/bin/presenterm-export", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/l-lin/work/welcoming-kubernetes/.venv/lib/python3.12/site-packages/presenterm_export/cli.py", line 112, in main
    run(args, metadata)
  File "/home/l-lin/work/welcoming-kubernetes/.venv/lib/python3.12/site-packages/presenterm_export/cli.py", line 49, in run
    presentation = capture_slides(args.rest, metadata.commands)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/l-lin/work/welcoming-kubernetes/.venv/lib/python3.12/site-packages/presenterm_export/capture.py", line 31, in capture_slides

[presenter-export]: GitHub - mfontanini/presenterm-export: PDF exporter for presenterm presentations

🗒️ l-lin

Explorer

create python nix derivation

Context

nix derivation

❌ Find the nixpkgs SHA to download the specific version of a dependency

❌ dependencies as `nativeBuildInputs`

Use python virtual environment

nix derivation for `weasyprint`

Explorer

Recent Notes

sops

kubetailrb

spring boot 3.4 unwrapped

tidy first - part 1 - tidyings

database sharding explained

Graph View

Table of Contents

Backlinks

🗒️ l-lin

Explorer

create python nix derivation

Context

nix derivation

❌ Find the nixpkgs SHA to download the specific version of a dependency

❌ dependencies as nativeBuildInputs

Use python virtual environment

nix derivation for weasyprint

Explorer

Recent Notes

sops

kubetailrb

spring boot 3.4 unwrapped

tidy first - part 1 - tidyings

database sharding explained

Graph View

Table of Contents

Backlinks

❌ dependencies as `nativeBuildInputs`

nix derivation for `weasyprint`