ERROR: Failed cleaning build dir for numpy, git clean -xdf (cleans all files) in QEMU + KVM + cloud image of Ubuntu 18.04.5 LTS

[PedroRegisPOAR]

First thank you all for this great tool, it is really cool!

TL;DR

I am trying to find why and how to solve a error that happens in some environments that are really specific i would say.

The only environment that i was able to reproduce deterministically the error was using a flake with QEMU + KVM + cloud image of Ubuntu 18.04.5 LTS (i checked it with lsb_release -a).

Steps to reproduce

I did a flake of the QEMU VM in a repository (the main idea comes from this that zimbatm did), and here are the steps to reproduce.

nix build github:ES-Nix/nix-qemu-kvm/f991d5a3125df5524f92600e778ae3581f2a26b7#myqemu.prepare
ls
./result/runVM

I was not able to make the ssh work until now. So, a little manual input is necessary:

1. First build and boot the QEMU VM (copy paste the above code):
   

2. After a lot of logs of the boot system you may see some thing like this:
   

Just press any key. You might receive some random stdout lines, just ignore that, and press any key.

3. Press Ctrl + d and login. The user name is ubuntu and the password is b (yes just b, it only exists because i was not able to remove it and let it passwordless):
   

4. Install nix + flake:

For my use i created my own installer, it only injects a few stuff like some bash function to not have to type a lot of code and the KVM config for nix, it is almost this line echo 'system-features = kvm nixos-test' >> ~/.config/nix/nix.conf. I tested it too with the official installation, it reproduces the error even with the official installer and adding flakes carefully (dont forget to add the KVM thing).

test -d /nix || sudo mkdir --mode=0755 /nix \
&& sudo chown "$USER": /nix \
&& curl -fsSL https://raw.githubusercontent.com/ES-Nix/get-nix/bc848b1e30df8ba2ec391e95639235e9051bc6ca/get-nix.sh | sh \
&& . ~/.profile \
&& flake

You should endup in some thing like this:

5. Finally the flake that produces the error

In the flake shell copy paste and run:

nix develop github:ES-Nix/poetry2nix-examples/2cb6663e145bbf8bf270f2f45c869d69c657fef2

After that

In text format:

error: --- Error ----------------------------------------------------------- nix
builder for '/nix/store/lp9nld2g8649f3c7chlgwajhsj3ax906-python3.8-numpy-1.20.1.drv' failed with exit code 1; last 10 log lines:
      - `git clean -xdf` (cleans all files)
      - `git clean -Xdf` (cleans all versioned files, doesn't touch
                          files that aren't checked into the git repo)
    
    Add `--force` to your command to use it anyway if you must (unsupported).
    
    ----------------------------------------
    ERROR: Failed cleaning build dir for numpy
  Failed to build numpy
  ERROR: Failed to build one or more wheels
error: --- Error ----------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/rbpvhsnzkqmgh86cghrb5kzrriv4nsvk-python3-3.8.7-env.drv' failed to build
error: --- Error ----------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/s8xjw6gi1ibc2d8xwww26rr0fm8hxz7l-nix-shell-env.drv' failed to build

I tried too to build an OCI image using dockerTools and poetry2nix.

nix build github:ES-Nix/poetry2nix-examples/2cb6663e145bbf8bf270f2f45c869d69c657fef2#poetry2nixOCIImage

In text format:

error: --- Error ----------------------------------------------------------- nix
builder for '/nix/store/lp9nld2g8649f3c7chlgwajhsj3ax906-python3.8-numpy-1.20.1.drv' failed with exit code 1; last 10 log lines:
      - `git clean -xdf` (cleans all files)
      - `git clean -Xdf` (cleans all versioned files, doesn't touch
                          files that aren't checked into the git repo)
    
    Add `--force` to your command to use it anyway if you must (unsupported).
    
    ----------------------------------------
    ERROR: Failed cleaning build dir for numpy
  Failed to build numpy
  ERROR: Failed to build one or more wheels
error: --- Error ----------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/lli0wvnmgrzr6wx7c28z0749x242n0b2-python3.8-nixfriday-0.1.0.drv' failed to build
error: --- Error ----------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/2jnnq6lc2m4rdm5nwriy3xhmsiblwds4-closure.drv' failed to build
error: --- Error ----------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/l02c294jblr1g1naa5p3p20qxcrl8f4j-numtild-dockertools-poetry2nix-conf.json.drv' failed to build
error: --- Error ----------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/zipwcb1l8nww70p4xm20xid8nfp4zzwf-stream-numtild-dockertools-poetry2nix.drv' failed to build
error: --- Error ----------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/qv80zcwhbvbj1wv224m6250g4sq116jb-numtild-dockertools-poetry2nix.tar.gz.drv' failed to build

For completness i tested the mach-nix and it fails too:

nix develop github:davhau/mach-nix#shellWith.numpy

Long trace log. Click to expand!

error: --- Error ----------------------------------------------------------- nix
failed to extract archive: Write failed
---------------------------------- show-trace ----------------------------------
trace: while evaluating 'removeSuffix'
at: (473:5) in file: /nix/store/kcv8hh2nrrpqm6d04rvw5dccrd5fdrhr-source/lib/strings.nix

   472|     # Input string
   473|     str:
      |     ^
   474|     let

trace: from call site
at: (13:25) in file: /nix/store/zc822dgnnkdpssb9vdgzq8fbgj0iyg8f-source/mach_nix/nix/deps-db-and-fetcher.nix

    12|   };
    13|   pypi_fetcher_commit = removeSuffix "\n" (readFile "${deps_db_src}/PYPI_FETCHER_COMMIT");
      |                         ^
    14|   pypi_fetcher_sha256 = removeSuffix "\n" (readFile "${deps_db_src}/PYPI_FETCHER_SHA256");

trace: while evaluating the attribute 'pypi_fetcher'
at: (30:10) in file: /nix/store/zc822dgnnkdpssb9vdgzq8fbgj0iyg8f-source/mach_nix/nix/deps-db-and-fetcher.nix

    29| in
    30| { inherit
      |          ^
    31|     pypi_deps_db_src

trace: while evaluating 'extendDerivation'
at: (144:43) in file: /nix/store/kcv8hh2nrrpqm6d04rvw5dccrd5fdrhr-source/lib/customisation.nix

   143|      the derivation itself and check a given condition when evaluating. */
   144|   extendDerivation = condition: passthru: drv:
      |                                           ^
   145|     let

trace: from call site
at: (341:7) in file: /nix/store/kcv8hh2nrrpqm6d04rvw5dccrd5fdrhr-source/pkgs/stdenv/generic/make-derivation.nix

   340| 
   341|       lib.extendDerivation
      |       ^
   342|         validity.handled

trace: while evaluating 'mkDerivation'
at: (22:5) in file: /nix/store/kcv8hh2nrrpqm6d04rvw5dccrd5fdrhr-source/pkgs/stdenv/generic/make-derivation.nix

    21|   mkDerivation =
    22|     {
      |     ^
    23| 

trace: from call site
at: (344:31) in file: /nix/store/kcv8hh2nrrpqm6d04rvw5dccrd5fdrhr-source/pkgs/stdenv/generic/make-derivation.nix

   343|         ({
   344|            overrideAttrs = f: mkDerivation (attrs // (f attrs));
      |                               ^
   345| 

trace: while evaluating 'overrideAttrs'
at: (344:28) in file: /nix/store/kcv8hh2nrrpqm6d04rvw5dccrd5fdrhr-source/pkgs/stdenv/generic/make-derivation.nix

   343|         ({
   344|            overrideAttrs = f: mkDerivation (attrs // (f attrs));
      |                            ^
   345| 

trace: from call site
at: (86:32) in file: /nix/store/kcv8hh2nrrpqm6d04rvw5dccrd5fdrhr-source/lib/customisation.nix

    85|           ${if result ? overrideAttrs then "overrideAttrs" else null} = fdrv:
    86|             overrideResult (x: x.overrideAttrs fdrv);
      |                                ^
    87|         }

trace: while evaluating anonymous lambda
at: (86:29) in file: /nix/store/kcv8hh2nrrpqm6d04rvw5dccrd5fdrhr-source/lib/customisation.nix

    85|           ${if result ? overrideAttrs then "overrideAttrs" else null} = fdrv:
    86|             overrideResult (x: x.overrideAttrs fdrv);
      |                             ^
    87|         }

trace: from call site
at: (79:60) in file: /nix/store/kcv8hh2nrrpqm6d04rvw5dccrd5fdrhr-source/lib/customisation.nix

    78|       # Change the result of the function call by applying g to it
    79|       overrideResult = g: makeOverridable (copyArgs (args: g (f args))) origArgs;
      |                                                            ^
    80|     in

trace: while evaluating anonymous lambda
at: (79:54) in file: /nix/store/kcv8hh2nrrpqm6d04rvw5dccrd5fdrhr-source/lib/customisation.nix

    78|       # Change the result of the function call by applying g to it
    79|       overrideResult = g: makeOverridable (copyArgs (args: g (f args))) origArgs;
      |                                                      ^
    80|     in

trace: from call site
at: (69:16) in file: /nix/store/kcv8hh2nrrpqm6d04rvw5dccrd5fdrhr-source/lib/customisation.nix

    68|     let
    69|       result = f origArgs;
      |                ^
    70| 

trace: while evaluating 'makeOverridable'
at: (67:24) in file: /nix/store/kcv8hh2nrrpqm6d04rvw5dccrd5fdrhr-source/lib/customisation.nix

    66|   */
    67|   makeOverridable = f: origArgs:
      |                        ^
    68|     let

trace: from call site
at: (79:27) in file: /nix/store/kcv8hh2nrrpqm6d04rvw5dccrd5fdrhr-source/lib/customisation.nix

    78|       # Change the result of the function call by applying g to it
    79|       overrideResult = g: makeOverridable (copyArgs (args: g (f args))) origArgs;
      |                           ^
    80|     in

trace: while evaluating 'overrideResult'
at: (79:24) in file: /nix/store/kcv8hh2nrrpqm6d04rvw5dccrd5fdrhr-source/lib/customisation.nix

    78|       # Change the result of the function call by applying g to it
    79|       overrideResult = g: makeOverridable (copyArgs (args: g (f args))) origArgs;
      |                        ^
    80|     in

trace: from call site
at: (86:13) in file: /nix/store/kcv8hh2nrrpqm6d04rvw5dccrd5fdrhr-source/lib/customisation.nix

    85|           ${if result ? overrideAttrs then "overrideAttrs" else null} = fdrv:
    86|             overrideResult (x: x.overrideAttrs fdrv);
      |             ^
    87|         }

trace: while evaluating 'overrideAttrs'
at: (85:73) in file: /nix/store/kcv8hh2nrrpqm6d04rvw5dccrd5fdrhr-source/lib/customisation.nix

    84|           overrideDerivation = fdrv: overrideResult (x: overrideDerivation x fdrv);
    85|           ${if result ? overrideAttrs then "overrideAttrs" else null} = fdrv:
      |                                                                         ^
    86|             overrideResult (x: x.overrideAttrs fdrv);

trace: from call site
at: (14:15) in file: /nix/store/zc822dgnnkdpssb9vdgzq8fbgj0iyg8f-source/mach_nix/nix/withDot.nix

    13|       drv = if attr == "" then pyEnvBase else pyEnvBase."${attr}";
    14|       pyEnv = drv.overrideAttrs (oa: {
      |               ^
    15|         passthru =

trace: while evaluating 'gen'
at: (5:15) in file: /nix/store/zc822dgnnkdpssb9vdgzq8fbgj0iyg8f-source/mach_nix/nix/withDot.nix

     4|   names = pypiFetcher.allNames;
     5|   gen = attr: selected:
      |               ^
     6|     let

trace: from call site
at: (26:22) in file: /nix/store/zc822dgnnkdpssb9vdgzq8fbgj0iyg8f-source/mach_nix/nix/withDot.nix

    25|  "shellWith" = gen "env" [];
    26|  "dockerImageWith" = gen "dockerImage" [];
      |                      ^
    27| }

trace: while evaluating the attribute 'dockerImageWith'
at: (26:2) in file: /nix/store/zc822dgnnkdpssb9vdgzq8fbgj0iyg8f-source/mach_nix/nix/withDot.nix

    25|  "shellWith" = gen "env" [];
    26|  "dockerImageWith" = gen "dockerImage" [];
      |  ^
  [ 9347.781168] serial8250: too much work for irq4
  27| }

trace: while evaluating the attribute 'dockerImageWith'
at: (88:3) in file: /nix/store/zc822dgnnkdpssb9vdgzq8fbgj0iyg8f-source/default.nix

    87|   shellWith = (withDot (__mkPython "'.shellWith'")).shellWith;
    88|   dockerImageWith = (withDot (__mkPython "'.dockerImageWith'")).dockerImageWith;
      |   ^
    89| 

trace: while evaluating the attribute 'dockerImageWith'
at: (23:37) in file: /nix/store/zc822dgnnkdpssb9vdgzq8fbgj0iyg8f-source/flake.nix

    22|         packages = flake-utils.lib.flattenTree rec {
    23|           inherit (mach-nix-default)
      |                                     ^
    24|             mach-nix

trace: while evaluating 'op'
at: (3:19) in file: /nix/store/7zj2cs6ka320hplg4ia7rlmgy54z5war-source/flattenTree.nix

     2| let
     3|   op = sum: path: val:
      |                   ^
     4|     let

trace: from call site
at: (30:18) in file: /nix/store/7zj2cs6ka320hplg4ia7rlmgy54z5war-source/flattenTree.nix

    29|     builtins.foldl'
    30|       (sum: key: op sum (path ++ [ key ]) val.${key})
      |                  ^
    31|       sum

trace: while evaluating anonymous lambda
at: (30:13) in file: /nix/store/7zj2cs6ka320hplg4ia7rlmgy54z5war-source/flattenTree.nix

    29|     builtins.foldl'
    30|       (sum: key: op sum (path ++ [ key ]) val.${key})
      |             ^
    31|       sum

trace: from call site
at: (29:5) in file: /nix/store/7zj2cs6ka320hplg4ia7rlmgy54z5war-source/flattenTree.nix

    28|   recurse = sum: path: val:
    29|     builtins.foldl'
      |     ^
    30|       (sum: key: op sum (path ++ [ key ]) val.${key})

trace: while evaluating 'recurse'
at: (28:24) in file: /nix/store/7zj2cs6ka320hplg4ia7rlmgy54z5war-source/flattenTree.nix

    27| 
    28|   recurse = sum: path: val:
      |                        ^
    29|     builtins.foldl'

trace: from call site
at: (35:1) in file: /nix/store/7zj2cs6ka320hplg4ia7rlmgy54z5war-source/flattenTree.nix

    34| in
    35| recurse { } [ ] tree
      | ^

trace: while evaluating anonymous lambda
at: (1:1) in file: /nix/store/7zj2cs6ka320hplg4ia7rlmgy54z5war-source/flattenTree.nix

     1| tree:
      | ^
     2| let

trace: from call site
at: (108:23) in file: /nix/store/7zj2cs6ka320hplg4ia7rlmgy54z5war-source/default.nix

   107|   #   }
   108|   flattenTree = tree: import ./flattenTree.nix tree;
      |                       ^
   109| 

trace: while evaluating 'flattenTree'
at: (108:17) in file: /nix/store/7zj2cs6ka320hplg4ia7rlmgy54z5war-source/default.nix

   107|   #   }
   108|   flattenTree = tree: import ./flattenTree.nix tree;
      |                 ^
   109| 

trace: from call site
at: (22:20) in file: /nix/store/zc822dgnnkdpssb9vdgzq8fbgj0iyg8f-source/flake.nix

    21|         };
    22|         packages = flake-utils.lib.flattenTree rec {
      |                    ^
    23|           inherit (mach-nix-default)

trace: while evaluating the attribute 'packages'
at: (22:9) in file: /nix/store/zc822dgnnkdpssb9vdgzq8fbgj0iyg8f-source/flake.nix

    21|         };
    22|         packages = flake-utils.lib.flattenTree rec {
      |         ^
    23|           inherit (mach-nix-default)

More details

I have found two this environment that it (mach-nix too with the same pattern) is broken.
Broke in (with the same error message) :

• QEMU with cloud image ubuntu + nix + flake (the one above that i described);
• Pop!_OS 20.04 LTS of my peer at work with nix + flake.

Works in (all using flake):

• OCI image from lnl7/nix:2.3.7 running with docker and host NixOS;
• OCI image from lnl7/nix:2.3.7 running with docker and host ubuntu:20.04;
• OCI image from lnl7/nix:2.3.7 running with podman and host ubuntu:20.04;
• In the same QEMU with cloud image ubuntu:18.04 that does not work with nix+ flake, but in this case using docker/podman to run the OCI image from lnl7/nix:2.3.7 i even tried others images, they all worked too!

How this is possible? I mean, how the same QEMU VM that does not work if the flake is ran from the nix in the VM fail, but if in this same VM is used docker/podman to run an OCI image that has nix inside it, and this nix inside it is used it works!

This weird thing made me have to say to my peer: "well, in all these environments it works", which is a version of "in my machine it works". It was really embarrassing/funny given that i am trying to improve the reproducibility using nix to do it and nix sells it self as a reproducibility tool. Just to be clear, i really like nix, i just does not understand how to troubleshoot this error to solve it.

Edit: the Pop!_OS problem was an mistake, it in fact was another problem. It was a No space left on device error. We manage after a long search to find this issue in NixOS/nix: Single-user install should not use /run/user/ for build directories. #2957, what solved the error in the Pop!_OS was to export TMPDIR=/tmp before run either nix develop ... or nix build ..., we are doing it manually for now to solve the error.