how are the tar.xz archives created? are they reproducible?

[milahu]

i want to use nix to build the freetz build environment
similar to nix-openwrt-imagebuilder

kconfig.nix

stdenv.mkDerivation rec {
  pname = "kconfig";

  # $(call TOOLS_INIT, v6.9)
  version = "6.9";

  # $(PKG)_SITE:=git_archive@git://repo.or.cz/linux.git,scripts/basic,scripts/kconfig,...
  src = fetchgit {
    #url = "git://repo.or.cz/linux.git";
    url = "https://github.com/torvalds/linux";
    rev = "v${version}";
    nonConeMode = true; # strict sparse checkout
    sparseCheckout = [
      "scripts/basic"
      "scripts/kconfig"
      "scripts/Kbuild.include"
      "scripts/Makefile.compiler"
      "scripts/Makefile.build"
      "scripts/Makefile.host"
      "scripts/Makefile.lib"
      "Documentation/kbuild/kconfig-language.rst"
      "Documentation/kbuild/kconfig-macro-language.rst"
      "Documentation/kbuild/kconfig.rst"
    ];

    # $(PKG)_SOURCE:=kconfig-$($(PKG)_VERSION).tar.xz
    # $(PKG)_HASH:=48a0e5e5ce72f0b23a868e2ce784033fadfdc7b3b199be5ef6efbfca02e0be70

    # $ sha256sum kconfig-v6.9.tar.xz
    # 48a0e5e5ce72f0b23a868e2ce784033fadfdc7b3b199be5ef6efbfca02e0be70  kconfig-v6.9.tar.xz

    #hash = "sha256:f7736f3ce00dfd0cd15f6ff025fd76fbfadd4e6348aad1ec0bfe28ce94017b69";
    hash = "sha256-93NvPOAN/QzRX2/wJf12+/rdTmNIqtHsC/4ozpQBe2k=";
  };

problem: i cannot reproduce the hash of kconfig-v6.9.tar.xz

i want to re-use the $(PKG)_HASH from kconfig-host.mk

$(call TOOLS_INIT, v6.9)
## patches/100-main_makefile.patch contains also the version
$(PKG)_SOURCE:=kconfig-$($(PKG)_VERSION).tar.xz
$(PKG)_HASH:=48a0e5e5ce72f0b23a868e2ce784033fadfdc7b3b199be5ef6efbfca02e0be70
$(PKG)_SITE:=git_archive@git://repo.or.cz/linux.git,scripts/basic,scripts/kconfig,scripts/Kbuild.include,scripts/Makefile.compiler,scripts/Makefile.build,scripts/Makefile.host,scripts/Makefile.lib,Documentation/kbuild/kconfig-language.rst,Documentation/kbuild/kconfig-macro-language.rst,Documentation/kbuild/kconfig.rst

nix-hash produces these hashes
which are used in fetchgit { hash = "..."; }

wget https://github.com/Freetz-NG/dl-mirror/raw/master/kconfig-v6.9.tar.xz

sha256sum kconfig-v6.9.tar.xz
# 48a0e5e5ce72f0b23a868e2ce784033fadfdc7b3b199be5ef6efbfca02e0be70  kconfig-v6.9.tar.xz

tar xf kconfig-v6.9.tar.xz

nix-hash --type sha256 kconfig-v6.9/
# f7736f3ce00dfd0cd15f6ff025fd76fbfadd4e6348aad1ec0bfe28ce94017b69

nix-hash --type sha256 --base64 kconfig-v6.9/
# 93NvPOAN/QzRX2/wJf12+/rdTmNIqtHsC/4ozpQBe2k=

fetchgit produces the exact same files

nix-build . -A freetz.pkgs.kconfig.src
# /nix/store/x7zkharaxhb5yw7jzm2p1wnj034zv5jn-linux

nix-hash --type sha256 --base64 /nix/store/x7zkharaxhb5yw7jzm2p1wnj034zv5jn-linux
# 93NvPOAN/QzRX2/wJf12+/rdTmNIqtHsC/4ozpQBe2k=

nix-hash --type sha256 --base64 kconfig-v6.9
# 93NvPOAN/QzRX2/wJf12+/rdTmNIqtHsC/4ozpQBe2k=

ideally freetz would use nix hashes : P

but i just need a way to reproduce the kconfig-v6.9.tar.xz archive
from the files downloaded from git

tar.xz hashes

wget https://github.com/Freetz-NG/dl-mirror/raw/master/kconfig-v6.9.tar.xz

tar xf kconfig-v6.9.tar.xz

sha256sum kconfig-v6.9.tar.xz
# 48a0e5e5ce72f0b23a868e2ce784033fadfdc7b3b199be5ef6efbfca02e0be70  kconfig-v6.9.tar.xz

SOURCE_DATE_EPOCH="1970-01-01 00:00:00 UTC"

tar --sort=name \
      --mtime="${SOURCE_DATE_EPOCH}" \
      --owner=0 --group=0 --numeric-owner \
      --pax-option=exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime \
      -cf kconfig-v6.9.tar kconfig-v6.9

for level in {0..9}; do xz -k -$level -c kconfig-v6.9.tar > kconfig-v6.9.level$level.tar.xz || break; done

sha256sum *.tar.xz | sort

these hashes dont match 48a0e5e5...

0ecae1ab36bd31c06a1620fa226ef7ecdac06b33eef877110dd88048b81ca52a  kconfig-v6.9.level7.tar.xz
109c1b4ec846f0d40b28d3f45ce8f446904cb7326e47bd4edc82c712f04db780  kconfig-v6.9.level4.tar.xz
1933b5491ab92f8099d3c4f1deea5af49b26c70eb30dfff6ef09004bd0d78ea2  kconfig-v6.9.level6.tar.xz
4ed24dbb54acfdab7ea7f33d83dbe1ffdd8c3a932ed7419812930a10eb6684f6  kconfig-v6.9.level1.tar.xz
5dd3f1995a0c4b28940d3e3d514460d5a936d173afe8bcd956f0ed8cff3c8678  kconfig-v6.9.level8.tar.xz
6e127f1183d1b8913010b2420bd9ecdaa94caa278e41899685a3eccedef5e4c6  kconfig-v6.9.level0.tar.xz
a518045af91d5b1c91673f0911b5defbfbea9b305cd621553dcba1c91c8ae9f9  kconfig-v6.9.level2.tar.xz
b3b104e54c92ce3c8ace240ed605bd75b31cb41e127de5b4be7f40a95660a705  kconfig-v6.9.level3.tar.xz
c27a79c297b22cce52c91e038266a1c85ba6533841f612fdbc920908759fd007  kconfig-v6.9.level5.tar.xz
d7f3869888a77214c5007b2709eef236697b800e7713f5c194b75ad9d33839ea  kconfig-v6.9.level9.tar.xz

see also

• https://reproducible-builds.org/docs/archives/
• How to create a tar file that omits timestamps for its contents?

[fda77]

I dont know "nix", try it and tell me if it is working
The archive files are created if not found (and shows the new hash), every created file has its own hash, but the content is identical (if the git tag "v6.9" is unchanged).
So get the file with hash 48a0e5e5ce72f0b23a868e2ce784033fadfdc7b3b199be5ef6efbfca02e0be70 and compare it with yours

[fda77]

Btw, you could set in menuconfig to prefer CVS checkouts before (untrusted) archive files, but it could be slow