OpenCV fails to fetch after enabling Download/SState Mirroring

[joekale-pp]

After following the directions from the S3 mirroring wiki post, most packages are properly uploading to and downloading from my s3 bucket mirror. Now, OpenCV fails to fetch.

Relevant values from local.conf:

BB_HASHSERVE = "localhost:8686"
BB_SIGNATURE_HANDLER = "OEEquivHash"

INHERIT += "own-mirrors"
SOURCE_MIRROR_URL = "s3://<bucket>/downloads"
SSTATE_MIRRORS = "\
    file://.* s3://<bucket>/sstate/PATH;downloadfilename=PATH"

INHERIT += "mirror_updates"
BB_GENERATE_MIRROR_TARBALLS = "1" 
UPDATE_DOWNLOADS_MIRROR = "1" 
#BB_FETCH_PREMIRRORONLY = "1" # Been turning off and on to diagnose
DOWNLOADS_MIRROR_URL = "s3://<bucket>/downloads"
UPDATE_SSTATE_MIRROR = "1" 
SSTATE_MIRROR_URL = "s3://<bucket>/sstate"

OE_IMPORTS += "oeaws.botos3fetcher"
BB_ENV_PASSTHROUGH_ADDITIONS="AWS_CONFIG_FILE AWS_PROFILE AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SHARED_CREDENTIALS_FILE AWS_SESSION_TOKEN AWS_DEFAULT_REGION"

I'm using SSO login so the only environment variable I've been setting is AWS_PROFILE even though all are passed through.

First Issue with fetching OpenCV: git fails to find /usr/share/git-core/templates/description. It does exist with 644 permissions

DEBUG: Trying Upstream
DEBUG: Fetcher accessed the network with the command LANG=C git -c core.fsyncobjectfiles=0 -c gc.autoDetach=false -c core.pager=cat clone --bare --mirror https://github.com/opencv/opencv.git /home/joe/dev/oe-vision/build/downloads/git2/github.com.opencv.opencv.git --progress
DEBUG: Running export PSEUDO_DISABLED=1; export AWS_PROFILE="rd-pwr-user"; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export PATH="/home/joe/dev/oe-vision/build/tmp/work/armv8a_tegra194-vision-linux/opencv/4.5.5-r0/recipe-sysroot-native/usr/local/cuda-10.2/bin:/home/joe/dev/oe-vision/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/joe/dev/oe-vision/build/tmp/work/armv8a_tegra194-vision-linux/opencv/4.5.5-r0/recipe-sysroot-native/usr/bin/python3-native:/home/joe/dev/oe-vision/layers/scripts:/home/joe/dev/oe-vision/build/tmp/work/armv8a_tegra194-vision-linux/opencv/4.5.5-r0/recipe-sysroot-native/usr/bin/aarch64-vision-linux:/home/joe/dev/oe-vision/build/tmp/work/armv8a_tegra194-vision-linux/opencv/4.5.5-r0/recipe-sysroot/usr/bin/crossscripts:/home/joe/dev/oe-vision/build/tmp/work/armv8a_tegra194-vision-linux/opencv/4.5.5-r0/recipe-sysroot-native/usr/sbin:/home/joe/dev/oe-vision/build/tmp/work/armv8a_tegra194-vision-linux/opencv/4.5.5-r0/recipe-sysroot-native/usr/bin:/home/joe/dev/oe-vision/build/tmp/work/armv8a_tegra194-vision-linux/opencv/4.5.5-r0/recipe-sysroot-native/sbin:/home/joe/dev/oe-vision/build/tmp/work/armv8a_tegra194-vision-linux/opencv/4.5.5-r0/recipe-sysroot-native/bin:/home/joe/dev/oe-vision/repos/poky/bitbake/bin:/home/joe/dev/oe-vision/build/tmp/hosttools"; export HOME="/home/joe"; LANG=C git -c core.fsyncobjectfiles=0 -c gc.autoDetach=false -c core.pager=cat clone --bare --mirror https://github.com/opencv/opencv.git /home/joe/dev/oe-vision/build/downloads/git2/github.com.opencv.opencv.git --progress
Cloning into bare repository '/home/joe/dev/oe-vision/build/downloads/git2/github.com.opencv.opencv.git'...
fatal: cannot copy '/usr/share/git-core/templates/description' to '/home/joe/dev/oe-vision/build/downloads/git2/github.com.opencv.opencv.git/description': No such file or directory

Second issue is harder to explain, but after the above fails other mirrors are tried. Those mirrors download a git2_github.com.opencv.opencv.git.tar.gz file, but after they are downloaded a tar command runs to unpack it to the downloads/git2 directory and the tar command fails to find the downloaded tarball as it has either moved or been deleted so the fetch is considered to have failed and the fetcher tries another mirror which repeats the same results.

last commit that I've rebased my fork of tegra-demo-distro on is 9db9d34d477849fafb5343aecd0adab76316c10b

[madisongh]

That's certainly strange, and an error I've not run into before. The No such file or directory error could be about the target directory rather than the source file, which makes me suspect either some kind of race somewhere, or possibly an issue with the filesystem you're running the builds on.

If it's readily reproducible, could you try running that task under strace to see exactly which syscall is reporting the error? Something like

$ strace -ff -s512 -ofetch-strace bitbake -c fetch opencv

should generate fetch-strace.<pid> files for every subprocess.

Also, are you using ext4 filesystem for your build tree, or something else? Have you run fsck on that filesystem recently?

[joekale-pp]

Here is the git-fail strace.
fetch-strace.2999458.git-fail.txt

[madisongh]

Very strange. There's a mkdir() call to create the directory, which is successful, but the openat() call shortly thereafter to create the description file in that directory is returning ENOENT.

[joekale-pp]

I have previously, while messing with this issue, deleted the entire build folder and restarted the entire process to build opencv from just the local.conf and I have hit the same errors each time.

[madisongh]

Definitely makes me suspect that the filesystem is corrupted.

[joekale-pp]

I'll run fsck on it and hopefully have some improvement. I'll report back when it's done if the error presents itself again.