Update source checksums across the distro #909
Comments
|
|
Doing this by hand is going to be nigh impossible and take forever. |
|
Yup, item (2) is easily scriptable but item (1) is mindless but only partially scriptable at best. Assuming SHA256 is state-of-the-art, should we set a goal of at least always using it for new packages and switching to it whenever we upgrade to a new version? |
|
Definitely should use SHA256 on any new packages and preferably on updates as well. Scripting to |
|
Also, this seems to work on most things that only have #!/bin/bash
# SOURCE_LEVEL="" (empty) for the primary source
SOURCE_LEVEL="2"
for i in `ls -1 *.info`; do
echo "transitioning $i"
unset PKG_SOURCE PKG_FULL_SOURCE PKG_SOURCE_MD5 PKG_INFO_MD5
if ! grep -q -Li "^Source${SOURCE_LEVEL}-Checksum:" $i; then
# only look in files not using Source-Checksum (can have it and Source-MD5, so don't just search for Source-MD5)
PKG_NAME=$(grep -i -m 1 ^Package: $i | cut -f 2 -d:)
if [[ "$PKG_NAME" == *"type_"* ]]; then
echo -e "File $i has varianted packages $PKG_NAME. Exiting...\n"
continue
fi
if [[ $(grep -i -m 1 -c "^Type: bundle" $i) == 1 ]]; then
echo -e "$PKG_NAME is a bundle with no source. Exiting...\n"
continue
fi
if grep -i "^Distribution:" $i | grep -q -v "10.14.5"; then
echo -e "$PKG_NAME Not available in this dist. Exiting...\n"
continue
fi
if grep -q -li "^Source${SOURCE_LEVEL}Rename:" $i; then
echo "Pkg uses Source${SOURCE_LEVEL}Rename"
PKG_SOURCE=$(fink dumpinfo -fsource${SOURCE_LEVEL}rename $PKG_NAME | cut -f 2 -d' ')
else
PKG_FULL_SOURCE=$(fink dumpinfo -fsource${SOURCE_LEVEL} $PKG_NAME | cut -f 2 -d' ')
if [[ -z $PKG_FULL_SOURCE ]]; then
echo -e "$PKG_NAME does not have a source${SOURCE_LEVEL} file. Exiting...\n"
continue
elif [[ "$PKG_FULL_SOURCE" == "mirror:"* ]]; then
echo "$PKG_NAME uses a mirror source"
PKG_SOURCE=$(echo $PKG_FULL_SOURCE | rev | cut -f 1 -d: | rev | xargs basename)
else
echo "just trim the basename"
PKG_SOURCE=$(basename $PKG_FULL_SOURCE)
fi
fi
echo "PKG_FULL_SOURCE: $PKG_FULL_SOURCE"
echo "PKG_SOURCE: $PKG_SOURCE"
if [ ! -f /sw/src/$PKG_SOURCE ]; then
fink -y fetch $PKG_NAME
fi
if [ -f /sw/src/$PKG_SOURCE ]; then
PKG_SOURCE_MD5=$(md5sum /sw/src/$PKG_SOURCE | cut -f 1 -d' ')
fi
PKG_INFO_MD5=$(grep -i -m 1 ^Source${SOURCE_LEVEL}-MD5 $i | cut -f 2 -d:)
echo "Do the MD5 sums agree?"
echo "PKG_SOURCE_MD5: $PKG_SOURCE_MD5"
echo "PKG_INFO_MD5: $PKG_INFO_MD5"
if [ $PKG_INFO_MD5 = $PKG_SOURCE_MD5 ]; then
PKG_SOURCE_SHA256=$(shasum -a 256 /sw/src/$PKG_SOURCE | cut -f 1 -d' ')
echo "replacing Source${SOURCE_LEVEL}-MD5 with Source${SOURCE_LEVEL}-256"
perl -pi -e "s|^[sS]ource${SOURCE_LEVEL}-[mM][dD]5: (.*)$|Source${SOURCE_LEVEL}-Checksum: SHA256($PKG_SOURCE_SHA256)|g" $i
else
echo "MD5 of tarball doesn't match .info. Will log this to ~/fink-md5-check.log"
echo "$PKG_NAME : $PKG_SOURCE $PKG_SOURCE_MD5 $PKG_INFO_MD5 do not match" >> ~/fink-md5-check.log
fi
else
echo "Package already uses Source-Checksum. Nothing to change"
fi
echo ""
done
echo "Done parsing folder."There are probably cuter ways to do it via perl and directly calling the package manager for package and file names, but we don't need perfect. Just good enough to get the bulk of the packages. |
|
The pull request about CRAN packages have SHA256 for new versions. Can anyone check and approve it? |
done |
|
I have an error: |
Fixed. The download path changed and what you were getting is just (nor redirecting) HTML file. I've added upstream's mirror. The source still exists in our own master mirror, so can also get it from there. |
|
The asciidoc source I just tried to download has slightly changed from the one I had downloaded last year, with conflicting shasums of course: on all mirrors I could reach and the original URL. |
|
can you unpack them into separate directories and see what's the difference? Sometimes there's a silent update that's just a repackaging and that changes the checksum. |
|
Uh, the source directory has changed to The Python files and the html*.conf, that's it – still great for not even changing the bugfix release number. |
|
Looks like upstream moved from github/asciidoc to github/asciidoc-py, but that's lame to silently change older tags to python3. Probably should deal with fixing this in another issue/PR since it's no longer about just the SHA256. |
|
Indeed, brilliant! |
|
devel/lazarus-doc.info @kamischi These are the only packages left in !base still using |
|
These are the only packages left in !base that are using gdcm-2.4.5.info (removed upstream) |
Thanks for doing the heavy lifting on this ticket! I support killing gdcm-2.4.5 altogether, as it's an older libversion and has a java dependency (unlike the newer libversion). There's only one dependant, I'll look af switching it up. UPDATE: I killed the java dep in the older gcdm. The dependant, insighttoolkit45, FTBFS even on my 10.13 with a ton of C++ errors, so I can't work on scrapping old gdcm. Upstream has a newer insighttoolkit (maybe 413?) if someone wants to look at updating it. Mean time, I'll update the gdcm-2.4.5 to SHA256. |
Kill. I don't see it in other distros; explicitly abandoned upstream in 2018. |
Fink originally used MD5 for sources. Then we added support for SHA1 and SHA256. Many (maybe even most) packages still use MD5, and a bunch SHA1. Should we work on upgrading the checksum fields to SHA256? The current SHA256 checksum feature requires any one of:
so I think it's safe to assume it will work everywhere without hauling anything else into fink core or requiring adding a
BuildDepends.The text was updated successfully, but these errors were encountered: