Publishing Site poolpOrg/poolp.org at 80f5b18 on Wed Nov 6 22:03:31 U…

…TC 2024

posts/2023-10-07/plakar-a-ton-of-changes/index.html

@@ -246,7 +246,7 @@
 just not for this specific use-case.</p><h1 class="relative group">Merged <code>go-fastcdc</code> and <code>go-ultracdc</code> into <code>go-cdc-chunkers</code> and switched <code>plakar</code> to it<div id=merged-go-fastcdc-and-go-ultracdc-into-go-cdc-chunkers-and-switched-plakar-to-it class=anchor></div><span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#merged-go-fastcdc-and-go-ultracdc-into-go-cdc-chunkers-and-switched-plakar-to-it aria-label=Anchor>#</a></span></h1><p>Given that I already had two CDC implementations,
 both following the same interface,
 I decided to merge them into a common <code>go-cdc-chunkers</code> project which is ISC-licensed.</p><p>This will allow me to reduce my workload by maintaining a single project,
-while making it easier to experiment with new CDC chunkers and malking sure that general optimizations apply to all my implementations.</p><a id=github-f1b9f99b3585142e78475407390a4ae6 target=_blank href=https://github.com/PlakarLabs/go-cdc-chunkers class=cursor-pointer><div class="w-full md:w-auto pt-3 p-5 border border-neutral-200 dark:border-neutral-700 border rounded-md shadow-2xl"><div class="flex items-center"><span class="text-2xl text-neutral-800 dark:text-neutral" style=margin-right:10px><span class="relative block icon"><svg viewBox="0 0 496 512"><path fill="currentcolor" d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6.0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6.0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3.0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1.0-6.2-.3-40.4-.3-61.4.0.0-70 15-84.7-29.8.0.0-11.4-29.1-27.8-36.6.0.0-22.9-15.7 1.6-15.4.0.0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5.0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9.0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4.0 33.7-.3 75.4-.3 83.6.0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6.0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9.0-6.2-1.4-2.3-4-3.3-5.6-2z"/></svg></span></span><div id=github-f1b9f99b3585142e78475407390a4ae6-full_name class="m-0 font-bold text-xl text-neutral-800 decoration-primary-500 hover:underline hover:underline-offset-2 dark:text-neutral">PlakarLabs/go-cdc-chunkers</div></div><p id=github-f1b9f99b3585142e78475407390a4ae6-description class="m-0 mt-2 text-md text-neutral-800 dark:text-neutral">A Golang package that implements CDC chunkers with a generic interface</p><div class="m-0 mt-2 flex items-center"><span class="mr-1 inline-block h-3 w-3 rounded-full" style=background-color:#00add8></span><div class="m-0 mr-5 text-md text-neutral-800 dark:text-neutral">Go</div><span class="text-md mr-1 text-neutral-800 dark:text-neutral"><span class="relative block icon"><svg viewBox="0 0 576 512"><path fill="currentcolor" d="M287.9.0C297.1.0 305.5 5.25 309.5 13.52L378.1 154.8l153.3 22.7C540.4 178.8 547.8 185.1 550.7 193.7 553.5 202.4 551.2 211.9 544.8 218.2L433.6 328.4l26.3 155.5C461.4 492.9 457.7 502.1 450.2 507.4 442.8 512.7 432.1 513.4 424.9 509.1L287.9 435.9 150.1 509.1C142.9 513.4 133.1 512.7 125.6 507.4 118.2 502.1 114.5 492.9 115.1 483.9l27.1-155.5L31.11 218.2C24.65 211.9 22.36 202.4 25.2 193.7 28.03 185.1 35.5 178.8 44.49 177.5L197.7 154.8 266.3 13.52C270.4 5.249 278.7.0 287.9.0zm0 78.95L235.4 187.2C231.9 194.3 225.1 199.3 217.3 200.5L98.98 217.9 184.9 303C190.4 308.5 192.9 316.4 191.6 324.1L171.4 443.7l105.2-56.2C283.7 383.7 292.2 383.7 299.2 387.5l105.2 56.2-20.2-119.6C382.9 316.4 385.5 308.5 391 303l85.9-85.1-118.3-17.4C350.7 199.3 343.9 194.3 340.5 187.2L287.9 78.95z"/></svg></span></span><div id=github-f1b9f99b3585142e78475407390a4ae6-stargazers class="m-0 mr-5 text-md text-neutral-800 dark:text-neutral">10</div><span class="text-md mr-1 text-neutral-800 dark:text-neutral"><span class="relative block icon"><svg viewBox="0 0 448 512"><path fill="currentcolor" d="M80 104c13.3.0 24-10.7 24-24S93.3 56 80 56 56 66.7 56 80s10.7 24 24 24zm80-24c0 32.8-19.7 61-48 73.3V192c0 17.7 14.3 32 32 32H304c17.7.0 32-14.3 32-32V153.3C307.7 141 288 112.8 288 80c0-44.2 35.8-80 80-80s80 35.8 80 80c0 32.8-19.7 61-48 73.3V192c0 53-43 96-96 96H256v70.7c28.3 12.3 48 40.5 48 73.3.0 44.2-35.8 80-80 80s-80-35.8-80-80c0-32.8 19.7-61 48-73.3V288H144c-53 0-96-43-96-96V153.3C19.7 141 0 112.8.0 80 0 35.8 35.8.0 80 0s80 35.8 80 80zm208 24c13.3.0 24-10.7 24-24s-10.7-24-24-24-24 10.7-24 24 10.7 24 24 24zM248 432c0-13.3-10.7-24-24-24s-24 10.7-24 24 10.7 24 24 24 24-10.7 24-24z"/></svg></span></span><div id=github-f1b9f99b3585142e78475407390a4ae6-forks class="m-0 mr-5 text-md text-neutral-800 dark:text-neutral">2</div></div></div><script>fetch("https://api.github.com/repos/PlakarLabs/go-cdc-chunkers",{headers:new Headers({"User-agent":"Mozilla/4.0 Custom User Agent"})}).then(e=>e.json()).then(e=>{document.getElementById("github-f1b9f99b3585142e78475407390a4ae6-full_name").innerHTML=e.full_name,document.getElementById("github-f1b9f99b3585142e78475407390a4ae6-description").innerHTML=e.description,document.getElementById("github-f1b9f99b3585142e78475407390a4ae6-stargazers").innerHTML=e.stargazers_count,document.getElementById("github-f1b9f99b3585142e78475407390a4ae6-forks").innerHTML=e.forks}).catch(e=>console.error(e))</script></a><p>Now that I had <code>go-cdc-chunkers</code> available which would allow me to experiment with different CDC algorithms and settings,
+while making it easier to experiment with new CDC chunkers and malking sure that general optimizations apply to all my implementations.</p><a id=github-f1b9f99b3585142e78475407390a4ae6 target=_blank href=https://github.com/PlakarLabs/go-cdc-chunkers class=cursor-pointer><div class="w-full md:w-auto pt-3 p-5 border border-neutral-200 dark:border-neutral-700 border rounded-md shadow-2xl"><div class="flex items-center"><span class="text-2xl text-neutral-800 dark:text-neutral" style=margin-right:10px><span class="relative block icon"><svg viewBox="0 0 496 512"><path fill="currentcolor" d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6.0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6.0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3.0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1.0-6.2-.3-40.4-.3-61.4.0.0-70 15-84.7-29.8.0.0-11.4-29.1-27.8-36.6.0.0-22.9-15.7 1.6-15.4.0.0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5.0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9.0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4.0 33.7-.3 75.4-.3 83.6.0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6.0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9.0-6.2-1.4-2.3-4-3.3-5.6-2z"/></svg></span></span><div id=github-f1b9f99b3585142e78475407390a4ae6-full_name class="m-0 font-bold text-xl text-neutral-800 decoration-primary-500 hover:underline hover:underline-offset-2 dark:text-neutral">PlakarLabs/go-cdc-chunkers</div></div><p id=github-f1b9f99b3585142e78475407390a4ae6-description class="m-0 mt-2 text-md text-neutral-800 dark:text-neutral">A Golang package that implements CDC chunkers with a generic interface</p><div class="m-0 mt-2 flex items-center"><span class="mr-1 inline-block h-3 w-3 rounded-full" style=background-color:#00add8></span><div class="m-0 mr-5 text-md text-neutral-800 dark:text-neutral">Go</div><span class="text-md mr-1 text-neutral-800 dark:text-neutral"><span class="relative block icon"><svg viewBox="0 0 576 512"><path fill="currentcolor" d="M287.9.0C297.1.0 305.5 5.25 309.5 13.52L378.1 154.8l153.3 22.7C540.4 178.8 547.8 185.1 550.7 193.7 553.5 202.4 551.2 211.9 544.8 218.2L433.6 328.4l26.3 155.5C461.4 492.9 457.7 502.1 450.2 507.4 442.8 512.7 432.1 513.4 424.9 509.1L287.9 435.9 150.1 509.1C142.9 513.4 133.1 512.7 125.6 507.4 118.2 502.1 114.5 492.9 115.1 483.9l27.1-155.5L31.11 218.2C24.65 211.9 22.36 202.4 25.2 193.7 28.03 185.1 35.5 178.8 44.49 177.5L197.7 154.8 266.3 13.52C270.4 5.249 278.7.0 287.9.0zm0 78.95L235.4 187.2C231.9 194.3 225.1 199.3 217.3 200.5L98.98 217.9 184.9 303C190.4 308.5 192.9 316.4 191.6 324.1L171.4 443.7l105.2-56.2C283.7 383.7 292.2 383.7 299.2 387.5l105.2 56.2-20.2-119.6C382.9 316.4 385.5 308.5 391 303l85.9-85.1-118.3-17.4C350.7 199.3 343.9 194.3 340.5 187.2L287.9 78.95z"/></svg></span></span><div id=github-f1b9f99b3585142e78475407390a4ae6-stargazers class="m-0 mr-5 text-md text-neutral-800 dark:text-neutral">15</div><span class="text-md mr-1 text-neutral-800 dark:text-neutral"><span class="relative block icon"><svg viewBox="0 0 448 512"><path fill="currentcolor" d="M80 104c13.3.0 24-10.7 24-24S93.3 56 80 56 56 66.7 56 80s10.7 24 24 24zm80-24c0 32.8-19.7 61-48 73.3V192c0 17.7 14.3 32 32 32H304c17.7.0 32-14.3 32-32V153.3C307.7 141 288 112.8 288 80c0-44.2 35.8-80 80-80s80 35.8 80 80c0 32.8-19.7 61-48 73.3V192c0 53-43 96-96 96H256v70.7c28.3 12.3 48 40.5 48 73.3.0 44.2-35.8 80-80 80s-80-35.8-80-80c0-32.8 19.7-61 48-73.3V288H144c-53 0-96-43-96-96V153.3C19.7 141 0 112.8.0 80 0 35.8 35.8.0 80 0s80 35.8 80 80zm208 24c13.3.0 24-10.7 24-24s-10.7-24-24-24-24 10.7-24 24 10.7 24 24 24zM248 432c0-13.3-10.7-24-24-24s-24 10.7-24 24 10.7 24 24 24 24-10.7 24-24z"/></svg></span></span><div id=github-f1b9f99b3585142e78475407390a4ae6-forks class="m-0 mr-5 text-md text-neutral-800 dark:text-neutral">2</div></div></div><script>fetch("https://api.github.com/repos/PlakarLabs/go-cdc-chunkers",{headers:new Headers({"User-agent":"Mozilla/4.0 Custom User Agent"})}).then(e=>e.json()).then(e=>{document.getElementById("github-f1b9f99b3585142e78475407390a4ae6-full_name").innerHTML=e.full_name,document.getElementById("github-f1b9f99b3585142e78475407390a4ae6-description").innerHTML=e.description,document.getElementById("github-f1b9f99b3585142e78475407390a4ae6-stargazers").innerHTML=e.stargazers_count,document.getElementById("github-f1b9f99b3585142e78475407390a4ae6-forks").innerHTML=e.forks}).catch(e=>console.error(e))</script></a><p>Now that I had <code>go-cdc-chunkers</code> available which would allow me to experiment with different CDC algorithms and settings,
 I switched <code>plakar</code> to using it as a dependency rather than <code>go-fastcdc</code>.
 It defaults to the FastCDC implementation,
 but I no longer have to swap dependencies and imports all around whenever I want to test something.</p><h1 class="relative group">Made compression and hashing algorithm configurable<div id=made-compression-and-hashing-algorithm-configurable class=anchor></div><span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#made-compression-and-hashing-algorithm-configurable aria-label=Anchor>#</a></span></h1><p>By default,