From 78a1a8923475e93bd527364c55fa426d366d6c68 Mon Sep 17 00:00:00 2001 From: Gonzalo Reyero Ferreras Date: Mon, 5 Feb 2024 04:52:56 -0600 Subject: [PATCH] Step to use /mnt for docker storage. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The job "smoke-tests-container" of the "Test Incoming Changes" workflow was running out of disk space again. Debugging with tmate I saw that there's another 75GB disk mounted (sdb) in /mnt that's not being used. runner@fv-az975-229:~/work/cnf-certification-test/cnf-certification-test$ df -h Filesystem Size Used Avail Use% Mounted on /dev/root 73G 73G 463M 100% / tmpfs 7.9G 172K 7.9G 1% /dev/shm tmpfs 3.2G 1.6M 3.2G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock /dev/sda15 105M 6.1M 99M 6% /boot/efi /dev/sdb1 74G 4.1G 66G 6% /mnt tmpfs 1.6G 12K 1.6G 1% /run/user/1001 runner@fv-az975-229:~/work/cnf-certification-test/cnf-certification-test$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 111.9M 1 loop /snap/lxd/24322 loop1 7:1 0 63.9M 1 loop /snap/core20/2105 loop2 7:2 0 40.4M 1 loop /snap/snapd/20671 sda 8:0 0 75G 0 disk ├─sda1 8:1 0 74.9G 0 part / ├─sda14 8:14 0 4M 0 part └─sda15 8:15 0 106M 0 part /boot/efi sdb 8:16 0 75G 0 disk └─sdb1 8:17 0 75G 0 part /mnt I've reconfigured the docker daemon to use a newly created folder /mnt/docker-storage as storage folder, so all the images (including the ones kind uses) should be stored there. With this change, before the cnf test suite starts, this is the new disk usage: runner@fv-az570-650:~/work/cnf-certification-test/cnf-certification-test$ df -h Filesystem Size Used Avail Use% Mounted on /dev/root 73G 54G 20G 74% / tmpfs 7.9G 172K 7.9G 1% /dev/shm tmpfs 3.2G 1.6M 3.2G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock /dev/sda15 105M 6.1M 99M 6% /boot/efi /dev/sdb1 74G 23G 48G 32% /mnt tmpfs 1.6G 12K 1.6G 1% /run/user/1001 The kind cluster and its containers are now createded in /mnt, so now there's 20G of free space left in /. Added the same workaround for the "smoke-tests-local" job. --- .github/workflows/pre-main.yaml | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/.github/workflows/pre-main.yaml b/.github/workflows/pre-main.yaml index 7f672d5ae..a9642fb6b 100644 --- a/.github/workflows/pre-main.yaml +++ b/.github/workflows/pre-main.yaml @@ -232,6 +232,20 @@ jobs: max_attempts: 3 command: cd ${GITHUB_WORKSPACE}/cnf-certification-test-partner && make bootstrap-cluster && make bootstrap-docker-ubuntu-local && make bootstrap-python-ubuntu-local && pip3 install pyyaml + # Restart docker using /mnt/docker-storage (sdb) instead of /var/lib/docker (sda). + # This step needs to be done right after the partner repo's bootstrap scripts, as they + # overwrite the docker's daemon.json. + - name: Make docker to use /mnt (sdb) for storage + run: | + df -h + lsblk + sudo mkdir /mnt/docker-storage + sudo jq '. +={"data-root" : "/mnt/docker-storage"}' < /etc/docker/daemon.json > /tmp/docker-daemon.json + sudo cp /tmp/docker-daemon.json /etc/docker/daemon.json + cat /etc/docker/daemon.json + sudo systemctl restart docker + sudo ls -la /mnt/docker-storage + - name: Run 'make rebuild-cluster' uses: nick-fields/retry@v3 with: @@ -327,6 +341,20 @@ jobs: max_attempts: 3 command: cd ${GITHUB_WORKSPACE}/cnf-certification-test-partner && make bootstrap-cluster && make bootstrap-docker-ubuntu-local && make bootstrap-python-ubuntu-local && pip3 install pyyaml + # Restart docker using /mnt/docker-storage (sdb) instead of /var/lib/docker (sda). + # This step needs to be done right after the partner repo's bootstrap scripts, as they + # overwrite the docker's daemon.json. + - name: Make docker to use /mnt (sdb) for storage + run: | + df -h + lsblk + sudo mkdir /mnt/docker-storage + sudo jq '. +={"data-root" : "/mnt/docker-storage"}' < /etc/docker/daemon.json > /tmp/docker-daemon.json + sudo cp /tmp/docker-daemon.json /etc/docker/daemon.json + cat /etc/docker/daemon.json + sudo systemctl restart docker + sudo ls -la /mnt/docker-storage + - name: Run 'make rebuild-cluster' uses: nick-fields/retry@v3 with: