Skip to content

Commit

Permalink
Step to use /mnt for docker storage.
Browse files Browse the repository at this point in the history
The job "smoke-tests-container" of the "Test Incoming Changes" workflow
was running out of disk space again. Debugging with tmate I saw that
there's another 75GB disk mounted (sdb) in /mnt that's not being used.

runner@fv-az975-229:~/work/cnf-certification-test/cnf-certification-test$
df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        73G   73G  463M 100% /
tmpfs           7.9G  172K  7.9G   1% /dev/shm
tmpfs           3.2G  1.6M  3.2G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda15      105M  6.1M   99M   6% /boot/efi
/dev/sdb1        74G  4.1G   66G   6% /mnt
tmpfs           1.6G   12K  1.6G   1% /run/user/1001

runner@fv-az975-229:~/work/cnf-certification-test/cnf-certification-test$
lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0     7:0    0 111.9M  1 loop /snap/lxd/24322
loop1     7:1    0  63.9M  1 loop /snap/core20/2105
loop2     7:2    0  40.4M  1 loop /snap/snapd/20671
sda       8:0    0    75G  0 disk
├─sda1    8:1    0  74.9G  0 part /
├─sda14   8:14   0     4M  0 part
└─sda15   8:15   0   106M  0 part /boot/efi
sdb       8:16   0    75G  0 disk
└─sdb1    8:17   0    75G  0 part /mnt

I've reconfigured the docker daemon to use a newly created folder
/mnt/docker-storage as storage folder, so all the images (including the
ones kind uses) should be stored there.

With this change, before the cnf test suite starts, this is the new disk
usage:
runner@fv-az570-650:~/work/cnf-certification-test/cnf-certification-test$
df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        73G   54G   20G  74% /
tmpfs           7.9G  172K  7.9G   1% /dev/shm
tmpfs           3.2G  1.6M  3.2G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda15      105M  6.1M   99M   6% /boot/efi
/dev/sdb1        74G   23G   48G  32% /mnt
tmpfs           1.6G   12K  1.6G   1% /run/user/1001

The kind cluster and its containers are now createded in /mnt, so
now there's 20G of free space left in /.

Added the same workaround for the "smoke-tests-local" job.
  • Loading branch information
greyerof committed Feb 5, 2024
1 parent b18404c commit 78a1a89
Showing 1 changed file with 28 additions and 0 deletions.
28 changes: 28 additions & 0 deletions .github/workflows/pre-main.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,20 @@ jobs:
max_attempts: 3
command: cd ${GITHUB_WORKSPACE}/cnf-certification-test-partner && make bootstrap-cluster && make bootstrap-docker-ubuntu-local && make bootstrap-python-ubuntu-local && pip3 install pyyaml

# Restart docker using /mnt/docker-storage (sdb) instead of /var/lib/docker (sda).
# This step needs to be done right after the partner repo's bootstrap scripts, as they
# overwrite the docker's daemon.json.
- name: Make docker to use /mnt (sdb) for storage
run: |
df -h
lsblk
sudo mkdir /mnt/docker-storage
sudo jq '. +={"data-root" : "/mnt/docker-storage"}' < /etc/docker/daemon.json > /tmp/docker-daemon.json
sudo cp /tmp/docker-daemon.json /etc/docker/daemon.json
cat /etc/docker/daemon.json
sudo systemctl restart docker
sudo ls -la /mnt/docker-storage
- name: Run 'make rebuild-cluster'
uses: nick-fields/retry@v3
with:
Expand Down Expand Up @@ -327,6 +341,20 @@ jobs:
max_attempts: 3
command: cd ${GITHUB_WORKSPACE}/cnf-certification-test-partner && make bootstrap-cluster && make bootstrap-docker-ubuntu-local && make bootstrap-python-ubuntu-local && pip3 install pyyaml

# Restart docker using /mnt/docker-storage (sdb) instead of /var/lib/docker (sda).
# This step needs to be done right after the partner repo's bootstrap scripts, as they
# overwrite the docker's daemon.json.
- name: Make docker to use /mnt (sdb) for storage
run: |
df -h
lsblk
sudo mkdir /mnt/docker-storage
sudo jq '. +={"data-root" : "/mnt/docker-storage"}' < /etc/docker/daemon.json > /tmp/docker-daemon.json
sudo cp /tmp/docker-daemon.json /etc/docker/daemon.json
cat /etc/docker/daemon.json
sudo systemctl restart docker
sudo ls -la /mnt/docker-storage
- name: Run 'make rebuild-cluster'
uses: nick-fields/retry@v3
with:
Expand Down

0 comments on commit 78a1a89

Please sign in to comment.