Task/white238/reduce size

[white238]

This is not finished but I wanted to get some feedback before I went even farther down this path.

[vsoch]

You’ll want to keep the args for the commit, etc, that’s how we get an updated build commit.

I thought you mentioned you were planning to remove spack? We don’t remove from these base images because most want a development environment with it.

[vsoch]

To step back - if you are just creating and view and nuking spack, my suggestion would be to add a set of images that are "minimal" on top of the existing ones to do that. We should probably not provide one and not the other.

[white238]

@vsoch @davidbeckingsale

This now shows what I am proposing working on the clang@14 images. I tested this locally and it does work. It unfortunately did not reduce the size of the docker images quite as much as I would have hoped but it is still substantial (3.29GB down to 2.41GB).

This is mainly due to how large the llvm package is installed.

root@fa2646b1d1df:/opt# du -h -d 1 linux-ubuntu22.04-x86_64/gcc-11.4.0/
480K    linux-ubuntu22.04-x86_64/gcc-11.4.0/zlib-1.2.13-t5xrgb3g4rdup7g2kh
440K    linux-ubuntu22.04-x86_64/gcc-11.4.0/libffi-3.4.4-53jki34t2ffgsh2ia
1.5G    linux-ubuntu22.04-x86_64/gcc-11.4.0/llvm-14.0.0-p3upcdtjcmaxljjyxt
356K    linux-ubuntu22.04-x86_64/gcc-11.4.0/perl-data-dumper-2.173-32ot75c
1.5G    linux-ubuntu22.04-x86_64/gcc-11.4.0/

This might be able to be reduced by lowering how many variants are on in the llvm package but I went with what is there.

I also haven't looked into how this compares to just apt-get'ing the whole llvm package.

How would we like to discuss moving forward with this?

[white238]

Just to be clear, I didn't go through all of the compiler docker files.

I think all the current pipelines are failing due to the script I created doesn't exist in the base ubuntu images. I assume this is a chicken and egg problem where the base images won't rebuild until pushed to main?

[vsoch]

To step back, I think I'd like to get a high level understanding of the wants / needs of the code teams before re-doing anything (it's been a while since we discussed any of this or worked on this PR)! I think we should capture a concise understanding / listing of the needs first, and then start again here with the best way to address. It's great to reduce size a little bit, but I'm not sure I am convinced there is overwhelming benefit to a marginal change like that, if that makes sense. @davidbeckingsale what are your thoughts? Does your RSE team have insight into said needs?

[white238]

These docker images are currently used in the following projects:

https://sourcegraph.com/search?q=context:global+repo:github.com/llnl/*+ghcr.io/rse-ops&patternType=standard&sm=0&groupBy=repo

They are used to run their PR pipelines on a various set of compilers to ensure their tests pass. This is done through Azure Pipelines.

Currently, Umpire and RAJA are having to turn off compilers because they are running out of space on the Azure Pipelines. If I remember correctly, there is a 14GB limit on the Azure Pipeline instance (not the individual docker images). When you add a few of the images together, it causes the pipelines to fail due to running out of that space.

The way to solve this is to lower the size of the images.

The current projects use these images directly. But there is another use case, which is to take this image and build third-party libraries (TPLs) and create another image that is then used in other pipelines. This is done because TPLs can take a very long time to build. For example, Axom's TPL builds take ~30 minutes each. This is too long for a PR pipeline. Building these TPLs also grows the images in size as well but we don't have to build on as many compilers. It is important that they don't grow too large. Our current ones are ~3.5GB with TPLs built and 800MB base (off the top of my head).

Axom and Serac use the same base docker images from the predecessor to this repository. I would like to upgrade the Ubuntu and move to a more modern clang and gcc. But I can't do this if the docker images end up being so large that I cannot run my tests.

To my knowledge, none of the existing or Axom/Serac need to use the Spack that is downloaded in these images. We need to use a specific Spack commit sha that has been vetted. So it is safe to remove it to save space. Hence why I moved all the Spack commands to a script that also removes everything except the installed packages.

@vsoch Does this answer all of your questions/concerns? I am free at any time tomorrow to meet or to continue this conversation here.

[vsoch]

I'd like to hear @davidbeckingsale thoughts. I'm not convinced that going from 3.29GB down to 2.41GB is a long term solution - I'd suspect the problem will re-occur and it needs more thought. I'm also wondering if Azure piplines is the best one to use, we don't run into these issues in other places.

Going to bed - chat more on this thread later!

[davidbeckingsale]

@vsoch - Chris is one of the primary users we are trying to help with these images so his comments regarding the wants/needs of the code teams are exactly what I would also say. I think we kind of tried to over-prepare by using and keeping Spack in the images (so that teams could easily add extra dependencies), but in reality the use case is just to get a specific compiler version to run CI testing. In this case, the image size is stopping us from being able to do anything - we have already had to disable multiple builds on RAJA and Umpire due to these problems.

[vsoch]

Understood. I don't think making the images slightly smaller is going to solve the problem in the long term, and I'd like to discuss more robust solutions that won't warrant coming back here in 6 months when everything is slightly larger. I'm wondering why spack is needed here?

[vsoch]

I'm going to pass maintainership over to @davidbeckingsale and his RSE team - I am not as much in touch with the needs of the code teams, and not officially on RADIUSS time anymore. So please do what you think is best here. Thanks!