Problem In Docker Appliance Minimal with Archiver Container

[utkonos]

Describe the bug
Running:

docker-compose up -d --wait

Results in error:

 ⠴ Container al-archiver-1        Waiting                                                                                                                                         31.0s
Error response from daemon: No such container: c77c9f8258745ad70a4b1fd90fb1167053e580ef155b4670212b2f6ce5ba3570

To Reproduce
Steps to reproduce the behavior:

1. Instantiate VM with Ubuntu 24.04 LTS (Noble Numbat) daily [20241210]
2. Configure Ubuntu using ubuntu.sh
3. Install Docker using docker.sh
4. Install Assemblyline up to docker-compose up -d --wait using assemblyline.sh

Expected behavior
No error.

Screenshots

Environment (please complete the following information if pertinent):

• Assemblyline Version:

# sudo docker inspect 3ffe1adff728  | grep Image
        "Image": "sha256:59f613857589be315897f563fb6b16b36ea63cab81c48be9199f891c367a5c18",
            "Image": "cccs/assemblyline-core:4.5.stable",

Additional context
I have a workaround which is shown in the assemblyline.sh script:

docker-compose up -d --wait --scale archiver=0

Reading the logs, it shows that there is just a warning about archiver not being enabled, so I figured that --scale archiver=0 is safe. Please let me know if this is the wrong approach.

For completeness, here is a screenshot of the results of the command including the workaround:

[cccs-rs]

This likely stems from the fact that a minimal/full Docker appliance doesn't have datastore.archive.enabled: true by default. Because of this, the container will actually terminate if these conditions aren't met per:
https://github.com/CybercentreCanada/assemblyline-core/blob/3c8375b5d839ba2f4fc38bacd88bf2911a413a5b/assemblyline_core/archiver/run_archiver.py#L28

Using the command you provided would work in this case, we could alternatively comment out the archiver service from the Compose file so it doesn't interfere with the --wait flag.

[cccs-rs]

We could refactor our Docker Compose files to use profiles, which I think should simplify things and remove duplication of files where it isn't necessary: https://docs.docker.com/compose/how-tos/profiles/

We could have a profile for full which would replace the need for a minimal vs full appliance directories, and have another profile for archive that would spin up archiver.

So by default docker compose would bring up the minimal appliance until you specify the full profile (which I think just brings up filebeat and metricbeat) which I think should integrate with your script well.

[cccs-rs]

@utkonos let me know what you think about these changes to restructure our Docker appliance repository to use profiles to specify what you want for your deployment:
https://github.com/CybercentreCanada/assemblyline-docker-compose/tree/profiles

[utkonos]

Looks good reading it over. I will run through it over the weekend and let you know how it goes.

[cccs-rs]

@utkonos thoughts on the proposed changes?

I opened #308 last night, don't know if this simplifies or complicates deployments 😅