resources stanza for Docker driver task somehow causes permission issues within the container

[efstajas]

Nomad version

Nomad v1.9.3
BuildDate 2024-11-11T16:35:41Z
Revision d92bf1014886c0ff9f882f4a2691d5ae8ad8131c

Operating system and Environment details

Linux pi-cluster-5-01 6.6.51+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.51-1+rpt3 (2024-10-08) aarch64

Docker version 27.4.0, build bde2b89

Issue

I'm trying to deploy an image lscr.io/linuxserver/nextcloud:latest. Here's my jobspec:

job "nextcloud" {
  region = "global"
  datacenters = ["dc1"]
  namespace   = "default"
  type        = "service"
  
  group "nextcloud" {
    network {
      mode = "bridge"
      port "http" {
        to = 80
      }
    }

    task "nextcloud" {
      driver = "docker"

      config {
        image = "lscr.io/linuxserver/nextcloud:latest"
        ports = ["http"]
      }

      env {
        TZ = "Etc/UTC"
        PGID = "1000"
        PUID = "1000"
      }

      resources {
        cpu    = 2000
        memory = 5000
      }
    }
  }
}

The env values are all the default values for that container. The PUID and GUID env vars are the standard values for that image. The problem is that the container fails to initialise because it gets permission denied errors trying to chown dirs within the container:

chown: changing ownership of '/app': Operation not permitted
chown: changing ownership of '/config': Operation not permitted
chown: changing ownership of '/defaults': Operation not permitted
mkdir: cannot create directory ‘/var/lib/nginx’: Permission denied
s6-rc: warning: unable to start service init-folders: command exited 1
chown: changing ownership of '/etc/crontabs/abc': Operation not permitted
crontab: setegid: Operation not permitted

... and indeed, when I sh into the container via Docker CLI and try to chown one of those dirs, I confusingly get permission denied even though I'm root with id 0:0.

I then tried deploying the image directly through the Docker CLI with the same config, and to my further confusion, everything worked fine. I sh into that container too, I'm also root there, but I can chown all the dirs just fine, and the init script also works. The output of id is 100% identical between the two containers.

So, I try to prepare a minimum reproducible example, and discover that it seems to be related to the resources stanza in the task, somehow. When I remove it, the nomad-orchestrated container has no permission issues. When I add it back, they're back. This seems to be reproducible on my end 100% of the time. I have no idea what could be going on here.

Reproduction steps

• Deploy the above jobspec, once with resources stanza and once without
• SSH into the hosts
• sudo docker ps to find the container IDs of the two containers
• sudo docker exec -it < container ID> to enter shell in both containers
• Attempt chown 1000:1000 on app/ (or any other dir)

Expected Result

• Since the containers are theoretically identical apart from one being resource-constrained, and the app/ dir in both is owned by root and the user is root, the chown should work on both containers

Actual Result

• chown fails with Permission denied on the resource-constrained container, but works as expected on the one that's not.

[pkazmierczak]

Hi @efstajas, thanks for reporting the issue. Sadly, I cannot reproduce. I deployed the jobspec with the resources block on ubuntu with nomad 1.9.3 with no issues, and was able to chown 1000:1000 the app directory inside the nextcloud container. Can you tell me more about your Nomad cluster setup?