Can't integrate rocks due to readOnlyRootFilesystem: true

[kimwnasptd]

Bug Description

This is a follow-up of #243

We can't integrate the Knative rocks because:

1. The generated manifests, applied from the Knative Operator, have hardcoded readOnlyRootFilesystem: true https://github.com/knative/operator/blob/knative-v1.12.4/cmd/operator/kodata/knative-serving/1.12.4/2-serving-core.yaml#L6254
2. Pebble right now always needs to be able to write its state in the filesystem Pebble silently fails on a readOnlyRootFilesystem pebble#462

To Reproduce

1. Deploy the following knative bundle
2. Use a custom rock for knative controller
3. Check the pods in knative-serving and their logs

bundle: kubernetes
name: knative
applications:
  knative-eventing:
    charm: knative-eventing
    channel: 1.12/stable
    scale: 1
    trust: true
    options:
      namespace: knative-eventing
    _github_repo_name: knative-operators
    _github_repo_branch: track/1.12
  knative-operator:
    charm: knative-operator
    channel: 1.12/stable
    scale: 1
    trust: true
    _github_repo_name: knative-operators
    _github_repo_branch: track/1.12
    resources:
      knative-operator-image: kimwnasptd/knative-operator:readonly-2
  knative-serving:
    charm: knative-serving
    channel: 1.12/stable
    scale: 1
    trust: true
    options:
      namespace: knative-serving
      istio.gateway.namespace: kubeflow
      istio.gateway.name: kubeflow-gateway
    _github_repo_name: knative-operators
    _github_repo_branch: track/1.12

Environment

Knative 1.12

Relevant Log Output

The same as described in canonical/pebble#462

Additional Context

This was also captured in #243 (comment), but to keep the 2 problems separated I've created this issue.

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6899.

This message was autogenerated

[kimwnasptd]

After syncing with @deusebio and @jnsgruk we discussed about the following as a plan:

1. Try to see if we can update pebble, to not need a read-only filesystem
2. In the interim, patch the code of Knative Operator to not apply manifests with readOnlyRootFilesystem: true

Also note that this is a problem because Knative Serving manifests (and Deployments) are deployed via the Knative Operator charm. Thus we can't control what the Knative Operator applies for the serving manifests, as seen in #243 (comment)

Another approach would be to create a sidecar charm for each Knative Serving workload (activator, autoscaler etc) and not rely on the Knative Operator for applying the manifests for serving. But this will require a significant refactor and creating a lot of new charms.

[kimwnasptd]

Thankfully, all the manifests that the Knative Operator applies for Knative Serving and Eventing are stored as yaml files, and included in the final container image:
https://github.com/knative/operator/tree/knative-v1.12.4/cmd/operator/kodata/knative-serving

So this should be quite a straight forward change to update the source code.

[kimwnasptd]

Initially I tried to do the following:

1. Create a git patch, and a script for generating the patch
2. Copy the patch next to the rockcraft.yaml file
3. Update the rockcraft.yaml file to:
   1. Copy the patch
   2. Apply the patch before building the image

The script for generating the patch would look like this. It's goal is to load the target git tag from the rockcraft.yaml and then create the patch file:

#!/bin/bash
# scripts/create-readonlyfs-patch.sh

set -xe

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
KNATIVE_OP=$SCRIPT_DIR/../knative-operator

# cleanup patches
rm $KNATIVE_OP/patches/*

# Copy the git tag from `knative-operator` rock
TARGET_TAG=$(cat $KNATIVE_OP/rockcraft.yaml | yq ".parts.operator.source-tag")

cd /tmp
git clone https://github.com/knative/operator knative-operator
cd knative-operator
git checkout $TARGET_TAG

# create the patch
find . -type f \
	-exec sed -i \
	"s#readOnlyRootFilesystem: true#readOnlyRootFilesystem: false#g" \
	{} +
git add . && git commit -m "readOnlyRootFilesystem false"
git format-patch -1 HEAD
# 0001-readOnlyRootFilesystem-false.patch

# Copy back the patch
cd $KNATIVE_OP
cp /tmp/knative-operator/0001-readOnlyRootFilesystem-false.patch patches

# cleanup cloned repo
rm -rf /tmp/knative-operator

Then the rockraft.yaml would be updated like this:

...
  copy-patch:
    plugin: dump
    source-type: local
    source: ./patches
    # exclude the patch from the final container    
    prime:
      - -0001-readOnlyRootFilesystem-false.patch

  operator:
    after: [copy-patch]
    plugin: go
    ...
    override-build: |
      # apply the readOnlyRootFilesystem patch
      cp $CRAFT_STAGE/0001-readOnlyRootFilesystem-false.patch .
      git apply 0001-readOnlyRootFilesystem-false.patch

      # regular build
      go mod download
      ...

[jnsgruk]

Yeh, and I'd just carry these as lightweight patches against the operators for now (patch at the point of build in the rock build) - rather than a full fork.

I'll investigate the possibility of some "retrograde" behaviour for Pebble on read-only filesystems, too.

[kimwnasptd]

But the above felt a bit of an overkill, since we can update the source code with a simple sed command. So in the end after discussing with @deusebio we'll just be updating the rockcraft.yaml file like this, to avoid creating a patch:

...
  operator:
    plugin: go
    ...
    override-build: |
      # apply the readOnlyRootFilesystem patch
     find . -type f \
        -exec sed -i \
        "s#readOnlyRootFilesystem: true#readOnlyRootFilesystem: false#g" \
        {} +

      # regular build
      go mod download
      ...

And with the above I confirm that we see the final Deployments have readOnlyRootFilesystem: false, which unblocks us for now.

[jnsgruk]

Yeh, that looks much better