Research all the changes necessary to support other OpenTelemetry deployments mode

[jvanz]

During the initial work done the #916 we detected the need to change how we integrate with OpenTelemetry. To properly collect metadata from the Kubernetes cluster is necessary to run the OpenTelemetry collector in other deployments modes, like daemonset or deployments. However, as the Kubewarden stack components expected the collector to be run in a sidecar, this is not possible. Therefore, we need to changes the Kubewarden stack components (e.g. controller and policy server) to support sending metrics and tracing data to Otel collector running somewhere else.

The goal of this issue is to research all the changes required to make that possible.

Acceptance criteria

• Map what changes are necessary to the controller to allow the communication with a Otel collector not running as a sidecar.
• Map what changes are necessary to the policy server to allow the communication with a Otel collector not running as a sidecar.
• Map the changes in the controller necessary to it to create policy servers properly configured to send data to Otel collector not running as a sidecar
• Map the missing Helm chart configuration to allow the Kubewarden components (after the changes described in the other acceptance criteria) to send data to remote Otel collectors.
• Write an epic issue to keep track of the required changes

[jvanz]

@kubewarden/kubewarden-developers please, review this epic issue about the Otel changes and let me know if you miss something.

[jvanz]

@kubewarden/kubewarden-developers I think we have a nice epic to work on and seems that all agree on how to do it. Therefore, I'm closing this issue. Let me know if you think we need something else to do in this spike.