[EPIC] Prometheus remote write with Thanos Receive

[richardtief]

Context

We create a second option for monitoring with multiple Prometheus per Kubernetes cluster. This allows us to utilize the Prometheus remote write functionality and Thanos. Each Prometheus writes its metrics remotely to a dedicated Thanos Receive. The Thanos receivers are located in the same or a dedicated Kubernetes cluster, which must be accessible via the network. This has an impact on the data storage of metrics - in this case we assume that Thanos Receive is connected to an object store.

To set up the appropriate number of Receives automatically, there must be a way to check the number of Prometheus sending from the clusters. Either it is possible to receive the nodes from the respective cluster itself, or another service discovery via the Greenhouse central cluster is preferable. This might require additional instrumentation.

Another aspect should be that Prometheus can be deployed per node as a deamon set. This statelessness of Prometheus can help to ensure that Prometheus can be brought up at almost any time without missing many samples.

Note:

• Possible data loss if Prometheus or Thanos Receive goes down

Acceptance criteria:

1. Take existing kube-monitoring and thanos plugin functions into account
2. Deployment of multiple Prometheus possible
3. Integrate Prometheus operator and CRDs as cluster singleton
4. Automated Thanos Receive Deployment per Prometheus
   5 Thanos Query to update new available endpoints
5. Testing and verification
6. Documentation