Consider using different db engine then Prometheus

[mpryc]

Current state

We are currently using Prometheus to store our 4 metrics with the following metadata associated with each of them.

Some of the metrics (marked with [T]) are stored with timestamp and some are not. All of the metrics store a unix EPOCH timestamp as the value (marked with [V])

deploy_timestamp [V] [T]

• namespace
• app
• image_sha

commit_timestamp [V]

• namespace
• app
• commit
• image_sha

failure_creation_timestamp and failure_resolution_timestamp [V]

• app
• issue_number

Problem

1. Prometheus is not allowing to store metrics that has timestamp() that are older than current head block operation from the time of scraping (~3h in the past)
2. Using timestamp() in metric is considered anti-pattern for how the Prometheus exporters should be used
3. Even if it's possible to back-fill samples from the past to the Prometheus, it is still considered a migration tool, rather than standard operation. Back-fill has some consequences and can not be performed to import data that are from the current Prometheus head block (last 3 hours)
4. Using Prometheus push mode isn't also an option, as those metrics should not include timestamp neither.
5. Not using timestamps() may cause false data to be processed, especially within time ranges where the calculations are made using timestamps and not the values.
6. If the metric does not use timestamp, it's being added as the current time when the scrape happens. This leads to the situation where the calculations based on values are considering historical data to be current.
7. Using timestamp() is not an option for some of the metrics such as commit time, because the commit that was used for deployment may have happened days/weeks/months prior to deployment and we are interested in the time between commit and deployment.

Possible work-around

Store some of the metrics with and some without timestamps.
Add time limit window to accept the metrics with timestamps including webhook exporter, so they will be always almost up to date.

Clean solution

Switch to a DB engine that can store historical values with timestamps that have happened in the past. Ideally if the DB engine supports both Prometheus exporters and Grafana, meaning minimal change on the Pelorus architecture will be made.

[mpryc]

@weshayutin @mateusoliveira43 FYI

[mateusoliveira43]

thanks for the doc Michal

I just disagree with item 7 of Problem section: I think the problem is how we collect commit timestamp in committime exporter

[mpryc]

Currently we don't have the problem with the commit time, but if we add timestamp to it, we will definitely hit the issue no matter how we collect that metric. The only way around would be to collect that metric from the Git provider using webhook that sends the data when the commit happens. However in the current way the metrics are collected we can not do that, because commit metric expects:

• namespace
• app
• commit
• image_sha

And those are available later in the application lifecycle, many times way after Prometheus acceptance limit.

[mateusoliveira43]

That is what I see wrong in committime exporter: it should work like failure, scraping data over time, needing git URL and branch only, not getting time from a specific commit hash