Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #RHIROS-1401 - Dropping csv records with missing resource usage… #144

Merged
merged 3 commits into from
Nov 15, 2023
Merged

Fixes #RHIROS-1401 - Dropping csv records with missing resource usage… #144

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e6eb00c
Fixes #RHIROS-1401 - Dropping csv records with missing resource usage…
patilsuraj767 Oct 30, 2023
2dab9b1
Fix logging
patilsuraj767 Oct 30, 2023
1277252
Added unit test cases
patilsuraj767 Nov 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion internal/logging/logging.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,12 +62,13 @@ func GetLogger() *logrus.Entry {
}

func Set_request_details(data types.KafkaMsg) *logrus.Entry {
return log.WithFields(logrus.Fields{
log = log.WithFields(logrus.Fields{
"request_id": data.Request_id,
"account": data.Metadata.Account,
"org_id": data.Metadata.Org_id,
"source_id": data.Metadata.Source_id,
"cluster_uuid": data.Metadata.Cluster_uuid,
"cluster_alias": data.Metadata.Cluster_alias,
})
return log
}
51 changes: 51 additions & 0 deletions internal/utils/aggregator.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,12 @@ import (
"github.com/go-gota/gota/dataframe"
"github.com/go-gota/gota/series"

"github.com/redhatinsights/ros-ocp-backend/internal/logging"
w "github.com/redhatinsights/ros-ocp-backend/internal/types/workload"
)

func Aggregate_data(df dataframe.DataFrame) dataframe.DataFrame {
log = logging.GetLogger()
df = df.FilterAggregation(
dataframe.And,
dataframe.F{Colname: "owner_kind", Comparator: series.Neq, Comparando: ""},
Expand Down Expand Up @@ -40,6 +42,33 @@ func Aggregate_data(df dataframe.DataFrame) dataframe.DataFrame {

df = df.Mutate(s.Col("X0")).Rename("k8s_object_type", "X0")
df = df.Mutate(s.Col("X1")).Rename("k8s_object_name", "X1")

// filter out only valid workload type
df = df.Filter(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to get this out as separate function? I understand its just a call to Filter however the Aggregate_data is already lengthy. With that can we have tests then? So looking at those one would understand what is invalid?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added test cases

dataframe.F{
Colname: "k8s_object_type",
Comparator: series.In,
Comparando: []string{
w.Daemonset.String(),
w.Deployment.String(),
w.Deploymentconfig.String(),
w.Replicaset.String(),
w.Replicationcontroller.String(),
w.Statefulset.String(),
saltgen marked this conversation as resolved.
Show resolved Hide resolved
}},
)

// Validation to check if metrics for cpuUsage, memoryUsage and memoryRSS are missing
df, no_of_dropped_records := filter_valid_csv_records(df)
if no_of_dropped_records != 0 {
invalidDataPoints.Add(float64(no_of_dropped_records))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand then we are never going to return float from filter_valid_csv_records, is it prometheus which expects float value?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, prometheus Add method requires the float value.

log.Infof("Invalid records in CSV - %v", no_of_dropped_records)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be at Info level? and do you think we should also print more about owner_name and workload?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, good to have at info level so that using request_id we can check in kibana how many rows where dropped for particular request/CSV. Our logging system by default logs all the request related info - https://github.com/RedHatInsights/ros-ocp-backend/blob/main/internal/logging/logging.go#L66-L71

saltgen marked this conversation as resolved.
Show resolved Hide resolved
}

if df.Nrow() == 0 {
return df
}

dfGroups := df.GroupBy(
"namespace",
"k8s_object_type",
Expand Down Expand Up @@ -87,3 +116,25 @@ func Aggregate_data(df dataframe.DataFrame) dataframe.DataFrame {
df = dfGroups.Aggregation(columnsAggregationType, columnsToAggregate)
return df
}

func filter_valid_csv_records(main_df dataframe.DataFrame) (dataframe.DataFrame, int) {
patilsuraj767 marked this conversation as resolved.
Show resolved Hide resolved
df := main_df.FilterAggregation(
dataframe.And,
dataframe.F{Colname: "memory_rss_usage_container_sum", Comparator: series.GreaterEq, Comparando: 0},
dataframe.F{Colname: "memory_rss_usage_container_max", Comparator: series.GreaterEq, Comparando: 0},
dataframe.F{Colname: "memory_rss_usage_container_min", Comparator: series.GreaterEq, Comparando: 0},
dataframe.F{Colname: "memory_rss_usage_container_avg", Comparator: series.GreaterEq, Comparando: 0},
dataframe.F{Colname: "memory_usage_container_sum", Comparator: series.GreaterEq, Comparando: 0},
dataframe.F{Colname: "memory_usage_container_max", Comparator: series.GreaterEq, Comparando: 0},
dataframe.F{Colname: "memory_usage_container_min", Comparator: series.GreaterEq, Comparando: 0},
dataframe.F{Colname: "memory_usage_container_avg", Comparator: series.GreaterEq, Comparando: 0},
dataframe.F{Colname: "cpu_usage_container_sum", Comparator: series.GreaterEq, Comparando: 0},
dataframe.F{Colname: "cpu_usage_container_max", Comparator: series.GreaterEq, Comparando: 0},
dataframe.F{Colname: "cpu_usage_container_min", Comparator: series.GreaterEq, Comparando: 0},
dataframe.F{Colname: "cpu_usage_container_avg", Comparator: series.GreaterEq, Comparando: 0},
)

no_of_dropped_records := main_df.Nrow() - df.Nrow()

return df, no_of_dropped_records
}
13 changes: 13 additions & 0 deletions internal/utils/metrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
package utils

import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
)

var (
invalidDataPoints = promauto.NewCounter(prometheus.CounterOpts{
Name: "rosocp_invalid_datapoints_total",
Help: "The total number of invalid datapoints(rows) found in CSVs recevied",
saltgen marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
Name: "rosocp_invalid_datapoints_total",
Help: "The total number of invalid datapoints(rows) found in CSVs recevied",
Name: "rosocp_total_invalid_datapoints",
Help: "The total number of invalid datapoints(rows) found in received CSVs",

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to prometheus metric naming convention - https://prometheus.io/docs/practices/naming/ suffix should describe the unit.

})
)