Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Support To Provide Dimensions #93

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Add Support To Provide Dimensions #93

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4a31a81
Add Dimension Support and List Namespace/Definitions using Tags
Oct 7, 2020
daaa126
Reformat defination list with dimensions
Oct 8, 2020
9c1e83a
Fix vet warnings and add documentation
Oct 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,14 @@ resource_tags:
- "Microsoft.Compute/virtualMachines"
metrics:
- name: "CPU Credits Consumed"
- resource_tag_name: "dbtype"
resource_tag_value: "document"
resource_types:
- Microsoft.DocumentDB/databaseAccounts
metrics:
- name: "TotalRequestUnits"
- name: "TotalRequests"
dimensions: "CollectionName eq '*' and StatusCode eq '*'"

```

Expand All @@ -106,6 +114,9 @@ When the metric namespace is specified, it will be added as a prefix of the metr
It can be used to target [custom metrics](https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-custom-overview), such as [guest OS performance counters](https://docs.microsoft.com/en-us/azure/azure-monitor/platform/collect-custom-metrics-guestos-vm-classic).
If not specified, the default metric namespace of the resource will apply.

The `dimensions` property is optional for all filtering types. If `dimensions` property is provided, it will add the provided dimensions as label in the metrics.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused. The example is a filter, however this then talks about adding labels to the output. Those sound like orthogonal things to me.

If there's labels we need for correctness, we should already be applying them automatically.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the confusion. I should be more clear about this.
I mean to say Metrics dimensions configuration can be provided in all filtering types (ResourceTags, ResourceGroups, and Targets).
Adding a label with each dimension will provide the ability to consume/filter these metrics based on labels for visualization and alerting.

This is useful in the case of multi-dimensional metrics. For example, CosmosDB's TotalRequests metric provides Total Request Count without any dimension.

totalrequests_count_average{resource_group="cna_nonprod_db",resource_name="cna-nonprod-db"} 0
totalrequests_count_average{resource_group="cosmosdb_nonprod",resource_name="adobeio-runtime-mw-nonprod"} 1.8411764705882352

if the dimension StatusCode and CollectionName provided then the metric response will contain for each StatusCode for CollectionName and StatusCode and CollectionName will be added as label in the metric.

totalrequests_count_average{collectionname="Activations",resource_group="cosmosdb_nonprod",resource_name="runtime-mw-nonprod",statuscode="201"} 7.105298457411133
totalrequests_count_average{collectionname="aio-state-container",resource_group="cna_nonprod_db",resource_name="cna-nonprod-db",statuscode="200"} 0

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's two separate features that'd need to be evaluated independently then. Filtering is for performance, the presence of labels is for correctness and should never depend on filtering being configured.

Copy link

@hacst hacst Feb 3, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brian-brazil I think this approach is required by the underlying API. See https://docs.microsoft.com/en-us/azure/azure-monitor/platform/rest-api-walkthrough#retrieve-metric-values-multi-dimensional-api

If no dimension filters are specified, the rolled up aggregated metric is returned.

My reading is that this means you cannot retrieve dimensions without a filter. From a users point of view I also think this behavior is desirable as otherwise you are likely to retrieve too much or future changes to the dimensionality might break you.

It would be great to see this PR go ahead as it implements very useful functionality. E.g. I'm currently trying to get the message count in azure service bus metrics. azure-metrics-exporter seems perfect for that but without dimensions I only get message count aggregates at the azure service bus namespace level instead of per queue/topic information with is in a EntityName dimension.

You can get the available `dimensions` for a given resource metrics using [metrics definitions](#retrieving-metric-definitions).

### Resource group filtering

Resources in a resource group can be filtered using the the following keys:
Expand Down
46 changes: 45 additions & 1 deletion azure.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,13 @@ type AzureMetricValueResponse struct {
Minimum float64 `json:"minimum"`
Maximum float64 `json:"maximum"`
} `json:"data"`
Dimensions []struct {
Name struct {
LocalizedValue string `json:"localizedValue"`
Value string `json:"value"`
} `json:"name"`
Value string `json:"value"`
} `json:"metadatavalues"`
} `json:"timeseries"`
ID string `json:"id"`
Name struct {
Expand Down Expand Up @@ -279,6 +286,25 @@ func (ac *AzureClient) getMetricDefinitions() (map[string]AzureMetricDefinitionR
definitions[defKey] = *def
}
}
resourcesCache := make(map[string][]byte)
for _, resourceTag := range sc.C.ResourceTags {
resources, err := ac.filteredListByTag(resourceTag, resourcesCache)
if err != nil {
return nil, fmt.Errorf("Failed to get resources for resource tag:value %s:%s and resource types %s: %v",
resourceTag.ResourceTagName, resourceTag.ResourceTagValue, resourceTag.ResourceTypes, err)
}
for _, resource := range resources {
def, err := ac.getAzureMetricDefinitionResponse(resource.ID, resourceTag.MetricNamespace)
if err != nil {
return nil, err
}
defKey := resource.ID
if len(resourceTag.MetricNamespace) > 0 {
defKey = fmt.Sprintf("%s (Metric namespace: %s)", defKey, resourceTag.MetricNamespace)
}
definitions[defKey] = *def
}
}
return definitions, nil
}

Expand Down Expand Up @@ -307,6 +333,21 @@ func (ac *AzureClient) getMetricNamespaces() (map[string]MetricNamespaceCollecti
namespaces[resource.ID] = *namespaceCollection
}
}
resourcesCache := make(map[string][]byte)
for _, resourceTag := range sc.C.ResourceTags {
resources, err := ac.filteredListByTag(resourceTag, resourcesCache)
if err != nil {
return nil, fmt.Errorf("Failed to get resources for resource tag:value %s:%s and resource types %s: %v",
resourceTag.ResourceTagName, resourceTag.ResourceTagValue, resourceTag.ResourceTypes, err)
}
for _, resource := range resources {
namespaceCollection, err := ac.getMetricNamespaceCollectionResponse(resource.ID)
if err != nil {
return nil, err
}
namespaces[resource.ID] = *namespaceCollection
}
}
return namespaces, nil
}

Expand Down Expand Up @@ -584,7 +625,7 @@ type batchRequest struct {
Method string `json:"httpMethod"`
}

func resourceURLFrom(resource string, metricNamespace string, metricNames string, aggregations []string) string {
func resourceURLFrom(resource string, metricNamespace string, metricNames string, aggregations []string, dimensions string) string {
apiVersion := "2018-01-01"

path := fmt.Sprintf(
Expand All @@ -602,6 +643,9 @@ func resourceURLFrom(resource string, metricNamespace string, metricNames string
if metricNamespace != "" {
values.Add("metricnamespace", metricNamespace)
}
if dimensions != "" {
values.Add("$filter", dimensions)
}
filtered := filterAggregations(aggregations)
values.Add("aggregation", strings.Join(filtered, ","))
values.Add("timespan", fmt.Sprintf("%s/%s", startTime, endTime))
Expand Down
3 changes: 3 additions & 0 deletions config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,7 @@ type Target struct {
MetricNamespace string `yaml:"metric_namespace"`
Metrics []Metric `yaml:"metrics"`
Aggregations []string `yaml:"aggregations"`
Dimensions string `yaml:"dimensions"`

XXX map[string]interface{} `yaml:",inline"`
}
Expand All @@ -162,6 +163,7 @@ type ResourceGroup struct {
ResourceNameExcludeRe []Regexp `yaml:"resource_name_exclude_re"`
Metrics []Metric `yaml:"metrics"`
Aggregations []string `yaml:"aggregations"`
Dimensions string `yaml:"dimensions"`

XXX map[string]interface{} `yaml:",inline"`
}
Expand All @@ -174,6 +176,7 @@ type ResourceTag struct {
ResourceTypes []string `yaml:"resource_types"`
Metrics []Metric `yaml:"metrics"`
Aggregations []string `yaml:"aggregations"`
Dimensions string `yaml:"dimensions"`

XXX map[string]interface{} `yaml:",inline"`
}
Expand Down
20 changes: 15 additions & 5 deletions main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,12 +76,17 @@ func (c *Collector) extractMetrics(ch chan<- prometheus.Metric, rm resourceMeta,
if rm.metricNamespace != "" {
metricName = strings.ToLower(rm.metricNamespace + "_" + metricName)
}

metricName = invalidMetricChars.ReplaceAllString(metricName, "_")

if len(value.Timeseries) > 0 {
metricValue := value.Timeseries[0].Data[len(value.Timeseries[0].Data)-1]
labels := CreateResourceLabels(rm.resourceURL)

if len(value.Timeseries[0].Dimensions) > 0 {
for _, dimension := range value.Timeseries[0].Dimensions {
labels[dimension.Name.Value] = dimension.Value
}
}
if hasAggregation(rm.aggregations, "Total") {
ch <- prometheus.MustNewConstMetric(
prometheus.NewDesc(metricName+"_total", metricName+"_total", nil, labels),
Expand Down Expand Up @@ -234,7 +239,7 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
rm.metricNamespace = target.MetricNamespace
rm.metrics = strings.Join(metrics, ",")
rm.aggregations = filterAggregations(target.Aggregations)
rm.resourceURL = resourceURLFrom(target.Resource, rm.metricNamespace, rm.metrics, rm.aggregations)
rm.resourceURL = resourceURLFrom(target.Resource, rm.metricNamespace, rm.metrics, rm.aggregations, target.Dimensions)
incompleteResources = append(incompleteResources, rm)
}

Expand All @@ -259,7 +264,7 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
rm.metricNamespace = resourceGroup.MetricNamespace
rm.metrics = metricsStr
rm.aggregations = filterAggregations(resourceGroup.Aggregations)
rm.resourceURL = resourceURLFrom(f.ID, rm.metricNamespace, rm.metrics, rm.aggregations)
rm.resourceURL = resourceURLFrom(f.ID, rm.metricNamespace, rm.metrics, rm.aggregations, resourceGroup.Dimensions)
rm.resource = f
resources = append(resources, rm)
}
Expand Down Expand Up @@ -287,7 +292,7 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
rm.metricNamespace = resourceTag.MetricNamespace
rm.metrics = metricsStr
rm.aggregations = filterAggregations(resourceTag.Aggregations)
rm.resourceURL = resourceURLFrom(f.ID, rm.metricNamespace, rm.metrics, rm.aggregations)
rm.resourceURL = resourceURLFrom(f.ID, rm.metricNamespace, rm.metrics, rm.aggregations, resourceTag.Dimensions)
incompleteResources = append(incompleteResources, rm)
}
}
Expand Down Expand Up @@ -333,7 +338,12 @@ func main() {
for k, v := range results {
log.Printf("Resource: %s\n\nAvailable Metrics:\n", k)
for _, r := range v.MetricDefinitionResponses {
log.Printf("- %s\n", r.Name.Value)
log.Printf("\n\nMetric:\n")
log.Printf("- %s", r.Name.Value)
log.Printf("\nDimensions:\n")
for _, d := range r.Dimensions {
log.Printf("- %s\n", d.Value)
}
}
}
os.Exit(0)
Expand Down