Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Collector doesn't appear to collect all metrics from an endpoint #18

Open
asachs01 opened this issue Nov 26, 2019 · 2 comments
Open

Collector doesn't appear to collect all metrics from an endpoint #18

asachs01 opened this issue Nov 26, 2019 · 2 comments

Comments

@asachs01
Copy link
Contributor

I'm observing a case where the prometheus collector doesn't appear to be collecting all the metrics from an endpoint. Here is the check config:

Check Definition

type: CheckConfig
api_version: core/v2
metadata:
  name: example-prom-exporter
  namespace: default
spec:
  command: sensu-prometheus-collector -exporter-url http://metrics:9499/metrics
  high_flap_threshold: 0
  interval: 10
  low_flap_threshold: 0
  output_metric_format: 
  output_metric_handlers:
  - influxdb
  round_robin: true
  proxy_entity_name: example.com
  publish: true
  runtime_assets:
  - sensu/sensu-prometheus-collector
  subscriptions:
  - entity:backend0.example.com

The output on the dashboard and in Influx only show a portion of the metrics:

Dashboard metrics 
go_memstats_gc_sys_bytes value=2404352 1574793200

go_memstats_heap_released_bytes value=0 1574793200

go_memstats_sys_bytes value=72284408 1574793200

promhttp_metric_handler_requests_in_flight value=1 1574793200

smartthings_switch_enabled,id=xxxxxxxxx-xxxxxxxxxx-xxxxxxxx,name=Patio value=0 1574793200

smartthings_switch_enabled,id=xxxxxxxxx-xxxxxxxxxx-xxxxxxxx,name=Driveway value=1 1574793200

smartthings_temperature_fahrenheit,id=xxxxxxxxx-xxxxxxxxxx-xxxxxxxx,name=Sachshaus value=69 1574793200

smartthings_temperature_fahrenheit,id=xxxxxxxxx-xxxxxxxxxx-xxxxxxxx,name=Basement value=81 1574793200

process_virtual_memory_bytes value=954318848 1574793200

promhttp_metric_handler_requests_total,code=200 value=1057 1574793200

promhttp_metric_handler_requests_total,code=500 value=0 1574793200

promhttp_metric_handler_requests_total,code=503 value=0 1574793200

go_gc_duration_seconds,quantile=0 value=0.000016094 1574793200

go_gc_duration_seconds,quantile=0.25 value=0.000040034 1574793200

go_gc_duration_seconds,quantile=0.5 value=0.000056948 1574793200

go_gc_duration_seconds,quantile=0.75 value=0.000110458 1574793200

go_gc_duration_seconds,quantile=1 value=0.001143649 1574793200

go_gc_duration_seconds_sum value=0.077228025 1574793200

go_gc_duration_seconds_count value=617 1574793200

go_memstats_alloc_bytes_total value=1281701824 1574793200

go_memstats_frees_total value=8766037 1574793200

go_memstats_heap_idle_bytes value=60751872 1574793200

go_memstats_lookups_total value=0 1574793200

go_memstats_stack_sys_bytes value=753664 1574793200

go_memstats_gc_cpu_fraction value=0.000014811654179269153 1574793200

go_memstats_heap_alloc_bytes value=3812088 1574793200

go_memstats_mallocs_total value=8785354 1574793200

go_memstats_next_gc_bytes value=4194304 1574793200

process_virtual_memory_max_bytes value=-1 1574793200

go_memstats_buck_hash_sys_bytes value=1495715 1574793200

go_memstats_heap_sys_bytes value=66322432 1574793200

go_memstats_mcache_inuse_bytes value=6912 1574793200

go_memstats_other_sys_bytes value=1209941 1574793200

go_threads value=13 1574793200

process_resident_memory_bytes value=23400448 1574793200

go_info,version=go1.11.3 value=1 1574793200

go_memstats_mspan_inuse_bytes value=64296 1574793200

process_open_fds value=8 1574793200

go_goroutines value=8 1574793200

go_memstats_alloc_bytes value=3812088 1574793200

go_memstats_heap_inuse_bytes value=5570560 1574793200

go_memstats_last_gc_time_seconds value=1574793195.47246 1574793200

process_cpu_seconds_total value=4.72 1574793200

process_max_fds value=1024 1574793200

smartthings_battery_percentage,name=Driveway,id=a2788c30-d77c-46bc-8966-082c6899b43a value=100 1574793200

smartthings_invalid_metric value=0 1574793200

smartthings_exporter_build_info,branch=HEAD,goversion=go1.11.3,revision=f77d40beb6474e35a501154bc376173dcb1fb41a,version=0.0.2 value=1 1574793200

smartthings_motion_detected,id=xxxxxxxxx-xxxxxxxxxx-xxxxxxxx,name=Driveway value=0 1574793200

smartthings_motion_detected,id=xxxxxxxxx-xxxxxxxxxx-xxxxxxxx,name=Basement value=0 1574793200

go_memstats_heap_objects value=19317 1574793200

go_memstats_mcache_sys_bytes value=16384 1574793200

go_memstats_mspan_sys_bytes value=81920 1574793200

go_memstats_stack_inuse_bytes value=753664 1574793200

process_start_time_seconds value=1574780661.53 1574793200

Screen Shot 2019-11-26 at 2 09 50 PM

Compared with the metrics exposed by the actual endpoint:

Prometheus Endpoint Metrics 
# HELP go_gc_duration_seconds A summary of the GC invocation durations.

# TYPE go_gc_duration_seconds summary

go_gc_duration_seconds{quantile="0"} 1.6094e-05

go_gc_duration_seconds{quantile="0.25"} 3.8547e-05

go_gc_duration_seconds{quantile="0.5"} 5.6086e-05

go_gc_duration_seconds{quantile="0.75"} 0.00010805

go_gc_duration_seconds{quantile="1"} 0.001143649

go_gc_duration_seconds_sum 0.074622899

go_gc_duration_seconds_count 591

# HELP go_goroutines Number of goroutines that currently exist.

# TYPE go_goroutines gauge

go_goroutines 8

# HELP go_info Information about the Go environment.

# TYPE go_info gauge

go_info{version="go1.11.3"} 1

# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.

# TYPE go_memstats_alloc_bytes gauge

go_memstats_alloc_bytes 2.330568e+06

# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.

# TYPE go_memstats_alloc_bytes_total counter

go_memstats_alloc_bytes_total 1.225650584e+09

# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.

# TYPE go_memstats_buck_hash_sys_bytes gauge

go_memstats_buck_hash_sys_bytes 1.495411e+06

# HELP go_memstats_frees_total Total number of frees.

# TYPE go_memstats_frees_total counter

go_memstats_frees_total 8.41828e+06

# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.

# TYPE go_memstats_gc_cpu_fraction gauge

go_memstats_gc_cpu_fraction 1.4951260252455343e-05

# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.

# TYPE go_memstats_gc_sys_bytes gauge

go_memstats_gc_sys_bytes 2.371584e+06

# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.

# TYPE go_memstats_heap_alloc_bytes gauge

go_memstats_heap_alloc_bytes 2.330568e+06

# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.

# TYPE go_memstats_heap_idle_bytes gauge

go_memstats_heap_idle_bytes 6.217728e+07

# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.

# TYPE go_memstats_heap_inuse_bytes gauge

go_memstats_heap_inuse_bytes 4.243456e+06

# HELP go_memstats_heap_objects Number of allocated objects.

# TYPE go_memstats_heap_objects gauge

go_memstats_heap_objects 11313

# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.

# TYPE go_memstats_heap_released_bytes gauge

go_memstats_heap_released_bytes 0

# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.

# TYPE go_memstats_heap_sys_bytes gauge

go_memstats_heap_sys_bytes 6.6420736e+07

# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.

# TYPE go_memstats_last_gc_time_seconds gauge

go_memstats_last_gc_time_seconds 1.574792577780887e+09

# HELP go_memstats_lookups_total Total number of pointer lookups.

# TYPE go_memstats_lookups_total counter

go_memstats_lookups_total 0

# HELP go_memstats_mallocs_total Total number of mallocs.

# TYPE go_memstats_mallocs_total counter

go_memstats_mallocs_total 8.429593e+06

# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.

# TYPE go_memstats_mcache_inuse_bytes gauge

go_memstats_mcache_inuse_bytes 6912

# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.

# TYPE go_memstats_mcache_sys_bytes gauge

go_memstats_mcache_sys_bytes 16384

# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.

# TYPE go_memstats_mspan_inuse_bytes gauge

go_memstats_mspan_inuse_bytes 61408

# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.

# TYPE go_memstats_mspan_sys_bytes gauge

go_memstats_mspan_sys_bytes 81920

# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.

# TYPE go_memstats_next_gc_bytes gauge

go_memstats_next_gc_bytes 4.506176e+06

# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.

# TYPE go_memstats_other_sys_bytes gauge

go_memstats_other_sys_bytes 1.210245e+06

# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.

# TYPE go_memstats_stack_inuse_bytes gauge

go_memstats_stack_inuse_bytes 688128

# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.

# TYPE go_memstats_stack_sys_bytes gauge

go_memstats_stack_sys_bytes 688128

# HELP go_memstats_sys_bytes Number of bytes obtained from system.

# TYPE go_memstats_sys_bytes gauge

go_memstats_sys_bytes 7.2284408e+07

# HELP go_threads Number of OS threads created.

# TYPE go_threads gauge

go_threads 13

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.

# TYPE process_cpu_seconds_total counter

process_cpu_seconds_total 4.53

# HELP process_max_fds Maximum number of open file descriptors.

# TYPE process_max_fds gauge

process_max_fds 1024

# HELP process_open_fds Number of open file descriptors.

# TYPE process_open_fds gauge

process_open_fds 8

# HELP process_resident_memory_bytes Resident memory size in bytes.

# TYPE process_resident_memory_bytes gauge

process_resident_memory_bytes 2.3400448e+07

# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.

# TYPE process_start_time_seconds gauge

process_start_time_seconds 1.57478066153e+09

# HELP process_virtual_memory_bytes Virtual memory size in bytes.

# TYPE process_virtual_memory_bytes gauge

process_virtual_memory_bytes 9.54318848e+08

# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.

# TYPE process_virtual_memory_max_bytes gauge

process_virtual_memory_max_bytes -1

# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.

# TYPE promhttp_metric_handler_requests_in_flight gauge

promhttp_metric_handler_requests_in_flight 1

# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.

# TYPE promhttp_metric_handler_requests_total counter

promhttp_metric_handler_requests_total{code="200"} 1014

promhttp_metric_handler_requests_total{code="500"} 0

promhttp_metric_handler_requests_total{code="503"} 0

# HELP smartthings_alarm_cleared 0 if the alarm is clear.

# TYPE smartthings_alarm_cleared gauge

smartthings_alarm_cleared{id="40bdab6e-96a7-49c7-94de-744a22e4244a",name="Master Smoke/CO Detector"} 0

smartthings_alarm_cleared{id="45f970d3-9c4a-40af-84b8-22b022dfe02b",name="Kitchen Smoke/CO Detector"} 0

smartthings_alarm_cleared{id="64635c2b-f82c-43b5-a96a-b5af2dc9e598",name="Upstairs Hallway Smoke/CO Detector"} 0

smartthings_alarm_cleared{id="7f15c986-3d3f-4a15-a0b5-e0129ed1cb85",name="Guest Bed Smoke/CO Detector"} 0

smartthings_alarm_cleared{id="bb5e5605-47b5-4877-a654-4ced63e36d51",name="Lennon's Room Smoke/CO Detector"} 0

smartthings_alarm_cleared{id="c1749d9a-ce21-4ee5-ba15-6adeaa9b2487",name="Basement Smoke/CO Detector"} 0

# HELP smartthings_battery_percentage Percentage of battery remaining.

# TYPE smartthings_battery_percentage gauge

smartthings_battery_percentage{id="03487219-5a70-43fb-83d9-7dca2b2a1d5b",name="Sunroom Door Sensor"} 100

smartthings_battery_percentage{id="0b2711f7-cb92-4c9f-bdc4-e3782414bf2f",name="Dining Room Multisensor"} 50

smartthings_battery_percentage{id="0c3cf5cf-98df-4ade-b6a9-cb2738b8d4a1",name="Back Door Sensor"} 100

smartthings_battery_percentage{id="127132a0-3d77-465b-a5af-99f0b8c32b57",name="Sunroom Door"} 100

smartthings_battery_percentage{id="1f1d2f79-f3e3-439d-a06c-75a9ecde5579",name="Right Patio Door Sensor"} 100

smartthings_battery_percentage{id="3eea2ef8-15b1-40e5-8217-399b4e161748",name="Front door sensor"} 24

smartthings_battery_percentage{id="40bdab6e-96a7-49c7-94de-744a22e4244a",name="Master Smoke/CO Detector"} 84

smartthings_battery_percentage{id="45f970d3-9c4a-40af-84b8-22b022dfe02b",name="Kitchen Smoke/CO Detector"} 87

smartthings_battery_percentage{id="47147a31-cc96-4712-ac6d-bc7c442e9a0a",name="Back Deck"} 92

smartthings_battery_percentage{id="59bc0caa-c71e-4b0a-9147-460c069fabfa",name="Lennon's Multisensor"} 100

smartthings_battery_percentage{id="64635c2b-f82c-43b5-a96a-b5af2dc9e598",name="Upstairs Hallway Smoke/CO Detector"} 88

smartthings_battery_percentage{id="687bc3ec-594c-46e2-90da-ee258e008368",name="Sun Room Door Light"} 42

smartthings_battery_percentage{id="6b5c83ac-ce60-43af-8fe5-8051fb5a78c9",name="Driveway Light"} 48

smartthings_battery_percentage{id="6cfeab8b-00b9-49c4-a9df-50c42e2cc905",name="Downstairs WallMote Quad"} 39

smartthings_battery_percentage{id="758783b4-7be5-4b8f-bba3-5b1e89542cc3",name="Back Deck Light"} 56

smartthings_battery_percentage{id="7f15c986-3d3f-4a15-a0b5-e0129ed1cb85",name="Guest Bed Smoke/CO Detector"} 81

smartthings_battery_percentage{id="844ad255-5f08-4b39-9eff-08866570cea3",name="Nanny Arrival Sensor"} 70

smartthings_battery_percentage{id="876726ba-9c09-41bc-a503-4c9451014978",name="Driveway Door Sensor"} 92

smartthings_battery_percentage{id="945b8651-b8f2-4580-b387-54455da733c2",name="Left Patio Door Sensor"} 56

smartthings_battery_percentage{id="a2788c30-d77c-46bc-8966-082c6899b43a",name="Driveway"} 100

smartthings_battery_percentage{id="af83e723-0ffb-4313-be1f-f356c76f0e9f",name="Front Door"} 52

smartthings_battery_percentage{id="bb5e5605-47b5-4877-a654-4ced63e36d51",name="Lennon's Room Smoke/CO Detector"} 86

smartthings_battery_percentage{id="c1749d9a-ce21-4ee5-ba15-6adeaa9b2487",name="Basement Smoke/CO Detector"} 90

smartthings_battery_percentage{id="c94eaa6d-7528-46ad-a790-1b8e711ce467",name="Kitchen Basement Door Sensor"} 100

smartthings_battery_percentage{id="cf54173d-16ed-4ff6-90be-9245931e3d08",name="Mailbox Sensor"} 41

smartthings_battery_percentage{id="e3798909-f8ed-4168-9014-8b5e0713ad85",name="Basement breaker light"} 96

smartthings_battery_percentage{id="edf921b7-e304-4d15-a194-d0ba4a059336",name="Guest WallMote"} 0

smartthings_battery_percentage{id="ffd694c7-9fff-489e-9823-a05b71daff5a",name="Front of House"} 100

# HELP smartthings_contact_closed 1 if the contact is closed.

# TYPE smartthings_contact_closed gauge

smartthings_contact_closed{id="03487219-5a70-43fb-83d9-7dca2b2a1d5b",name="Sunroom Door Sensor"} 1

smartthings_contact_closed{id="0c3cf5cf-98df-4ade-b6a9-cb2738b8d4a1",name="Back Door Sensor"} 1

smartthings_contact_closed{id="1f1d2f79-f3e3-439d-a06c-75a9ecde5579",name="Right Patio Door Sensor"} 1

smartthings_contact_closed{id="3eea2ef8-15b1-40e5-8217-399b4e161748",name="Front door sensor"} 1

smartthings_contact_closed{id="40bdab6e-96a7-49c7-94de-744a22e4244a",name="Master Smoke/CO Detector"} 0

smartthings_contact_closed{id="45f970d3-9c4a-40af-84b8-22b022dfe02b",name="Kitchen Smoke/CO Detector"} 0

smartthings_contact_closed{id="64635c2b-f82c-43b5-a96a-b5af2dc9e598",name="Upstairs Hallway Smoke/CO Detector"} 0

smartthings_contact_closed{id="7f15c986-3d3f-4a15-a0b5-e0129ed1cb85",name="Guest Bed Smoke/CO Detector"} 0

smartthings_contact_closed{id="876726ba-9c09-41bc-a503-4c9451014978",name="Driveway Door Sensor"} 0

smartthings_contact_closed{id="945b8651-b8f2-4580-b387-54455da733c2",name="Left Patio Door Sensor"} 1

smartthings_contact_closed{id="bb5e5605-47b5-4877-a654-4ced63e36d51",name="Lennon's Room Smoke/CO Detector"} 0

smartthings_contact_closed{id="c1749d9a-ce21-4ee5-ba15-6adeaa9b2487",name="Basement Smoke/CO Detector"} 0

smartthings_contact_closed{id="c94eaa6d-7528-46ad-a790-1b8e711ce467",name="Kitchen Basement Door Sensor"} 1

smartthings_contact_closed{id="cf54173d-16ed-4ff6-90be-9245931e3d08",name="Mailbox Sensor"} 1

# HELP smartthings_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which smartthings_exporter was built.

# TYPE smartthings_exporter_build_info gauge

smartthings_exporter_build_info{branch="HEAD",goversion="go1.11.3",revision="f77d40beb6474e35a501154bc376173dcb1fb41a",version="0.0.2"} 1

# HELP smartthings_invalid_metric Total number of metrics that were invalid.

# TYPE smartthings_invalid_metric counter

smartthings_invalid_metric 0

# HELP smartthings_motion_detected 1 if presence is detected.

# TYPE smartthings_motion_detected gauge

smartthings_motion_detected{id="0b2711f7-cb92-4c9f-bdc4-e3782414bf2f",name="Dining Room Multisensor"} 0

smartthings_motion_detected{id="127132a0-3d77-465b-a5af-99f0b8c32b57",name="Sunroom Door"} 0

smartthings_motion_detected{id="246659a9-b03d-4dff-961a-02b7cec8028c",name="Master Bed Sensor"} 0

smartthings_motion_detected{id="47147a31-cc96-4712-ac6d-bc7c442e9a0a",name="Back Deck"} 0

smartthings_motion_detected{id="57a3c6b3-ca4b-4066-8dec-56da0c19cdbf",name="Kitchen Sensor"} 1

smartthings_motion_detected{id="59bc0caa-c71e-4b0a-9147-460c069fabfa",name="Lennon's Multisensor"} 0

smartthings_motion_detected{id="673f592d-d39a-4ceb-a243-858d474a8fe9",name="Downstairs Bathroom"} 0

smartthings_motion_detected{id="687bc3ec-594c-46e2-90da-ee258e008368",name="Sun Room Door Light"} 0

smartthings_motion_detected{id="6b5c83ac-ce60-43af-8fe5-8051fb5a78c9",name="Driveway Light"} 0

smartthings_motion_detected{id="758783b4-7be5-4b8f-bba3-5b1e89542cc3",name="Back Deck Light"} 0

smartthings_motion_detected{id="a2788c30-d77c-46bc-8966-082c6899b43a",name="Driveway"} 0

smartthings_motion_detected{id="aea84a0e-c1dc-46e9-90f8-b2134c5f7d8e",name="Guest Bedroom Sensor"} 0

smartthings_motion_detected{id="af83e723-0ffb-4313-be1f-f356c76f0e9f",name="Front Door"} 0

smartthings_motion_detected{id="bd1c27be-2a54-4a6b-b412-cd8601260292",name="Upstairs Bathroom"} 0

smartthings_motion_detected{id="e3798909-f8ed-4168-9014-8b5e0713ad85",name="Basement breaker light"} 0

smartthings_motion_detected{id="faad395a-6ec4-4bd0-a28d-07f827cd42ed",name="Basement"} 0

smartthings_motion_detected{id="ffd694c7-9fff-489e-9823-a05b71daff5a",name="Front of House"} 0

# HELP smartthings_presence_detected 1 if presence is detected.

# TYPE smartthings_presence_detected gauge

smartthings_presence_detected{id="213bcf73-1c26-472e-ad46-60878f917b41",name="Ashley's Pixel 2"} 1

smartthings_presence_detected{id="844ad255-5f08-4b39-9eff-08866570cea3",name="Nanny Arrival Sensor"} 0

# HELP smartthings_smoke_detected 1 if smoke is detected.

# TYPE smartthings_smoke_detected gauge

smartthings_smoke_detected{id="40bdab6e-96a7-49c7-94de-744a22e4244a",name="Master Smoke/CO Detector"} 0

smartthings_smoke_detected{id="45f970d3-9c4a-40af-84b8-22b022dfe02b",name="Kitchen Smoke/CO Detector"} 0

smartthings_smoke_detected{id="64635c2b-f82c-43b5-a96a-b5af2dc9e598",name="Upstairs Hallway Smoke/CO Detector"} 0

smartthings_smoke_detected{id="7f15c986-3d3f-4a15-a0b5-e0129ed1cb85",name="Guest Bed Smoke/CO Detector"} 0

smartthings_smoke_detected{id="bb5e5605-47b5-4877-a654-4ced63e36d51",name="Lennon's Room Smoke/CO Detector"} 0

smartthings_smoke_detected{id="c1749d9a-ce21-4ee5-ba15-6adeaa9b2487",name="Basement Smoke/CO Detector"} 0

# HELP smartthings_switch_enabled 1 if the switch is on.

# TYPE smartthings_switch_enabled gauge

smartthings_switch_enabled{id="015cb4c3-01f7-491a-b83a-5bc65795dcc8",name="Main Bath Light Switch"} 1

smartthings_switch_enabled{id="06ffe7f6-8b7e-4532-99b7-eb5af15643f9",name="Master Bedroom Smart Plug"} 0

smartthings_switch_enabled{id="0a492585-4f21-4d23-896f-90682ec78fc9",name="Downstairs Hall Light"} 0

smartthings_switch_enabled{id="0a94cd3b-ee3c-4f47-9cc9-182f33965d82",name="Guest Bedroom Fan High"} 0

smartthings_switch_enabled{id="1249e8fd-01fc-40bf-9726-2dd3c46a3c21",name="Lennon's Room Fan Low"} 0

smartthings_switch_enabled{id="127132a0-3d77-465b-a5af-99f0b8c32b57",name="Sunroom Door"} 1

smartthings_switch_enabled{id="15cd9063-0233-4092-9848-6d8252f19ace",name="Master Bedroom Fan Comfort Breeze™"} 0

smartthings_switch_enabled{id="16857bf4-95ad-4ca7-a550-bfd28b55f8e5",name="Chandelier 5"} 1

smartthings_switch_enabled{id="190109fe-44c9-4709-bf35-ef1ebb58d609",name="Lennon's Fan Light"} 0

smartthings_switch_enabled{id="1a6b6976-c49b-4ad5-b3c9-723d4dac347f",name="Kitchen Can Light 5"} 1

smartthings_switch_enabled{id="2cb70fe1-a4af-49aa-8f2b-105b73e2404f",name="Back Deck Overhead Light"} 0

smartthings_switch_enabled{id="2d2ce7f4-b4af-4677-acf3-a7ba89a5b0a1",name="Sachshaus Arlo Base"} 0

smartthings_switch_enabled{id="2e96b91c-373f-456a-9166-56343d757afc",name="Kitchen Can Light 2"} 1

smartthings_switch_enabled{id="2eb7c4c7-bbd2-4280-b38d-03ba31c18698",name="Guest Bedroom Fan Med"} 0

smartthings_switch_enabled{id="31080ae9-77b8-4ba4-86db-6241248ef250",name="Small Lamp Living Room"} 1

smartthings_switch_enabled{id="32cacd70-c25b-4d15-9303-a3051b1d3843",name="Living Room - Back"} 1

smartthings_switch_enabled{id="37065022-fb1c-4621-a2ff-3ae942193aec",name="Patio"} 0

smartthings_switch_enabled{id="3892f04a-b390-4b6f-aab4-32c3484d913d",name="Kitchen Table"} 0

smartthings_switch_enabled{id="38ffdbf5-0a64-4d35-a63d-16bb96f61355",name="Chandelier 4"} 1

smartthings_switch_enabled{id="3c390cdd-14d8-4684-ba0c-3cb429dd3fd6",name="Kitchen Can Light 1"} 1

smartthings_switch_enabled{id="43055cac-5722-4624-8367-402b80cf04f0",name="Kitchen Can Light 3"} 1

smartthings_switch_enabled{id="47147a31-cc96-4712-ac6d-bc7c442e9a0a",name="Back Deck"} 1

smartthings_switch_enabled{id="47cc71dc-4d56-4d02-a3d2-1d82d12b27f6",name="Guest Bedroom Fan Med-Hi"} 0

smartthings_switch_enabled{id="483ecbb6-30f4-4adf-b645-c8195fcb4aa7",name="Living Room Smart Plug"} 1

smartthings_switch_enabled{id="4a05aaa7-1e8c-4154-8f94-d1dca0660a4a",name="Ashley Dimmer Plug"} 0

smartthings_switch_enabled{id="4c49cc88-ff86-4a9b-9893-8673d7bc59fa",name="Kitchen Island"} 1

smartthings_switch_enabled{id="4f296356-d138-441b-9576-da277203a62b",name="Guest Bedroom Lamp"} 0

smartthings_switch_enabled{id="4f6e78a1-899e-4c59-8673-509023e7618a",name="TV Right"} 1

smartthings_switch_enabled{id="52c5c384-e3a4-45f1-a445-cf28fe057a38",name="Lennon's Room Fan Med-Hi"} 0

smartthings_switch_enabled{id="54050708-2b92-4ad3-8c4c-9896c04b1e94",name="Chandelier 1"} 1

smartthings_switch_enabled{id="57c29ba2-c58f-440b-a504-b8c545d12196",name="Sun Room Fan Med"} 0

smartthings_switch_enabled{id="635a9d9e-cb24-4daf-b77e-b85596b7c8a8",name="Guest Bedroom Fan"} 1

smartthings_switch_enabled{id="6782b85b-af8e-42a4-b660-10acbe44269c",name="Guest Bedroom Fan Comfort Breeze™"} 1

smartthings_switch_enabled{id="687bc3ec-594c-46e2-90da-ee258e008368",name="Sun Room Door Light"} 0

smartthings_switch_enabled{id="6b5c83ac-ce60-43af-8fe5-8051fb5a78c9",name="Driveway Light"} 0

smartthings_switch_enabled{id="6fbbb2cb-ebc7-4120-97f0-0743978402c7",name="Coffee Cabinet Light"} 1

smartthings_switch_enabled{id="758783b4-7be5-4b8f-bba3-5b1e89542cc3",name="Back Deck Light"} 0

smartthings_switch_enabled{id="773b2311-7a2b-4c96-81fa-28a66e62002d",name="Entryway Light"} 0

smartthings_switch_enabled{id="7bbd264c-2679-45e7-ba66-d9095348efd6",name="Range 1"} 1

smartthings_switch_enabled{id="7e04ecbd-f8d6-4093-aed1-6a3929c6d27e",name="Master Bedroom Fan Med-Hi"} 1

smartthings_switch_enabled{id="809ecd17-15d2-4197-9a2b-0af9f9cda2ea",name="Kitchen Sink Cabinet"} 1

smartthings_switch_enabled{id="8111d614-7ea5-4e94-a58e-96af673aa34f",name="Chandelier 3"} 1

smartthings_switch_enabled{id="8133a045-f4d2-4e51-a779-4704b63dc726",name="Master Bedroom Fan Low"} 0

smartthings_switch_enabled{id="85a1c8e7-808c-433c-a499-93404f3dd387",name="Kitchen Can Light 4"} 1

smartthings_switch_enabled{id="8877471c-8fe8-40e9-855f-f237d7ce2b1e",name="Patio Sonos Speaker"} 1

smartthings_switch_enabled{id="890b7e02-d6ac-4df3-a34c-7c88b07e81e1",name="Sun Room Fan Med-Hi"} 0

smartthings_switch_enabled{id="905c59b2-7604-4659-8f57-cd1423513b44",name="Master Bedroom Fan Light"} 1

smartthings_switch_enabled{id="953800e8-9a9c-4c19-8574-8c6b0c615c09",name="TV Left"} 1

smartthings_switch_enabled{id="9c8f2d46-ac28-4241-a2b0-b9176dcb6975",name="Master Bedroom Fan"} 1

smartthings_switch_enabled{id="a16fafd3-8da2-43f7-b66e-226f533307a0",name="Master Bedroom Fan High"} 0

smartthings_switch_enabled{id="a2788c30-d77c-46bc-8966-082c6899b43a",name="Driveway"} 1

smartthings_switch_enabled{id="a39daa5b-9a9a-476f-91dd-1638651f711c",name="Sun Room Fan Comfort Breeze™"} 0

smartthings_switch_enabled{id="a44125df-7b1c-41bb-985a-108a59573851",name="Lennon's Room Fan Comfort Breeze™"} 0

smartthings_switch_enabled{id="ad8a4753-1349-4b8b-bb89-9af7305a5df3",name="Guest Bedroom Smart Plug"} 1

smartthings_switch_enabled{id="ae283b7a-769e-4061-98f0-0c22c5b4edb1",name="Lennon's Room Fan"} 0

smartthings_switch_enabled{id="aec01b91-6c7b-4e1d-8cb0-015a96c4507d",name="Living Room - Front"} 1

smartthings_switch_enabled{id="af83e723-0ffb-4313-be1f-f356c76f0e9f",name="Front Door"} 1

smartthings_switch_enabled{id="bcd398e7-5dd1-4cef-87c8-9709cd717db5",name="Guest Bedroom Fan Low"} 0

smartthings_switch_enabled{id="caa2f9e7-f781-43cb-8987-4d40339b8279",name="Front Door Lights"} 0

smartthings_switch_enabled{id="cbde5d32-d40c-43f0-be7f-8eb70df727f2",name="Range Extender"} 1

smartthings_switch_enabled{id="ce356136-9992-4087-a8c1-fea3bccfb65c",name="Master Bedroom Fan Med"} 0

smartthings_switch_enabled{id="ce4d5018-793e-46da-b0ff-c29ce10af296",name="Sun Room Fan Light"} 0

smartthings_switch_enabled{id="cf0e47bc-41bb-4b83-9345-14e476960304",name="Sun Room Fan High"} 0

smartthings_switch_enabled{id="d22ed617-ed59-4d64-8cd0-68e7d15559c8",name="Range 2"} 1

smartthings_switch_enabled{id="e240ff11-c046-43bd-8874-c57cc7f03108",name="Lennon's Room Fan Med"} 0

smartthings_switch_enabled{id="e2e05a25-2495-4146-b69e-4dd05517c5bb",name="Chandelier 2"} 1

smartthings_switch_enabled{id="e3798909-f8ed-4168-9014-8b5e0713ad85",name="Basement breaker light"} 0

smartthings_switch_enabled{id="e60ff3a2-81cf-42d1-a5c2-664d829dc94e",name="Sun Room Fan Low"} 0

smartthings_switch_enabled{id="ea4bc661-140d-4524-83af-e054c37b25d9",name="Aaron Dimmer Plug"} 0

smartthings_switch_enabled{id="ec1125fd-c780-4373-8b80-687c402c10cd",name="Patio Lights"} 0

smartthings_switch_enabled{id="edad6bb5-52a6-40d7-b044-161a5eb87721",name="Guest Bedroom Fan Light"} 0

smartthings_switch_enabled{id="f1718594-8f71-4fa1-bd45-d7f4a2683765",name="Sun Room Fan"} 0

smartthings_switch_enabled{id="f90ee6a2-994b-4a0b-991e-e35454d85837",name="Kitchen Island Can Light"} 0

smartthings_switch_enabled{id="f94d4c87-538f-4051-b6a3-025affac506b",name="Kitchen Can Light 6"} 1

smartthings_switch_enabled{id="fd8f3243-f562-4409-bdc8-0b7635cf4ca4",name="Lennon's Lamp"} 0

smartthings_switch_enabled{id="fe5e3883-b0c1-48a3-bf1b-454c973177b5",name="Lennon's Room Fan High"} 0

smartthings_switch_enabled{id="ffd694c7-9fff-489e-9823-a05b71daff5a",name="Front of House"} 1

# HELP smartthings_temperature_fahrenheit Temperature in fahrenheit.

# TYPE smartthings_temperature_fahrenheit gauge

smartthings_temperature_fahrenheit{id="0765ab0d-1ff0-4942-87df-82cd0ab1327d",name="Sachshaus"} 69

smartthings_temperature_fahrenheit{id="0b2711f7-cb92-4c9f-bdc4-e3782414bf2f",name="Dining Room Multisensor"} 71.4

smartthings_temperature_fahrenheit{id="246659a9-b03d-4dff-961a-02b7cec8028c",name="Master Bed Sensor"} 71

smartthings_temperature_fahrenheit{id="57a3c6b3-ca4b-4066-8dec-56da0c19cdbf",name="Kitchen Sensor"} 68

smartthings_temperature_fahrenheit{id="59bc0caa-c71e-4b0a-9147-460c069fabfa",name="Lennon's Multisensor"} 67.1

smartthings_temperature_fahrenheit{id="673f592d-d39a-4ceb-a243-858d474a8fe9",name="Downstairs Bathroom"} 78

smartthings_temperature_fahrenheit{id="aea84a0e-c1dc-46e9-90f8-b2134c5f7d8e",name="Guest Bedroom Sensor"} 73

smartthings_temperature_fahrenheit{id="bd1c27be-2a54-4a6b-b412-cd8601260292",name="Upstairs Bathroom"} 64

smartthings_temperature_fahrenheit{id="faad395a-6ec4-4bd0-a28d-07f827cd42ed",name="Basement"} 81

I'm not quite sure where to begin on troubleshooting the delta here, and I'm now a bit curious if it's missing data from other endpoints.
@ipaqmaster
Copy link

ipaqmaster commented Jan 13, 2020 •
edited
Loading

Hi,

I hope this is relevant to your issue too.

Spent this afternoon troubleshooting this problem. Your formatting reads similar to mine may have the same solution.

I also had missing metrics and the the solution for my case was making sure the label_value for any labels in any metric line didn't contain any whitespace characters, even though quoted.

E.G

Fails, not visible in output:
net_bytes_recv{env="None",host="myPC",interface="My Ethernet Adapter Name"} 123456
Succeeds, visible in output:
net_bytes_recv{env="None",host="myPC",interface="MyEthernetAdapterName"} 123456

After removing white space present in the interface="xxxx" label's label_value by renaming the adapter on the affected host, the problem went away and sensu-prometheus-collector started including it in stdout.

@ipaqmaster
Copy link

Doesn't seem to be any new information about this since my last post so I've made a quick little workaround for myself.

This /missing metric problem/ only seems to occur when using the default -output-format influx format with a metric containing whitespace in any tag names or values. I found that using -output-format json included the lines missing from the influx format.

I think main.go:L111-0114 is where these lines vanish. This sanity check looks for exactly 3 segments per influx metric but is whitespace delimited. This is likely causing our whitespace'd tag names/values to "fail" and not be passed into the final output in the (metrics variable).

My "for_now" solution has been to replace the whitespace with underscores in the tag parsing section so they're sent and visible in my network graphs.

This involved replacing line 95 of main.go:L95
tags := fmt.Sprintf(",%s=%s", name, value)
with:
tags := strings.Replace(fmt.Sprintf(",%s=%s", name, value), " ", "_", -1)

To replace any whitespace of generated tags with an underscore via the strings.Replace function. While this isn't very clean nor ideal, it's allowed me to see and include include those network ints in my Grafana graphs once I built and tested it. So there's my personal solution for the time being.

This isn't an influx limitation as I crafted a fake metric with curl http://127.0.0.1:8086/write?db=my_sensu_db --data-binary 'testmeasurement,testtag=y\ es value=12345'. and was able to select it in grafana against this database. The tag testtag had the correct value of y es (with the whitespace). Perhaps sensu-go's output_metric_format: influxdb_line check setting has strict formatting expectations for this requirement? I might test that later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@asachs01 @ipaqmaster