Additions to the Monitoring API

[acchen97]

Next round of per plugin stats for the monitoring API:

• Beats input
  • beat_types - Beats currently connected and their counts, i.e. Filebeat 20, Metricbeat 6, etc.
• Elasticsearch output
  • bulk_requests - number of bulk requests sent PR add a few metrics to the plugin logstash-plugins/logstash-output-elasticsearch#585
    • Success count
    • Failure count
  • concurrent_bulk_requests - Monitoring # of concurrent bulk requests #6544 PR add a few metrics to the plugin logstash-plugins/logstash-output-elasticsearch#585
  • retry_count - number of retries PR add a few metrics to the plugin logstash-plugins/logstash-output-elasticsearch#585
  • sniffing_count - number of times sniffing was triggered PR add a few metrics to the plugin logstash-plugins/logstash-output-elasticsearch#585
  • HTTP response code count, buckets i.e. 2xx, 3xx, 4xx, 5xx

Elasticsearch output metrics example payload:

{
  "bulk_requests": {
     "responses": {  # counts per top level status code returned in the bulk response
       200: 346300,
       201: 712,
       404: 61,
       429: 36
       # other status codes ...
     },
     "successes": 346819,  # successful bulk requests with no errors (sum of 200 and 201 minus partial_successes)
     "failures": 23,         # no http response from ES. will be retried. see point 1 above for causes
     "partial_successes": 134,  # bulk request successful (status 20X) but contains errors
     "drops": 5,   # bulk request dropped
  },
  "documents": {
       "successes": 173409500,
       "retryable_failures": 450,
       "non_retryable_failures": 12
  },
  "sniffing_count": 621
}

[pemontto]

I would really like to see something like 1, 5 and 15 log rate per second. Currently I have to query multiple times and store the number of processed logs to calculate the rate over time. Unless I'm missing something entirely.

[acchen97]

@pemontto have you checked out the Logstash Monitoring UI? It's an X-Pack feature, but under the Basic license so therefore free to all users alike.

https://www.elastic.co/guide/en/x-pack/current/monitoring-logstash.html

[jsvd]

I'm adding here an entry to desirable metrics for the http output: logstash-plugins/logstash-output-http#82

[jsvd]

beats input already provides a count for concurrent and peak connection counts, the only thing missing is the count per beat type, which required extra metadata in the lumberjack protocol, so closing this, want we can reopen an issue on the beats input repo for a specific requirement.