[x-pack][metricbeat][iis] improve error handling on pdh counters in application_pool data stream

[stefans-elastic]

Proposed commit message

The error occurs initAppPools > calls GetCounterPaths > calls ExpandWildCardPath > calls PdhExpandWildCardPath which returns no data and no error > PdhExpandWildCardPathreturns PdhErrno(syscall.ERROR_NOT_FOUND) error here > the error returns all the way back to initAppPools and since the error type isn't part of this condition an error log appears (logged here).

On why PDH's function returns no data and no error - I didn't find a coverage of this behavior in the docs. My guess is that object/counter exists but has no data in it (for w3wp process).
The fix I suggest is simply adding a check for PdhErrno(syscall.ERROR_NOT_FOUND) error in this condition which prevents those errors causing unnecessary error level log entries.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Closes [IIS] [application_pool] Element Not Found for Application Pools with 'No Managed Code' #41596

Screenshots

Before the change (Element not found.failed to expand counter path errors show up in the logs)

After the change (NO Element not found.failed to expand counter path errors show up in the logs)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @stefans-elastic? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[shmsr]

diff --git a/x-pack/metricbeat/module/iis/application_pool/reader.go b/x-pack/metricbeat/module/iis/application_pool/reader.go
index 61522ba128..e11e1381f0 100644
--- a/x-pack/metricbeat/module/iis/application_pool/reader.go
+++ b/x-pack/metricbeat/module/iis/application_pool/reader.go
@@ -10,6 +10,7 @@ import (
 	"errors"
 	"fmt"
 	"strings"
+	"syscall"
 
 	"github.com/elastic/beats/v7/metricbeat/helper/windows/pdh"
 	"github.com/elastic/elastic-agent-libs/mapstr"
@@ -111,16 +112,26 @@ func (r *Reader) initAppPools() error {
 		r.log.Info("no running application pools found")
 		return nil
 	}
+
+	isPDHError := func(err error) bool {
+		return errors.Is(err, pdh.PdhErrno(syscall.ERROR_NOT_FOUND)) ||
+			errors.Is(err, pdh.PDH_CSTATUS_NO_COUNTER) ||
+			errors.Is(err, pdh.PDH_CSTATUS_NO_COUNTERNAME) ||
+			errors.Is(err, pdh.PDH_CSTATUS_NO_INSTANCE) ||
+			errors.Is(err, pdh.PDH_CSTATUS_NO_OBJECT)
+	}
+
 	var newQueries []string
 	r.workerProcesses = make(map[string]string)
 	for key, value := range appPoolCounters {
 		childQueries, err := r.query.GetCounterPaths(value)
 		if err != nil {
-			if errors.Is(err, pdh.PDH_CSTATUS_NO_COUNTER) || errors.Is(err, pdh.PDH_CSTATUS_NO_COUNTERNAME) || errors.Is(err, pdh.PDH_CSTATUS_NO_INSTANCE) || errors.Is(err, pdh.PDH_CSTATUS_NO_OBJECT) {
+			if isPDHError(err) {
 				r.log.Infow("Ignoring non existent counter", "error", err,
-					logp.Namespace("application pool"), "query", value)
+					logp.Namespace("application pool"), "query", value,
+				)
 			} else {
-				r.log.Error(err, `failed to expand counter path (query= "%v")`, value)
+				r.log.Errorf(`failed to expand counter path (query= "%v"): %w`, value, err)
 			}
 			continue
 		}

Your changes look but I do have some suggestion to make the change more readable. Also, the log "failed to expand counter path" has a bug i.e., it does have format parameter that accepts %v; now it will. I have fixed it, so could you please make that change as well?

[shmsr]

^ Just one comment; rest looks good.