Add SDK span telemetry metrics

[JonasKunz]

Changes

With this PR I'd like to start a discussion around adding SDK self-monitoring metrics to the semantic conventions.
The goal of these metrics is to give insights into how the SDK is performing, e.g. whether data is being dropped due to overload / misconfiguration or everything is healthy.
I'd like to add these to semconv to keep them language agnostic, so that for example a single dashboard can be used to visualize the health state of all SDKs used in a system.

We checked the SDK implementations, it seems like only the Java SDK currently has some health metrics implemented.
This PR took some inspiration from those and is intended to improve and therefore supersede them.

I'd like to start out with just span related metrics to keep the PR and discussions simpler here, but would follow up with similar PRs for logs and traces based on the discussion results on this PR.

Prior work

This PR can be seen as a follow up to the closed OTEP 259:

• The OTEP originally superseded Add processed/exported Span metrics. #184, which initially focused only on SDK exporter metrics.
• Add processed/exported Span metrics. #184 was closed in favor of the predecessor of OTEP 259 to instead allow monitoring of entire "pipelines" using unified metrics across SDK exporters and collector components : A pipeline starts with an SDK exporter and goes through the processing of collector(s)
• Finally, that OTEP was closed in favor of this collector RFC which adds metrics just for the collector components.

So we kind of have gone full circle: The discussion started with just SDK metrics (only for exporters), going to an approach to unify the metrics across SDK-exporters and collector, which then ended up with just collector metrics.
So this PR can be seen as the required revival of #184 (see also this comment).

In my opinion, it is a good thing to separate the collector and SDK self-metrics:

• There have been concerns about both using the same metrics for both: How do you distinguish the metrics exposed by collector components from the self-monitoring metrics exposed by an Otel-SDK used in the collector for e.g. tracing the collector itself?
• Though many concepts between the collector and SDK share the same name, they are not the same thing (to my knowledge, I'm not a collector expert): For example processors in the collector are designed to form pipelines potentially mutating the data as it passes through. In contrast, SDK span processor don't form pipelines (at least not visible to the SDK, those would be hidden custom implementations). Instead SDK span processors are merely observers with multiple callbacks for the span lifecycle. So it would feel like "shoehorning" things into the same metric, even though they are not the same concepts.
• Separating collector and SDK metrics makes their evolution and reaching agreements a lot easier: When using separate metrics and namespaces, collector metrics can focus on the collector implementation and SDK metrics can be defined just using the SDK spec. If combine both in shared metrics, those will have to be always be aligned with both the SDK spec and the collector implementation. I think this would make maintenance much harder for little benefit.
• I have a hard time finding benefits of sharing metrics for SDK and collector: The main benefit I find would of course be easier dashboarding / analysis. However, I do think having to look at two sets of metrics to do so is a fine tradeoff, considering the difficulties with the unification listed above and shown by the history of OTEP 259.

Existing Metrics in Java SDK

For reference, here is what the existing health metrics currently look like in the Java SDK:

Batch Span Processor metrics

• Gauge queueSize, value is the current size of the queue
  • Attribute spanProcessorType=BatchSpanProcessor (there was a former ExecutorServiceSpanProcessor which has been removed)
  • This metric currently causes collisions if two BatchSpanProcessor instances are used
• Counter processedSpans, value is the number of spans submitted to the Processor
  • Attribute spanProcessorType=BatchSpanProcessor
  • Attribute dropped (boolean), true for the number of spans which could not be processed due to a full queue

The SDK also implements pretty much the same metrics for the BatchLogRecordProcessor just span replaced everywhere with log

Exporter metrics

Exporter metrics are the same for spans, metrics and logs. They are distinguishable based on a type attribute.
Also the metric names are dependent on a "name" and "transport" defined by the exporter. For OTLP those are:

• exporterName=otlp
• transport is one of grpc, http (= protobuf) or http-json

The transport is used just for the instrumentation scope name: io.opentelemetry.exporters.<exporterName>-<transport>

Based on that, the following metrics are exposed:

• Counter <exporterName>.exporter.seen: The number of records (spans, metrics or logs) submitted to the exporter

  • Attribute type: one of span, metric or log

• Counter <exporterName>.exporter.exported: The number of records (spans, metrics or logs) actually exported (or failed)

  • Attribute type: one of span, metric or log
  • Attribute success (boolean): false for exporter failures

Merge requirement checklist

• CONTRIBUTING.md guidelines followed.
• Change log entry added, according to the guidelines in When to add a changelog entry.
  • If your PR does not need a change log, start the PR title with [chore]
• schema-next.yaml updated with changes to existing conventions.

[lmolkova]

Related #1580

[reyang]

Is there some information to tell the application restart? (e.g. PID + start_time)

[lmolkova]

we have uptime metric for this -

semantic-conventions/docs/system/process-metrics.md

Line 341 in 21c230d

### Metric: `process.uptime`

[reyang]

Do we allow each language implementations to have additional attributes that are language specific?

[JonasKunz]

I don't see a reason why implementations shouldn't be allowed to add additional attributes. I would expect this to be the general case for all semconv metrics? Metrics are aggregateable, so they can be analyzed and presented as if those additional attributes weren't present.

There are two caveats I can think of:

• The metrics are recommended to be enabled by default. Therefore they must have a very, very low cardinality to justify this and not cause to much overhead. So depending on the cardinality of the additional attributes, they should probably be opt-in.
• The attributes might conflict with future additions to the spec, so you'll end up with breaking changes. So best to use some language-specific attribute naming.

[reyang]

What about retry? - e.g. the first attempt failed, the second attempt succeeded.

[lmolkova]

we don't record intermediate results on logical metrics, we could report another layer like otel.sdk.span.exporter.attempts (or let HTTP/grpc metric instrumentation do its thing).

[lmolkova]

we don't record intermediate results on logical metrics, we could report another layer like otel.sdk.span.exporter.attempts (or let HTTP/grpc metric instrumentation do its thing).

[lmolkova]

could we do

Suggested change

	metric_name: otel.sdk.span.processor.spans_processed
	metric_name: otel.sdk.processor.span.count

to avoid repeating span ?

[JonasKunz]

otel.sdk.processor.span.count doesn't specify what kind of spans it counts. What about
otel.sdk.processor.span.processed.count just like otel.sdk.exporter.span.exported.count?

I'd also switch to otel.sdk.processor.span.queue.capacity and otel.sdk.processor.span.queue.size then to keep the namespaces consistent

[lmolkova]

sounds great!

[lmolkova]

do we need to repeat otel.sdk everywhere? can we do otel ? it's pretty obvious it's about SDK and we usually omit obvious things in attribute and metric names.

If we just stick to otel, there is a chance collector could reuse some of the attributes and metrics

[JonasKunz]

The Otel SDK batching span processor (defined by the spec) for example is different from the collector batch processor.

SDK and collector have different concepts and specifications, therefore evolve differently. That's why I think it causes more confusion trying to combine those instead of accepting bits of duplication and keeping them separated. See also the "Prior Work" section of the PR description.

To give a concrete example, imagine we add a otel.component.cpu_usage metric to quantify the overhead of a component.
You now have in a collector:

• a collector batch span processor processing incoming OTLP data
• The Otel SDK monitoring the collector itself. It exports the monitoring data (e.g. spans about collector components) via a SDK batching span processor.

You know encounter the otel.component.cpu_usage with otel.component.type=batch_span_processor. Which of the processors does it correspond to? This won't happen if you use otel.sdk and otel.collector namespaces.

So to summarize: I think due to the fact that sdk and collector use similar names to talk about different things, it makes sense to use the sdk namespace.

[lmolkova]

Do I remember correctly that collector uses otelcol as a metric namespace? would we change it to otel.collector?

My main motivation for this proposal is

do we need to repeat otel.sdk everywhere? can we do otel ? it's pretty obvious it's about SDK and we usually omit obvious things in attribute and metric names.

if we use otel for otel SDK (resource attributes along with component names should make it obvious that it's reported by the SDK), and otelcol for collector, then we keep SDK metrics nice and short and there is no ambiguity.

[JonasKunz]

that collector uses otelcol as a metric namespace?

I think so, yes.

would we change it to otel.collector

I think we could, but I also think it isn't necessary. With my reasoning I just wanted to make sure that we use separate namespaces for collector and SDK to avoid confusion.

Placing SDK metrics directly in otel.* is fine from a maintenance / evolution perspective imo as long as collector stuff always uses otel.otelcol (or whatever namespace is selected).

From a useability perspective I prefer having metric/attribute names in isolation being precise and self-explanatory.
From that perspective e.g. looking at just otel.span.processor.processed.count doesn't make it obvious whether this an SDK metric or a collector metric, you'd need to dig deeper into the definition. In contrast otel.sdk.span.processor.processed.count is unambigous.

The useability perspective is just a matter of taste, so I'm okay with removing the .sdk. namespace from all metrics and attributes if you disagree here

[lmolkova]

You convinced me on the metric names, but I feel we don't need to have distinct attribute names for collector and SDK.
E.g. otel.component.type sounds perfectly fine and the metric name would give it all the necessary context. WDYT?

[lmolkova]

I think we should include recommended server.address and server.port attributes on exporter metrics. It's good to know where you are sending data to.

[dashpole]

Those would not apply to all exporters (e.g. stdout). My thinking is that we should encourage using protocol-level instrumentation (e.g. http/gRPC) for details like this.

[JonasKunz]

Kind of agree with @dashpole here. I don't think this belongs in this metric.
Nonetheless, I think it would make sense to add exporter.request.* metrics to track request stats (e.g. bytes sent, response codes, server details). However, I don't think that this should happen in this PR, but rather in a separate, follow-up PR. It is an enhancement to gain more fine grained insights in addition to the metrics to this PR, but doesn't have an impact on them.

[lmolkova]

Those would not apply to all exporters (e.g. stdout).

not a problem, just add them with requirement level recommended: when applicable. We do include these attributes on logical operations across semconv, so they do belong here.

[JonasKunz]

Added in 830edfb.

However, for attribute references for metrics there is no schema.compliant recommended: when applicable IINM.
I added note: recommended when applicable instead, please let me know if this is the correct approach.

[dashpole]

A question came up when I was implementing this: Should this include non-recording spans? Right now, non-recording spans are essentially no-op spans. Adding instrumentation to them might have performance implications, since the overhead of non-recording spans is currently close to zero.

[JonasKunz]

Good question.
I think it is very valuable to have a way of computing the effective sampling rate. This implies that we need the number of unrecorded spans, because the number of recorded unsampled spans is only a subset of the total number of unsampled spans.

I think we should however do this by adding a separate sampling-result metric (using the tri-state sample_result attribute suggested here. This means unrecorded spans don't need to track their liveness or end and we can still easily compute the effective sampling rate.

Alternatively, we could add back the created_count metric and enforce the tri-state sampled attribute from this comment.
For live_count and ended_count we could either:

• Disallow them to be tracked for unrecorded spans: I think this would lead to confusion due to the mismatch when looking at the aggregated created_count and ended_count metrics.
• Force live_count and ended_count to be tracked for unrecorded spans and omit created_count: This would means we have the overhead of tracking two metrics instead of one and TracerProviders wouldn't be able to return a simple NooOp span, but one which tracks the end() call exactly once.

That's why I'm thinking adding a separate sampler metric is the best compromise. However, I think we should do this in a separate PR for sampler metrics I'd say. WDYT?

[dashpole]

Raised this at the Go SIG today. I think we should include metrics for non-recording spans to start, since they are very useful. When we implement this, we can benchmark the actual implications of this decision. But since most instrumentation libraries that record a span also make a metric observation for each request, this shouldn't be a huge deal. If it turns out to be bad, we can revisit this.

[JonasKunz]

I think we should include metrics for non-recording spans to start, since they are very useful

Could you elaborate more, I'm not sure if you mean any of my proposed solutions or a different one?

And what purpose are you thinking of with they are very useful ?

• to get the effective sampling rate
• to detect span leaks (missing end() calls) even for unrecorded spans
• Something else?

[dashpole]

Both getting the sampling rate and detecting span leaks. I like your proposed solution.

[JonasKunz]

So you mean we change the spec and enforce this proposal?

Force live_count and ended_count to be tracked for unrecorded spans and omit created_count: This would mean we have the overhead of tracking two metrics instead of one and TracerProviders wouldn't be able to return a simple NooOp span, but one which tracks the end() call exactly once

So we say implementation MUST track live/ended instead of MAY track for unrecorded spans ?
In the current state of my PR it is optional for implementations to track these for unrecorded spans.

[dashpole]

I would prefer SHOULD for now. If there are languages where it really isn't feasible, then they should probably omit the non-sampled metric entirely. But that kind of language really belongs in the SDK specification, rather than in the convention.

[dashpole]

I completed the Go prototype of the proposed semantic conventions: open-telemetry/opentelemetry-go#6153

[dashpole]

I think we should be a bit more prescriptive here and provide example values. When I was implementing this, it wasn't clear how granular I should be. Should this just always be "rejected" if the backend returned an error code? Or should it be more specific, like a gRPC status code: "deadline_exceeded" or "invalid_argument".

Personally, I prefer a more restrictive set of values for the error, like "rejected", "dropped", "timeout", but this metric will be much more useful for users if exporters use consistent values for this.

[JonasKunz]

IMO the main value of the metric here comes from detecting whether there is an error or not rather than having the same failure reasons across languages. The metrics are about detecting failures, for analyzing/mitigating failures you'd usually dive deeper (e.g. inspect the logs of the failing services).

I think for example http.server.request.duration is a suitable comparison here. That metrics also doesn't prescribe values for error.type, even though HTTP protocol errors are well defined (though there are very many).

Describing concrete error.type values would be even harder, because they will highly depend on the exporter protocol. Also where to stop? E.g. timeout or dns-timeout / response-timeout?

That's why I would say just use what makes sense for your language and what you'd use when writing e.g. an HTTP instrumentation.

[dashpole]

That seems reasonable. An enum would definitely be too restrictive, and I agree SDKs need the ability to deviate when it makes sense. WDYT about providing example values as an anchor for style and granularity? It just seems annoying to not be able to group (or to have to write a complex query) just because one language chose "timeout" and another chose "Timeout".

[JonasKunz]

Sure, we can add more example attribute values: rejected is already there, we can add timeout.
With dropped I don't know how this fits in here: When an error occurs, data is dropped. But "dropped" isn't the cause, it's the result what happens with the data.

Any other ones you can think of? Maybe just 500 and java.net.UnknownHostException from the HTTP error.type examples to also have some protocol / language specific examples?

[dashpole]

Yeah, dropped probably doesn't make sense in this context (didn't think it through entirely). java.net.UnknownHostException is probably a good one to include as well. It paints the correct picture: use a common error string when possible, otherwise use a language-specific one.