Add otel instrumentation for http-grpc gateway

[py4chen]

Currently no metrics are collected if the requests come through grpc-gateway because grpc interceptors don't work in http request flow.

PR adds otel instrumentation to http gmux, and collects metrics for panic.

--

Verification

• Run a local otel collector container

$ cat <<EOF > ./local-collector/otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  debug:
    verbosity: detailed

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, prometheus]
EOF

$ docker run --rm --network host -p 4317:4317 -p 4318:4318 -p 8889:8889 -v ./local-collector/otel-collector-config.yaml:/etc/otel-collector-config.yaml otel/opentelemetry-collector:latest --config /etc/otel-collector-config.yaml

• Modify Crypki sourcecode adding panic intentionally, and run the crypki container
• Send requests to the container

❯ curl -X GET https://localhost:4443/v3/sig/ssh-host-cert/keys --cert tls-crt/client.crt --key tls-crt/client.key --cacert tls-crt/ca.crt -v

❯ curl -X GET https://localhost:4443/v3/sig/ssh-user-cert/keys --cert tls-crt/client.crt --key tls-crt/client.key --cacert tls-crt/ca.crt -v

• find metrics

# HELP http_handler_panic_total Count the number of HTTP handler panic
# TYPE http_handler_panic_total counter
http_handler_panic_total{http_method="GET",http_target="/v3/sig/ssh-user-cert/keys",job="crypki",panic_message="debug panic!"} 2
...
cert/keys",job="crypki",net_host_name="localhost",net_host_port="4443",net_protocol_name="http",net_protocol_version="2.0"} 0.742625
http_server_duration_milliseconds_count{http_method="GET",http_scheme="https",http_status_code="200",http_target="/v3/sig/ssh-host-cert/keys",job="crypki",net_host_name="localhost",net_host_port="4443",net_protocol_name="http",net_protocol_version="2.0"} 1
...

--

I confirm that this contribution is made under the terms of the license found in the root directory of this repository's source tree and that I have the authority necessary to make this contribution on behalf of its copyright owner.

[py4chen]

@dependenbot rebase