Memory grows slowly

[dxsup]

Describe the bug

The memory of the agent grows slowly.

Screenshots

The screenshot from a user shows that the memory usage of some agents is over 1GiB.

Trouble shooting
Run the following command:

curl http://IP:6060/debug/pprof/heap -o heap
go tool pprof heap
> web

We can get the following heap graph:

It shows the inuse heap of kindling-collector is only 365MiB. So the kindling-probe may use more than 700MiB memory.

Environment (please complete the following information)

• Kindling agent version: v0.8.0
• Node Kernel version: 5.4.240-1.el7.elrepo.x86_64

[masterjyq]

Memory usage graph

Environment (please complete the following information)

• Kindling agent version: v0.8.0
• Node Kernel version: 4.19.90-17.ky10.x86_64

[masterjyq]

Memory usage graph Environment (please complete the following information)

• Kindling agent version: v0.8.0
• Node Kernel version: 4.19.90-17.ky10.x86_64

After OOM

[xuchuan-666]

配置文件networkanalyzer.url_clustering_method改成blank，来降低指标数量后观察一天，agent服务内存还是一直上涨，并oom

[dxsup]

目前已知指标kindling_trace_request_duration_nanoseconds的cardinality较高，该指标数量会持续增加导致探针内存持续升高。我们正在尝试降低指标持续堆积对内存的影响。

在该优化实现前，可以先通过将配置exporters.otelexporter.adapter_config.need_trace_as_metric修改为false关闭该指标，这样可以显著减缓内存上升的速度。