[Question] http_log_x_request_id 添加自定义 Header 链路追踪问题

[gbling]

Search before asking

• I had searched in the issues and found no similar feature requirement.

DeepFlow Component

Agent

What you expected to happen

在测试分布式链路追踪的时候遇到的情况。

全链路使用的都是http1.1，系统中链路存在跨线程调用的情况，测试链路中 A、B、C 三个系统均添加Header（tq-trace-id）在链路中进行传递，deepflow-agent 配置 http_log_x_request_id、http_log_trace_id 也都添加了tq-trace-id，有发现请求同个接口多次，最终出来的结果是不同的，有时测试结果发现链路不完整（抓包是都能看到有正常传递tq-trace-id），如图

第一次测试结果

第二次测试结果

第三次测试结果

另外与 https://github.com/deepflowio/deepflow/issues/8237、https://github.com/deepflowio/deepflow/issues/8242 应该不是同个问题

How to reproduce

kubectl exec -it -n deepflow deploy/deepflow-server -- deepflow-server -v

Name: deepflow-server community edition
Branch: v6.6.5
CommitID: eb43fab
RevCount: 11188
Compiler: go version go1.21.13 linux/amd64
CompileTime: 2024-09-25 06:14:29

DeepFlow version

kubectl exec -it -n deepflow ds/deepflow-agent -- deepflow-agent -v

Defaulted container "deepflow-agent" out of: deepflow-agent, configure-sysctl (init)
11188-eb43fab700b5fa0a03764f0146f4c8ef4724cd1f
Name: deepflow-agent community edition
Branch: v6.6.5
CommitId: eb43fab
RevCount: 11188
Compiler: rustc 1.77.1 (7cf61ebde 2024-03-27)
CompileTime: 2024-09-25 06:15:33

DeepFlow agent list

deepflow-ctl agent list --api-port 32702

ID NAME TYPE CTRL_IP CTRL_MAC STATE GROUP EXCEPTIONS REVISION UPGRADE_REVISION
1 10.173.253.171-V6 K8S_VM 10.173.253.171 00:16:3e:09:f0:09 NORMAL default v6.6.5 11188
2 10.173.253.172-V7 K8S_VM 10.173.253.172 00:16:3e:14:a9:93 NORMAL default v6.6.5 11188
3 10.173.253.173-V10 K8S_VM 10.173.253.173 00:16:3e:05:72:38 NORMAL default v6.6.5 11188
4 10.173.252.128-V23 K8S_VM 10.173.252.128 00:16:3e:0f:3c:61 NORMAL default v6.6.5 11188
5 10.173.253.254-V24 K8S_VM 10.173.253.254 00:16:3e:05:73:08 NORMAL default v6.6.5 11188
6 10.173.244.238-V19 K8S_VM 10.173.244.238 00:16:3e:03:db:62 NORMAL default v6.6.5 11188
7 10.173.232.194-V12 K8S_VM 10.173.232.194 00:16:3e:04:5c:a8 NORMAL default v6.6.5 11188
8 10.173.254.3-V1 K8S_VM 10.173.254.3 00:16:3e:23:ff:62 NORMAL default v6.6.5 11188
9 10.173.250.231-V26 K8S_VM 10.173.250.231 00:16:3e:0b:8a:8b NORMAL default v6.6.5 11188
10 10.173.251.166-V14 K8S_VM 10.173.251.166 00:16:3e:0f:5a:c1 NORMAL default v6.6.5 11188
11 10.173.254.0-V2 K8S_VM 10.173.254.0 00:16:3e:0b:e8:75 NORMAL default v6.6.5 11188
12 10.173.244.239-V33 K8S_VM 10.173.244.239 00:16:3e:12:28:4d NORMAL default v6.6.5 11188
13 10.173.253.170-V32 K8S_VM 10.173.253.170 00:16:3e:13:72:12 NORMAL default v6.6.5 11188
14 10.173.246.202-V31 K8S_VM 10.173.246.202 00:16:3e:15:35:c3 NORMAL default v6.6.5 11188
15 10.173.244.229-V4 K8S_VM 10.173.244.229 00:16:3e:0f:fc:f1 NORMAL default v6.6.5 11188
16 10.173.253.255-V18 K8S_VM 10.173.253.255 00:16:3e:1e:e8:d6 NORMAL default v6.6.5 11188
17 10.173.254.4-V16 K8S_VM 10.173.254.4 00:16:3e:24:f6:85 NORMAL default v6.6.5 11188
18 10.173.254.2-V34 K8S_VM 10.173.254.2 00:16:3e:1a:95:92 NORMAL default v6.6.5 11188
19 10.173.231.4-V25 K8S_VM 10.173.231.4 00:16:3e:00:fa:98 NORMAL default v6.6.5 11188
20 10.173.252.131-V20 K8S_VM 10.173.252.131 00:16:3e:1f:36:21 NORMAL default v6.6.5 11188
21 10.173.254.1-V5 K8S_VM 10.173.254.1 00:16:3e:07:7f:76 NORMAL default v6.6.5 11188
22 10.173.254.6-V27 K8S_VM 10.173.254.6 00:16:3e:21:be:8c NORMAL default v6.6.5 11188
23 10.173.253.251-V11 K8S_VM 10.173.253.251 00:16:3e:26:6d:41 NORMAL default v6.6.5 11188
24 10.173.253.204-V3 K8S_VM 10.173.253.204 00:16:3e:21:e4:e3 NORMAL default v6.6.5 11188
25 10.173.251.13-V8 K8S_VM 10.173.251.13 00:16:3e:10:b0:93 NORMAL default v6.6.5 11188
26 10.173.244.240-V13 K8S_VM 10.173.244.240 00:16:3e:08:e6:9b NORMAL default v6.6.5 11188
27 10.173.253.205-V9 K8S_VM 10.173.253.205 00:16:3e:13:ff:5a NORMAL default v6.6.5 11188
28 10.173.238.97-V22 K8S_VM 10.173.238.97 00:16:3e:19:89:19 NORMAL default v6.6.5 11188
29 10.173.224.89-V28 K8S_VM 10.173.224.89 00:16:3e:00:53:a5 NORMAL default v6.6.5 11188
30 10.174.145.50-V21 K8S_VM 10.174.145.50 00:16:3e:04:90:45 NORMAL default v6.6.5 11188
31 10.173.231.5-V29 K8S_VM 10.173.231.5 00:16:3e:21:b0:0b NORMAL default v6.6.5 11188
32 10.173.224.88-V30 K8S_VM 10.173.224.88 00:16:3e:03:36:bd NORMAL default v6.6.5 11188
33 10.173.251.12-V17 K8S_VM 10.173.251.12 00:16:3e:1e:82:26 NORMAL default v6.6.5 11188
34 10.173.254.7-V15 K8S_VM 10.173.254.7 00:16:3e:09:9a:ab NORMAL default v6.6.5 11188

Kubernetes CNI

No response

Operation-System/Kernel version

awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release

"Alibaba Cloud Linux (Aliyun Linux) 2.1903 LTS (Hunting Beagle)"

uname -r

4.19.91-26.al7.x86_64

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[1473371932]

你好，能否大概说一下这个请求的链路是什么样的，例如：

client --> ingress --> server，以及在业务中透传的 trace id，在 header 中，叫什么呀

还有一点是，看起来网卡数据也没有，你们 deepflow-agent 主机上的网卡名称是什么呀？

[gbling]

链路是这样的，client --> slb --> eff-open-ops-test --> eff-noahbe-test --> j12-test
业务中透传的 trace id 在header 中名字是 tq-trace-id
主机网卡是 eth*的名字

[1473371932]

1. 网卡匹配倒是没什么问题
2. 你们的网关有没有做 nat，开源版不支持 nat，展示的效果就会从 nat 前后断开，如果没有做 nat 的话，需要在网关处开启 X-Requesr_ID，才能关联上网关前后的请求
3. 全局 id 需要通过 agent-group-config 配置让 deepflow 识别后才能解析，在此处把你们的 tq-trace-id 填上来就可以了：
   https://github.com/deepflowio/deepflow/blob/main/server/agent_config/example.yaml#L299