[BUG] http_log_x_request_id 添加自定义Header数据采集不全问题

[gbling]

Search before asking

• I had searched in the issues and found no similar feature requirement.

DeepFlow Component

Agent

What you expected to happen

agent配置http_log_x_request_id添加自定义采集Header：tq-trace-id，抓包数据中能看到这个参数，但是CK查询时找不到这条记录

• agent配置信息

vtap_group_id: g-3476caa59b
max_collect_pps: 200
max_npb_bps: 1000
max_cpus: 1
max_millicpus: 1000
max_memory: 768
sync_interval: 60
platform_sync_interval: 10
rsyslog_enabled: 1
system_load_circuit_breaker_threshold: 1
system_load_circuit_breaker_recover: 0.9
system_load_circuit_breaker_metric: load15
max_tx_bandwidth: 0
bandwidth_probe_interval: 10
tap_interface_regex: ^(tap.*|cali.*|veth.*|eth.*|en[osipx].*|lxc.*|lo|[0-9a-f]+_h)$
max_escape_seconds: 3600
mtu: 1500
output_vlan: 0
collector_socket_type: TCP
compressor_socket_type: TCP
npb_socket_type: RAW_UDP
npb_vlan_mode: 0
collector_enabled: 1
vtap_flow_1s_enabled: 1
l4_log_tap_types:
- 0
npb_dedup_enabled: 1
platform_enabled: 0
if_mac_source: 0
vm_xml_path: /etc/libvirt/qemu/
nat_ip_enabled: 0
capture_packet_size: 65535
inactive_server_port_enabled: 1
inactive_ip_enabled: 1
log_threshold: 300
log_level: INFO
log_retention: 300
http_log_proxy_client: X-Forwarded-For
http_log_trace_id: traceparent, sw8
l7_log_packet_size: 1024
l4_log_collect_nps_threshold: 10000
l7_log_collect_nps_threshold: 10000
l7_metrics_enabled: 1
l7_log_store_tap_types:
- 0
capture_socket_type: 0
tap_mode: 0
thread_threshold: 500
process_threshold: 10
ntp_enabled: 0
l4_performance_enabled: 1
pod_cluster_internal_ip: 0
domains:
- "0"
decap_type:
- 1
- 2
http_log_span_id: traceparent, sw8
sys_free_memory_limit: 0
log_file_size: 1000
http_log_x_request_id: tq-trace-id, Tq-Trace-Id, TQ-TRACE-ID, Tq-Trace-ID
external_agent_http_proxy_enabled: 1
external_agent_http_proxy_port: 38086
analyzer_port: 32700
proxy_controller_port: 32701
wasm_plugins:
- wasm
static_config:
  profiler: false
  afpacket-blocks-enabled: false
  afpacket-blocks: 128
  analyzer-raw-packet-block-size: 65536
  batched-buffer-size-limit: 131072
  enable-debug-stats: false
  analyzer-dedup-disabled: false
  default-tap-type: 3
  debug-listen-port: 0
  enable-qos-bypass: false
  multiple-sockets-to-ingester: false
  fast-path-map-size: 0
  first-path-level: 8
  local-dispatcher-count: 1
  packet-fanout-mode: 0
  src-interfaces:
  - dummy0
  - dummy1
  tap-interface-bond-groups:
  - {}
  cloud-gateway-traffic: false
  mirror-traffic-pcp: 0
  pcap:
    queue-size: 65536
    flow-buffer-size: 65536
    buffer-size: 98304
    flush-interval: 1m
  flow:
    established-timeout: 300s
    closing-rst-timeout: 35s
    others-timeout: 5s
    opening-rst-timeout: 1s
    flow-slots-size: 131072
    flow-count-limit: 65535
    flush-interval: 1s
    flow-aggr-queue-size: 65535
    memory-pool-size: 65536
    ignore-tor-mac: false
    ignore-l2-end: false
    ignore-idc-vlan: false
  flow-queue-size: 65536
  quadruple-queue-size: 262144
  analyzer-queue-size: 131072
  dpdk-enabled: false
  libpcap-enabled: false
  xflow-collector: {}
  npb-port: 4789
  vxlan-flags: 255
  ignore-overlay-vlan: false
  collector-sender-queue-size: 65536
  collector-sender-queue-count: 1
  toa-sender-queue-size: 65536
  toa-lru-cache-size: 65536
  flow-sender-queue-size: 65536
  flow-sender-queue-count: 1
  second-flow-extra-delay-second: 0s
  packet-delay: 1s
  kubernetes-poller-type: adaptive
  kubernetes-api-list-limit: 1000
  kubernetes-api-list-interval: 10m
  ingress-flavour: kubernetes
  grpc-buffer-size: 5
  l7-log-session-aggr-timeout: 120s
  l7-log-session-slot-capacity: 1024
  tap-mac-script: ""
  bpf-disabled: false
  l7-protocol-inference-max-fail-count: 5
  l7-protocol-inference-ttl: 60
  oracle-parse-config:
    is-be: true
    int-compress: true
    resp-0x04-extra-byte: false
  packet-sequence-block-size: 256
  packet-sequence-queue-size: 65536
  packet-sequence-queue-count: 1
  packet-sequence-flag: 0
  l7-protocol-enabled:
  - HTTP
  - HTTP2
  - SomeIP
  - MySQL
  - Redis
  - Kafka
  - Pulsar
  - DNS
  - TLS
  - Custom
  standalone-data-file-size: 200
  standalone-data-file-dir: /var/log/deepflow-agent/
  log-file: /var/log/deepflow-agent/deepflow-agent.log
  external-agent-http-proxy-compressed: false
  external-agent-http-proxy-profile-compressed: true
  l7-protocol-ports:
    AMQP: 1-65535
    Custom: 1-65535
    DNS: 53,5353
    Dubbo: 1-65535
    FastCGI: 1-65535
    HTTP: 1-65535
    HTTP2: 1-65535
    Kafka: 1-65535
    MQTT: 1-65535
    Memcached: "11211"
    MongoDB: 1-65535
    MySQL: 1-65535
    NATS: 1-65535
    OpenWire: 1-65535
    Oracle: "1521"
    PostgreSQL: 1-65535
    Pulsar: 1-65535
    Redis: 1-65535
    SofaRPC: 1-65535
    SomeIP: 1-65535
    TLS: 443,6443
    Tars: 1-65535
    ZMTP: 1-65535
    bRPC: 1-65535
  l7-log-blacklist:
    AMQP: []
    DNS: []
    Dubbo: []
    FastCGI: []
    HTTP: []
    HTTP2: []
    Kafka: []
    MQTT: []
    MongoDB: []
    MySQL: []
    NATS: []
    OpenWire: []
    Oracle: []
    PostgreSQL: []
    Pulsar: []
    Redis: []
    SOFARPC: []
    SomeIP: []
    TLS: []
    Tars: []
    ZMTP: []
    bRPC: []
    gRPC: []
  l7-protocol-advanced-features:
    http-endpoint-extraction:
      disabled: false
      match-rules:
      - prefix: ""
        keep-segments: 2
    obfuscate-enabled-protocols:
    - Redis
    extra-log-fields: {}
  consistent-timestamp-in-l7-metrics: false
  ebpf:
    disabled: false
    global-ebpf-pps-threshold: 0
    uprobe-process-name-regexs:
      golang-symbol: ""
      golang: ""
      openssl: ""
    kprobe-whitelist: {}
    kprobe-blacklist: {}
    thread-num: 1
    perf-pages-count: 128
    ring-size: 65536
    max-socket-entries: 131072
    max-trace-entries: 131072
    socket-map-max-reclaim: 120000
    go-tracing-timeout: 120
    io-event-collect-mode: 1
    io-event-minimal-duration: 1ms
    dwarf-disabled: true
    dwarf-regex: ""
    dwarf-process-map-size: 1024
    dwarf-shard-map-size: 128
    java-symbol-file-refresh-defer-interval: 60s
    on-cpu-profile:
      disabled: false
      frequency: 99
      cpu: 0
      regex: ^deepflow-.*
    off-cpu-profile:
      disabled: false
      regex: ^deepflow-.*
      cpu: 0
      minblock: 50us
    memory-profile:
      disabled: true
      regex: ^java
      report-interval: 10s
    preprocess:
      stack-compression: true
    syscall-out-of-order-cache-size: 16
    syscall-trace-id-disabled: false
  os-app-tag-exec-user: deepflow
  os-app-tag-exec:
  - cat
  - /tmp/tag.yaml
  os-proc-root: /proc
  os-proc-socket-sync-interval: 10
  os-proc-socket-min-lifetime: 3
  os-proc-regex:
  - {}
  os-proc-sync-enabled: false
  os-proc-sync-tagged-only: false
  guard-interval: 10s
  check-core-file-disabled: false
  memory-trim-disabled: false
  fast-path-disabled: false
  forward-capacity: 16384
  rrt-tcp-timeout: 1800s
  rrt-udp-timeout: 150s
  prometheus-extra-config:
    enabled: false
    labels:
    - label1
    - label2
    labels-limit: 1024
    values-limit: 4096
  process-scheduling-priority: 0
  external-profile-integration-disabled: false
  external-trace-integration-disabled: false
  external-metric-integration-disabled: false
  external-log-integration-disabled: false
  ntp-max-interval: 300s
  ntp-min-interval: 10s
  dispatcher-queue: false
  ebpf-collector-queue-size: 65535

• 抓包数据
  

• CK查询数据
  

How to reproduce

No response

DeepFlow version

# kubectl exec -it -n deepflow deploy/deepflow-server -- deepflow-server -v
Name: deepflow-server community edition
Branch: v6.6.5
CommitID: eb43fab700b5fa0a03764f0146f4c8ef4724cd1f
RevCount: 11188
Compiler: go version go1.21.13 linux/amd64
CompileTime: 2024-09-25 06:14:29

# kubectl exec -it -n deepflow ds/deepflow-agent -- deepflow-agent -v
Defaulted container "deepflow-agent" out of: deepflow-agent, configure-sysctl (init)
11188-eb43fab700b5fa0a03764f0146f4c8ef4724cd1f
Name: deepflow-agent community edition
Branch: v6.6.5
CommitId: eb43fab700b5fa0a03764f0146f4c8ef4724cd1f
RevCount: 11188
Compiler: rustc 1.77.1 (7cf61ebde 2024-03-27)
CompileTime: 2024-09-25 06:15:33

DeepFlow agent list

# deepflow-ctl agent list  --api-port 32702 
ID   NAME                                    TYPE     CTRL_IP          CTRL_MAC            STATE    GROUP     EXCEPTIONS   REVISION       UPGRADE_REVISION   
1    cn-zhangjiakou.10.173.253.171-V6        K8S_VM   10.173.253.171   00:16:3e:09:f0:09   NORMAL   default                v6.6.5 11188                      
2    cn-zhangjiakou.10.173.253.172-V7        K8S_VM   10.173.253.172   00:16:3e:14:a9:93   NORMAL   default                v6.6.5 11188                      
3    cn-zhangjiakou.10.173.253.173-V10       K8S_VM   10.173.253.173   00:16:3e:05:72:38   NORMAL   default                v6.6.5 11188                      
4    cn-zhangjiakou.10.173.252.128-V23       K8S_VM   10.173.252.128   00:16:3e:0f:3c:61   NORMAL   default                v6.6.5 11188                      
5    taqu-test10-10.173.253.254-V24          K8S_VM   10.173.253.254   00:16:3e:05:73:08   NORMAL   default                v6.6.5 11188                      
6    cn-zhangjiakou.10.173.244.238-V19       K8S_VM   10.173.244.238   00:16:3e:03:db:62   NORMAL   default                v6.6.5 11188                      
7    cn-zhangjiakou.10.173.232.194-V12       K8S_VM   10.173.232.194   00:16:3e:04:5c:a8   NORMAL   default                v6.6.5 11188                      
8    cn-zhangjiakou.10.173.254.3-V1          K8S_VM   10.173.254.3     00:16:3e:23:ff:62   NORMAL   default                v6.6.5 11188                      
9    cn-zhangjiakou.10.173.250.231-V26       K8S_VM   10.173.250.231   00:16:3e:0b:8a:8b   NORMAL   default                v6.6.5 11188                      
10   cn-zhangjiakou.10.173.251.166-V14       K8S_VM   10.173.251.166   00:16:3e:0f:5a:c1   NORMAL   default                v6.6.5 11188                      
11   cn-zhangjiakou.10.173.254.0-V2          K8S_VM   10.173.254.0     00:16:3e:0b:e8:75   NORMAL   default                v6.6.5 11188                      
12   cn-zhangjiakou.10.173.244.239-V33       K8S_VM   10.173.244.239   00:16:3e:12:28:4d   NORMAL   default                v6.6.5 11188                      
13   cn-zhangjiakou.10.173.253.170-V32       K8S_VM   10.173.253.170   00:16:3e:13:72:12   NORMAL   default                v6.6.5 11188                      
14   cn-zhangjiakou.10.173.246.202-V31       K8S_VM   10.173.246.202   00:16:3e:15:35:c3   NORMAL   default                v6.6.5 11188                      
15   cn-zhangjiakou.10.173.244.229-V4        K8S_VM   10.173.244.229   00:16:3e:0f:fc:f1   NORMAL   default                v6.6.5 11188                      
16   cn-zhangjiakou.10.173.253.255-V18       K8S_VM   10.173.253.255   00:16:3e:1e:e8:d6   NORMAL   default                v6.6.5 11188                      
17   taqu-test9-10.173.254.4-V16             K8S_VM   10.173.254.4     00:16:3e:24:f6:85   NORMAL   default                v6.6.5 11188                      
18   cn-zhangjiakou.10.173.254.2-V34         K8S_VM   10.173.254.2     00:16:3e:1a:95:92   NORMAL   default                v6.6.5 11188                      
19   cn-zhangjiakou.10.173.231.4-V25         K8S_VM   10.173.231.4     00:16:3e:00:fa:98   NORMAL   default                v6.6.5 11188                      
20   cn-zhangjiakou.10.173.252.131-V20       K8S_VM   10.173.252.131   00:16:3e:1f:36:21   NORMAL   default                v6.6.5 11188                      
21   cn-zhangjiakou.10.173.254.1-V5          K8S_VM   10.173.254.1     00:16:3e:07:7f:76   NORMAL   default                v6.6.5 11188                      
22   taqu-test-eff-gray-10.173.254.6-V27     K8S_VM   10.173.254.6     00:16:3e:21:be:8c   NORMAL   default                v6.6.5 11188                      
23   taqu-test9-10.173.253.251-V11           K8S_VM   10.173.253.251   00:16:3e:26:6d:41   NORMAL   default                v6.6.5 11188                      
24   cn-zhangjiakou.10.173.253.204-V3        K8S_VM   10.173.253.204   00:16:3e:21:e4:e3   NORMAL   default                v6.6.5 11188                      
25   cn-zhangjiakou.10.173.251.13-V8         K8S_VM   10.173.251.13    00:16:3e:10:b0:93   NORMAL   default                v6.6.5 11188                      
26   cn-zhangjiakou.10.173.244.240-V13       K8S_VM   10.173.244.240   00:16:3e:08:e6:9b   NORMAL   default                v6.6.5 11188                      
27   cn-zhangjiakou.10.173.253.205-V9        K8S_VM   10.173.253.205   00:16:3e:13:ff:5a   NORMAL   default                v6.6.5 11188                      
28   cn-zhangjiakou.10.173.238.97-V22        K8S_VM   10.173.238.97    00:16:3e:19:89:19   NORMAL   default                v6.6.5 11188                      
29   cn-zhangjiakou.10.173.224.89-V28        K8S_VM   10.173.224.89    00:16:3e:00:53:a5   NORMAL   default                v6.6.5 11188                      
30   cn-zhangjiakou.10.174.145.50-V21        K8S_VM   10.174.145.50    00:16:3e:04:90:45   NORMAL   default                v6.6.5 11188                      
31   cn-zhangjiakou.10.173.231.5-V29         K8S_VM   10.173.231.5     00:16:3e:21:b0:0b   NORMAL   default                v6.6.5 11188                      
32   cn-zhangjiakou.10.173.224.88-V30        K8S_VM   10.173.224.88    00:16:3e:03:36:bd   NORMAL   default                v6.6.5 11188                      
33   cn-zhangjiakou.10.173.251.12-V17        K8S_VM   10.173.251.12    00:16:3e:1e:82:26   NORMAL   default                v6.6.5 11188                      
34   taqu-test-eff-online-10.173.254.7-V15   K8S_VM   10.173.254.7     00:16:3e:09:9a:ab   NORMAL   default                v6.6.5 11188               

Kubernetes CNI

No response

Operation-System/Kernel version

# awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release
"Alibaba Cloud Linux (Aliyun Linux) 2.1903 LTS (Hunting Beagle)"

# uname -r
4.19.91-26.al7.x86_64

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Fancyki1]

可以看下已经提的没必要重复提了：
#8242
#8237

[1473371932]

目前 header 中 tq-trace-id 是你网关生成的，还是业务透传的全局 id 呀，

看起来并不像网关生成的，如果是在业务中透传的，需要在此处配置

[gbling]

@Fancyki1 多谢解答，是在业务中透传，已经添加对应配置了

[gbling]

可以看下已经提的没必要重复提了： #8242 #8237

---

和这边提的issue似乎不是同个问题，我们是在测试分布式链路追踪的时候遇到的情况。

全链路使用的都是http1.1，测试链路中 A、B、C 三个系统均添加Header（x-trace-id）在链路中进行传递，deepflow-agent 配置 http_log_x_request_id、http_log_trace_id 也都添加了x-trace-id，有发现请求同个接口多次，最终出来的结果是不同的，有的系统调用链路都不全，如图

第一次测试结果

第二次测试结果

第三次测试结果