dnsdist: add multi-stream dnstap sockets

[johnhtodd]

• Program: dnsdist
• Issue type: Feature request

Short description

Adding multiple streams of dnstap data emitted from dnsdist would allow better scaling for dnstap consumers.

Usecase

We're using Vector to consume dnstap streams from dnsdist. There apparently are bottlenecks with single-socket models of dnstap data transmission. This I'm sure would be exacerbated by slower latency on the LAN and larger traffic volumes - the ACK traffic will start to cause pileups with a single socket. Spreading the data load out across many dnstap sockets like pdns-rec does would make sense. There is discussion of this here: vectordotdev/vector#20744 (see comment from james-stevens)

Description

Having dnsdist open multiple simultaneous dnstap sockets to any named endpoint would be useful. The number of sockets could be configurable, or it could be based on threading, or dynamic based on number of messages - no opinion on that. How does pdns-rec do it?

[omoerbeek]

The recursor opens a dnstap or protobuf logging stream per thread. Each thread logs to its associated socket.

[rgacogne]

At the moment DNSdist creates one fstrm_writer per newFrameStreamUnixLogger or newFrameStreamTcpLogger object, so in theory it is already possible to create more than one connection, but it might be quite cumbersome to use them so.

[johnhtodd]

More notes on this: When we had >60kqps flowing through a single TCP session to Vector, I was seeing what I would describe as high fluctuations in bandwidth between the transmitting server and the Vector instance. Meaning: for a few milliseconds, there would be many (20? I don't recall the number) megabits of throughput, which would then drop for a few milliseconds to 5 megabits of throughput, and then jump back to 20. This was observed via packet dumps with tshark. Sorry that I don't have the exact figures here; I was hunting a different problem and that behavior was a "Huh... interesting." moment, but I did not document it. I suspect this is a behavior that is made worse by the single-socket model in use. The same packet dump looking at data coming from pdns-rec (with less than 60kqps, admittedly) was extremely smooth - there were no fluctuations in the packet throughput between Vector and the recursive resolver on the dnstap data stream. Is this the "fault" of dnsdist? Not necessarily, but the single-socket model may make things worse than they need to be.

[esensar]

I am starting to work on this and my initial thought was to just add an additional option and open multiple connections inside of FrameStreamLogger and keep track of them there, but maybe adding a separate RemoteLoggerInterface that just acts as a group of other loggers would make more sense?

Not sure how to name it, but idea would be to manually build a number of loggers (or maybe provide a function that would do it based on a provided number parameter, or by number of available threads) and then just select a different one for each message. This would open up the possibility to easily providing different strategies, to achieve different goals, like passing same messages to all the different loggers or implementing the idea from this issue by selecting a different logger for each new message.

Does an approach like this make sense @rgacogne @omoerbeek ? I wanted to check before I head into implementation.