K8s Vector Agent w/ Sink Vector - What happens if buffer is full?

[svenmueller]

Hi,

We are using a centralized deployment topology as explained here in our Kubernetes clusters.

I would like to better understand what happens exactly on the Vector agent side (collecting container logs) when the sink type Vector is reaching the maximum allowed number of events in the buffer.

Default Settings:

• buffer.max_events: 500
• buffer.type: memory
• buffer.when_full: block

The documentation says:

Applies back pressure when the buffer is full. This prevents data loss, but will cause data to pile up on the edge.

Can someone please explain in detail what does this mean for the K8s Vector agent? Will it stop reading events from container log files completely?

Thx a lot,
Sven

[jszwedko]

Hi @svenmueller !

Good question. When the buffer is full, the sink, in this case the vector sink, will start applying back-pressure to the source. What is the source in this case? kubernetes_logs? If it is, that means that the source will slow down, or stop, reading the pod log files until the sink catches up.

Does this answer your question?

[roumigus]

Hi @jszwedko !

So if I understand correctly, the back pressure you mention (from the sink to the source) is entirely different and unrelated to the e2e acknowledgment feature, which is still mostly WIP. Is that correct ?

Last question: does this back-pressure mechanism for the vector sink work for "v1" and "v2" ? Currently, the default configuration for vector source and sink is v1.

[jszwedko]

Hi @svenmueller !

Back-pressure will be applied even in the absence of end-to-end acknowledgments, but the features do work together.

When end-to-end acknowledgements are disabled, the source will "ack" the data as soon as it receives it (for sources that support acking). For example, with the http source, Vector will respond to the request as soon as it has ingested the events for that request. Ingesting the events includes trying to push them to the next component. If the next component cannot receive them (because there is back-pressure) the source will block and hold the request open.

When end-to-end acknowledgements are enabled, the source will wait to "ack" the data until it has been persisted to a disk buffer or has been sent by the sink to the downstream target. This will increase the effects of back-pressure on the client sending to Vector.

End-to-end acknowledgements are still a work-in-progress, but we expect to announce them in 0.19.0 or 0.20.0.

The v1 version of the Vector source/sink protocol is just a raw TCP protocol. When there is back-pressure in the remote source, the sink will fill up any network buffers but then block. The v2 version is gRPC over HTTP/2. The back-pressure in this case will cause the sink to slow down sending requests to the remote source when adaptive concurrency is enabled. We plan to flip the default version to v1 in the next release and remove the v1 protocol in the release after that.

[roumigus]

Thanks a lot @jszwedko for those clear explanations. We really appreciate it. Have a great day 🙂