Performance issues, comparison between vector and filebeat

[StefanSa]

Hi there,
i am currently migrating some filebeat modules to vector.
I have noticed that vector causes more than 100% system load than filebeat with the same module, filebeat about 15% and vector 190%.
I would like to understand why this is so.
What can be the cause, too many remap or where is the bottleneck here ?
Is there a debug mode similar to vector tap where you can see what system load each module of vector is causing ?

systemload:

vector top:

Edit:

Ok, i made a mistake and was a bit too quick with the question here. Because the ingester nodes ultimately do the work and not Filebeat (pipelines). Nevertheless, 190% system load seems a bit too high for two firewalls that deliver data via syslog.

[StefanSa]

Can anyone tell if 190 to 400% is normal CPU load in this case ?

[jszwedko]

Hi @StefanSa ! If I remember correctly, your configuration has a large number of transforms, each of which will execute as one or more asynchronous tasks. It's hard to say if the CPU usage you see is normal or not since it can vary drastically based on the configuration. For example, using a lot of regexes will result in higher CPU utilization.

How many CPUs do you have (nproc)? Vector will use one thread per available CPU, by default, but you can restrict this by setting VECTOR_THREADS or using --threads on the CLI. Having Vector use less threads may result in lower CPU usage.

[StefanSa]

Jesse, thanks for your quick reply.
It has 32 CPU cores.
Is there a chance to understand via internal metric which units are causing the most CPU load here?

[jszwedko]

Hi @StefanSa . Unfortunately, not currently. We don't record the sort of metrics that would be required for that.

You can look at the utilization metric to understand where any bottlenecks are in your configuration, though. For each flow in your Vector config, look for the first component that has less than 1.0 utilization.

[StefanSa]

@jszwedko Jesse,
I could achieve an improvement of 15% - 20% by replacing regexes with includes.

[jszwedko]

Thanks for the report! Regexes are definitely one of the more CPU intensive operations you can do in Vector. includes is preferable when possible.

[StefanSa]

@jszwedko
Jesse thanks for your support, i think the question is answered.