Error with kafka source when specifying password via sasl (Vector 0.43.0)

[hartfordfive]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I'm attempting to connect to a secured Kafka cluster with the appropriate username and password. From my understanding, this is done via the sasl Kafka source configuration.

When starting a vector instances (in Kubernetes) with a Kafka sink such as this one:

src_kafka:
  type: kafka
  boostrap_servers: kafka01:9092,kafka02:9092,kafka03:9092
  commit_interval_ms: 2000
  auto_offset_reset: latest
  session_timeout_ms: 14000
  group_id: vector
  topics:
    - logs-topic-1
    - logs-topic-2
    - logs-topic-3
  sasl:
    enabled: true
    username: ${KAFKA_USER:?err}
    password: ${KAFKA_PASS:?err}
  librdkafka_options:
    heartbeat.interval.ms: "3500"
    fetch.max.bytes: "2097152"
    max.partition.fetch.bytes: "1048576" 
    queued.max.message.kbytes: "2048"

I observe the following error message:

ERROR vector::topology::builder: Configuration error.  error=source "src_kafka": Could not create kafka consumer: Client creation error: Invalid sasl.kerberos.kinit.cmd value: Property not available: "sasl.kerberos.keytab"

I don't see anything in the current Vector kafka source documentation (on 2025-01-14) about this sasl.kerberos parameter although I do see it in the librdkafka client configuration. I assume the librdkafka client configuration is being automatically built with the default value? What can I do to resolve this issue as I'm currently blocked and can't consume form this cluster.

I appreciate the help.

Configuration

No response

Version

0.43.0-distroless-libc

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[pront]

Hi @hartfordfive, these are the available options currently: https://vector.dev/docs/reference/configuration/sources/kafka/#sasl.

If you are motivated enough, you can add support for sasl.kerberos:

vector/src/kafka.rs

Lines 43 to 84 in ca73a06

	/// Kafka authentication configuration.
	#[configurable_component]
	#[derive(Clone, Debug, Default)]
	pub struct KafkaAuthConfig {
	#[configurable(derived)]
	pub(crate) sasl: Option<KafkaSaslConfig>,
	#[configurable(derived)]
	#[configurable(metadata(docs::advanced))]
	pub(crate) tls: Option<TlsEnableableConfig>,
	}
	/// Configuration for SASL authentication when interacting with Kafka.
	#[configurable_component]
	#[derive(Clone, Debug, Default)]
	pub struct KafkaSaslConfig {
	/// Enables SASL authentication.
	///
	/// Only `PLAIN`- and `SCRAM`-based mechanisms are supported when configuring SASL authentication using `sasl.*`. For
	/// other mechanisms, `librdkafka_options.*` must be used directly to configure other `librdkafka`-specific values.
	/// If using `sasl.kerberos.*` as an example, where `*` is `service.name`, `principal`, `kinit.md`, etc., then
	/// `librdkafka_options.*` as a result becomes `librdkafka_options.sasl.kerberos.service.name`,
	/// `librdkafka_options.sasl.kerberos.principal`, etc.
	///
	/// See the [librdkafka documentation](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) for details.
	///
	/// SASL authentication is not supported on Windows.
	pub(crate) enabled: Option<bool>,
	/// The SASL username.
	#[configurable(metadata(docs::examples = "username"))]
	pub(crate) username: Option<String>,
	/// The SASL password.
	#[configurable(metadata(docs::examples = "password"))]
	pub(crate) password: Option<SensitiveString>,
	/// The SASL mechanism to use.
	#[configurable(metadata(docs::examples = "SCRAM-SHA-256"))]
	#[configurable(metadata(docs::examples = "SCRAM-SHA-512"))]
	pub(crate) mechanism: Option<String>,
	}