Support wildcard pattern in kafka source

[msardara]

Is your feature request related to a problem? Please describe.
I am trying to create a data prepper pipeline that consumes data from kafka and ingest it into opensearch.
I need to consume from a set of topics whose name is generated dynamically over time, so I don't know it in
advance. I only know that the pattern is always "my-topic-*".

Describe the solution you'd like
I saw that KafkaConsumer supports the subscription to a wildcard topic using patterns,
but it seems data prepper only uses a list of fixed topics.

Is it possible to add the possibility to also consume wildcard topics?

Describe alternatives you've considered (Optional)
I saw that logstash supports this with the topics_pattern key.

[msardara]

Hey guys,

do you have any idea of when this feature will be available?