This community seeks to provide:
- Production-worthy Kafka setup for persistent (domain- and ops-) data at small scale.
- Operational knowledge, biased towards resilience over throughput, as Kubernetes manifest.
- A platform for event-driven (streaming!) microservices design using Kubernetes.
To quote @arthurk:
thanks for creating and maintaining this Kubernetes files, they're up-to-date (unlike the kubernetes contrib files, don't require helm and work great!
We suggest you apply -f manifests in the following order:
- Your choice of storage classes from ./configure
- namespace
- ./rbac-namespace-default
- ./zookeeper
- ./kafka
That'll give you client "bootstrap" bootstrap.kafka.svc.cluster.local:9092.
Our only dependency is kubectl. Not because we dislike Helm or Operators, but because we think plain manifests make it easier to collaborate.
If you begin to rely on this kafka setup we recommend you fork, for example to edit broker config.
With the introduction of app customization in kubectl 1.14 there's an alternative to forks. We as a community can maintain a set of overlays.
See the variants folder for different overlays. For example to scale to 1 kafka broker try kubectl apply -k variants/scale-1/.
Currently apply -k replaces apply -f ./zookeeper; apply -f ./kafka.
The original commands now result in error: unable to decode "zookeeper/kustomization.yaml": Object 'Kind' is missing in ...
and though they still seem to work you can get around that with a v1.14+ kubectl using: kubectl apply -k variants/as-is/.
kubectl apply -k takes a siungle overlay, meaning that you can't compose different overlays from this repo.
You'll probably want to maintain your own variant.
One option is to keep kubernets-kafka as a git submodule and edit the relative path from an example variant.
| tag | k8s ≥ | highlights |
|---|---|---|
| v5.1.0 | 1.11+ | Kafka 2.1.1 |
| v5.0.3 | 1.11+ | Zookeeper fix #227 + maxClientCnxns=1 |
| v5.0 | 1.11+ | Destabilize because in Docker we want Java 11 #197 #191 |
| v4.3.1 | 1.9+ | Critical Zookeeper persistence fix #228 |
| v4.3 | 1.9+ | Adds a proper shutdown hook #207 |
| v4.2 | 1.9+ | Kafka 1.0.2 and tools upgrade |
| ... see releases for full history ... | ||
| v1.0 | 1 | Stateful? In Kubernetes? In 2016? Yes. |
Have a look at:
- ./prometheus
- ./linkedin-burrow
- or plain JMX
- what's happening in the monitoring label.
- Note that this repo is intentionally light on automation. We think every SRE team must build the operational knowledge first.
Available for: