forked from elastic/logstash
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Logstash Getting started with Kubernetes Guide (elastic#14655)
This PR presents a getting started guide to working with Logstash in Kubernetes, and comes in three parts: - A walkthrough of a sample scenario, setting up a Kubernetes cluster with - Elasticsearch - Kibana - Filebeat to read Kubernetes API server logs and sent to Elasticsearch via Logstash - Logstash - Metricbeat to monitor Logstash and walks through the setup of these files - An annotated guide to these files - Explanations of how Logstash configuration and settings files map to the Kubernetes environment Relates elastic#14576 Closes elastic#14576 Co-authored-by: Karen Metts <[email protected]>
- Loading branch information
Showing
5 changed files
with
809 additions
and
78 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
307 changes: 307 additions & 0 deletions
307
docsk8s/quick-start/ls-k8s-configuration-files.asciidoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,307 @@ | ||
| [[ls-k8s-configuration-files]] | ||
| === Logstash configuration files in Kubernetes | ||
|
|
||
| WARNING: This documentation is still in development. This feature may be changed or removed in a future release. | ||
|
|
||
| This guide walks you through configuring {ls} and setting up {ls} pipelines in {k8s}. | ||
|
|
||
| * <<qs-pipeline-configuration>> | ||
| * <<qs-logstash-yaml>> | ||
| * <<qs-jvm-options>> | ||
| * <<qs-logging>> | ||
|
|
||
| {ls} uses two types of configuration files: | ||
|
|
||
| * _pipeline configuration files_, which define the Logstash processing pipeline | ||
| * _settings files_ which specify options that control {ls} startup and execution. | ||
| {logstash-ref}/config-setting-files.html[{ls} configuration files] topic contains information on these files. | ||
| This guide explains how these map to a {k8s} configuration. | ||
|
|
||
| [discrete] | ||
| [[qs-pipeline-configuration]] | ||
| === Pipeline configuration | ||
|
|
||
| This section explains how to configure single and multiple pipeline {ls} configurations. | ||
| Note that this section does not cover using {logstash-ref}/logstash-centralized-pipeline-management.html[Centralized Pipeline Management]. | ||
|
|
||
| Each of these configurations requires creating one or more `ConfigMap` definitions to define the pipeline, creating a volume to be made available to the Logstash container, and then mounting the definition in these volumes | ||
|
|
||
| [discrete] | ||
| [[qs-single-pipeline-config]] | ||
| ==== Single pipeline | ||
|
|
||
| The {ls} {logstash-ref}/docker.html[existing docker image] contains a default `pipeline.yml`, which expects a single pipeline, with the definition of that pipeline present in `/usr/share/logstash/pipeline`, as either a single file or collection of files, typically defined as a `ConfigMap` or series of `ConfigMaps` - note that | ||
| a single Kubernetes `ConfigMap` has a size limit of 1MB. | ||
|
|
||
|
|
||
| This example contains a simple pipeline definition, with the inputs and outputs split into separate configuration files: | ||
|
|
||
|
|
||
| [source,yaml] | ||
| -- | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: logstash-pipeline <1> | ||
| labels: | ||
| app: logstash-demo | ||
| data: | ||
| logstash-input.conf: | <2> | ||
| input { | ||
| beats { | ||
| port => "5044" | ||
| } | ||
| } | ||
| logstash-output.conf: | <3> | ||
| output { | ||
| elasticsearch { | ||
| hosts => ["https://demo-es-http:9200"] | ||
| } | ||
| } | ||
| -- | ||
|
|
||
| <1> Name of `ConfigMap` to be referenced in `Deployment`. | ||
| <2> Creates a `ConfigMap` representing the inputs for a pipeline. | ||
| <3> Creates a `CongigMap` representing the outputs for a pipeline. | ||
|
|
||
| Next, define your `Volume` in your `Deployment` template: | ||
|
|
||
| [source,yaml] | ||
| -- | ||
| volumes: | ||
| - name: logstash-pipeline | ||
| configMap: | ||
| name: logstash-pipeline | ||
| -- | ||
|
|
||
| and mount the volume in your container: | ||
|
|
||
| [source,yaml] | ||
| -- | ||
| volumeMounts: | ||
| - name: logstash-pipeline | ||
| mountPath: /usr/share/logstash/pipeline | ||
| -- | ||
|
|
||
|
|
||
| [float] | ||
| [[qs-multiple-pipeline-config]] | ||
| ==== Multiple pipelines | ||
|
|
||
| {ls} uses the `pipelines.yml` file to define {logstash-ref}/multiple-pipelines.html[multiple pipelines]. | ||
| {ls} in {k8s} requires a `ConfigMap` to represent the content that would otherwise be in `pipelines.yml`. | ||
| You can create pipeline configurations inline, or in separate `configMap` files or folders. | ||
|
|
||
|
|
||
| *Example: Pipelines.yml `ConfigMap` with an inline pipeline definition* | ||
|
|
||
| [source,yaml] | ||
| -- | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: logstash-pipeline-yaml <1> | ||
| labels: | ||
| app: logstash-demo | ||
| data: | ||
| pipelines.yml: | <2> | ||
| - pipeline.id: test <3> | ||
| pipeline.workers: 1 | ||
| pipeline.batch.size: 1 | ||
| config.string: "input { generator {} } filter { sleep { time => 1 } } output { stdout { codec => dots } }" | ||
| - pipeline.id: pipeline2 <4> | ||
| pipeline.workers: 8 | ||
| path.config: "/usr/share/logstash/pipeline2" | ||
| -- | ||
| <1> Name of `ConfigMap` to be referenced in `Deployment`. | ||
| <2> Defines a `pipelines.yml` `ConfigMap`. | ||
| <3> Defines a pipeline inside the `pipelines.yml`. | ||
| <4> Defines a pipeline, and a location where the pipeline definitions are stored. See below for these pipeline definitions. | ||
|
|
||
| *Example: Pipelines defined in separate files* | ||
|
|
||
|
|
||
| [source,yaml] | ||
| -- | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: pipeline2 | ||
| labels: | ||
| app: logstash-demo | ||
| data: | ||
| logstash-input.conf: | | ||
| input { | ||
| beats { | ||
| port => "5044" | ||
| } | ||
| } | ||
| logstash-output.conf: | | ||
| output { | ||
| elasticsearch { | ||
| hosts => ["https://demo-es-http:9200"] | ||
| index => "kube-apiserver-%{+YYYY.MM.dd}" | ||
| cacert => "/usr/share/logstash/config/es_ca.crt" | ||
| user => 'elastic' | ||
| password => '${ELASTICSEARCH_PASSWORD}' | ||
| } | ||
| } | ||
| -- | ||
|
|
||
| [float] | ||
| [[expose-pipelines]] | ||
| ===== Make pipelines available to Logstash | ||
|
|
||
| Create the volume(s) in your `Deployment`/`StatefulSet` | ||
|
|
||
| [source,yaml] | ||
| -- | ||
| volumes: | ||
| - name: logstash-pipelines-yaml | ||
| configMap: | ||
| name: logstash-pipelines-yaml | ||
| - name: pipeline2 | ||
| configMap: | ||
| name: pipeline2 | ||
| -- | ||
|
|
||
| and mount the volume(s) in your container spec | ||
|
|
||
| [source,yaml] | ||
| -- | ||
| # | ||
| volumeMounts: | ||
| - name: pipeline2 | ||
| mountPath: /usr/share/logstash/pipeline2 | ||
| - name: logstash-pipelines-yaml | ||
| mountPath: /usr/share/logstash/config/pipelines.yml | ||
| subPath: pipelines.yml | ||
|
|
||
| -- | ||
|
|
||
| [float] | ||
| [[qs-settings]] | ||
| ==== Settings configuration | ||
|
|
||
| [float] | ||
| [[qs-logstash-yaml]] | ||
| ===== The logstash.yml file | ||
|
|
||
| Unless you specify a configuration file, default values for the {logstash-ref}/logstash-settings-file.html[logstash.yml file] are used. | ||
| To override the default values, create a `ConfigMap` with the settings that you want to override: | ||
|
|
||
| [source,yaml] | ||
| -- | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: logstash-config | ||
| labels: | ||
| app: logstash-demo | ||
| data: | ||
| logstash.yml: | | ||
| api.http.host: "0.0.0.0" | ||
| log.level: info | ||
| pipeline.workers: 2 | ||
| -- | ||
|
|
||
| In your `Deployment`/`StatefulSet`, create the `Volume`: | ||
|
|
||
| [source,yaml] | ||
| -- | ||
| volumes: | ||
| - name: logstash-config | ||
| configMap: | ||
| name: logstash-config | ||
| -- | ||
|
|
||
| Create the `volumeMount` in the `container`: | ||
|
|
||
| [source,yaml] | ||
| -- | ||
| volumeMounts: | ||
| - name: logstash-config | ||
| mountPath: /usr/share/logstash/config/logstash.yml | ||
| subPath: logstash.yml | ||
| -- | ||
|
|
||
|
|
||
| [float] | ||
| [[qs-jvm-options]] | ||
| ==== JVM options | ||
|
|
||
| JVM settings are best set using environment variables to override the default settings in `jvm.options`. | ||
| This approach ensures that the expected settings from `jvm.options` are set, and only those options that explicitly need to be overridden are. | ||
|
|
||
| The JVM settings should be added in the `LS_JAVA_OPTS` environment variable in the container definition of your `Deployment`/`StatefulSet`: | ||
|
|
||
| [source,yaml] | ||
| -- | ||
| spec: | ||
| containers: | ||
| - name: logstash | ||
| env: | ||
| - name: LS_JAVA_OPTS | ||
| value: "-Xmx2g -Xms2g" | ||
| -- | ||
|
|
||
| [float] | ||
| [[qs-logging]] | ||
| ==== Logging configuration | ||
|
|
||
| By default, we use the `log4j2.properties` from the logstash docker image, that will log to `stdout` only. To change the log level, to use debug logging, use the `log.level` option in <<qs-logstash-yaml, logstash.yml>> | ||
|
|
||
| NOTE: You can apply temporary logging changes using the {logstash-ref}/logging.html#_logging_apis[Logging APIs]. | ||
| If you require broader changes that persist across container restarts, you need to create a *full* and correct `log4j2.properties` file, and ensure that it is visible to the {ls} container. | ||
|
|
||
| This example uses a `configMap` and the base `log4j2.properties` file from the Docker container, adding debug logging for elasticsearch output plugins: | ||
|
|
||
| [source,yaml] | ||
| -- | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: logstash-log4j | ||
| labels: | ||
| app: logstash-demo | ||
| data: | ||
| log4j2.properties: | | ||
| status = error | ||
| name = LogstashPropertiesConfig | ||
|
|
||
| appender.console.type = Console | ||
| appender.console.name = plain_console | ||
| appender.console.layout.type = PatternLayout | ||
| appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]}%notEmpty{[%X{plugin.id}]} %m%n | ||
|
|
||
| appender.json_console.type = Console | ||
| appender.json_console.name = json_console | ||
| appender.json_console.layout.type = JSONLayout | ||
| appender.json_console.layout.compact = true | ||
| appender.json_console.layout.eventEol = true | ||
|
|
||
| rootLogger.level = ${sys:ls.log.level} | ||
| rootLogger.appenderRef.console.ref = ${sys:ls.log.format}_console | ||
| logger.elasticsearchoutput.name = logstash.outputs.elasticsearch | ||
| logger.elasticsearchoutput.level = debug | ||
| -- | ||
|
|
||
| In your `Deployment`/`StatefulSet`, create the `Volume`: | ||
|
|
||
| [source,yaml] | ||
| -- | ||
| volumes: | ||
| - name: logstash-log4j | ||
| configMap: | ||
| name: logstash-log4j | ||
| -- | ||
|
|
||
| Create the `volumeMount` in the `container`: | ||
|
|
||
| [source,yaml] | ||
| -- | ||
| volumeMounts: | ||
| - name: logstash-log4j | ||
| mountPath: /usr/share/logstash/config/log4j.properties | ||
| subPath: log4j.properties | ||
| -- |
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.