[Meta] Logstash to Logstash [Bootstrap]

[yaauie]

User Needs:

• As a user with distributed Logstash needs, I want a supported, straight-forward, and stable way to send events from one Logstash pipeline to another Logstash pipeline that exists in either a separate process on the same host or in a different data-center, optionally secured with TLS/SSL.
• As a maintainer of Logstash-to-Logstash pipelines, I do not want to be concerned with how the events are emitted, transmitted, or encoded, so long as the upstream Logstash has a TCP/HTTP route to the downstream Logstash.

See also: #14477

Task: API Boundary Bootstrap

Introduce a Logstash "integration" plugin that contains an output plugin and an input plugin whose minimal configuration allow transmission of events from one Logstash pipeline to another using a host, port, and optional standard SSL settings.

For example, transmitting events over plaintext would be as simple as defining an output in the upstream pipeline targeting a host and port:

output {
  logstash {
    host => "ls.mycorpnet.internal"
    port => "9820"
  }
}

-- upstream pipeline on ls.mycorpnet.external

And an accompanying input in the downstream pipeline that binds to the same port:

input {
  logstash {
    port => "9820"
  }
}

-- downstream pipeline on ls.mycorpnet.internal

Details: Security

SSL/TLS communications must be configured using standardized SSL settings.

Details: Open-to-extend

The configuration of these plugins are intended to be an API boundary so that:

• pipeline authors can use them without needing to consider the implementation details of protocol and/or encoding
• plugin/logstash maintainers can evolve the implementation details without impacting in-the-wild configurations, including adding protocols and/or encoding mechanisms, along with negotiation to ensure the "best available" protocol and/or encoding is used when the sending and receiving Logstash pipelines have differing versions of the plugin.

In the initial release of this plugin set, batches of events should be transmitted using newline-delimited JSON over HTTP(s) using POST /events requests, but these are internal implementation details. These details are not exposed to the user via plugin configuration options.

The behavior of the initial release of a logstash input plugin is explicitly undefined when receiving bare TCP or TLS TCP connections, when receiving HTTP(s) requests with verbs other than POST or to paths other than /events, or with content-types other than application/x-ndjson.

Bootstrap Requirements

• an output supports a single host/port pair
• http(s) with standard-form SSL identity and trust configuration (with workable subset)
• events sent via http(s) using POST /events and a Content-Type: application/x-ndjson header
• events received have zero implicit enrichment from the input plugin

Initial Extensions

• Startup Safety: an output that cannot connect to a downstream input should prevent the pipeline from starting (possibly after some reasonable delay)
• Shutdown Safety: an output that is being shut-down and is unable to deliver an in-flight batch in a timely manner should NACK the batch so that pipelines with PQ can re-attempt.
• Load-balancing: an output should be configurable with hosts => ["12.34.56.78:1234", "12.34.56.79:1234"] or hosts => ["12.34.56.78", "12.34.56.79"] port => 1234 (alternatives to host => "12.34.56.78" port => 1234) and should distribute events to the downstream hosts (naive whole-batch round-robin is ok at first, but then we could make it better)
• SSL: fill gaps in identity and trust configuration.
• Metadata: add support for propagating @metadata

Implementation Plan

Bootstrap

Preview Give feedback

1. bootstrap: minimal impl, docs, packaging logstash-plugins/logstash-integration-logstash#2
   +0
2. HTTP LS-to-LS - Bootstrap part 2 - Complete Initial Working Plugin logstash-plugins/logstash-integration-logstash#6
   bug +1
   
3. HTTP LS-to-LS - Support SSL standardization logstash-plugins/logstash-integration-logstash#4
   enhancement +1
   
4. LS-to-LS - Output to replace use of host/port to use hosts instead logstash-plugins/logstash-integration-logstash#12
   bug +1
   
5. Doc: Add documentation for new integration (index, input, output) logstash-plugins/logstash-integration-logstash#1
   docs documentation +2
   
6. Doc: Update Logstash-to-Logstash communication "how to" docs #15170
   docs +1
   
7. Release the logstash-integration-logstash plugin
8. LS-to-LS: Bundle logstash-integration-logstash with core. #15335
   Team:Logstash packaging +2
   

Advanced Features

Preview Give feedback

1. LS-to-LS - Enable HTTP data compression logstash-plugins/logstash-integration-logstash#11
   enhancement +1
   
2. HTTP LS-to-LS - Support Multiple Outputs / High Availability / Loadbalancing logstash-plugins/logstash-integration-logstash#5
   enhancement +1
   
3. [Docs] Update LS-to-LS native doc after HA support #15498
   docs +1
   

Testing

Preview Give feedback

1. LS-to-LS - Add Integration Testing #15496
   0 of 2
   +0
   
2. https://github.com/elastic/ingest-dev/issues/2574

[karenzone]

Related docs issues:

• [Meta] Doc: Add/update docs for Logstash-to-Logstash communication #15169
• Coordinate a plan for adding new integration docs #15213

[mbudge]

Time to live TTL may be a useful output setting if logstash(s) is behind a load balancer, which terminates the connection every X minutes. When this happens it can cause duplicates in elastic when Logstash re-tries the terminated request.

[roaksoax]

Closing this issue as the work is now complete. If you would like to file enhancement request on the matter, please open new issues.