Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update(Kafka integration): Update Memory Overload Protection config #2806

Open
wants to merge 1 commit into
base: release-5.8
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion en_US/data-integration/data-bridge-kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -400,7 +400,7 @@ This section describes some advanced configuration options that can optimize the
| Buffer Mode (Sink) | Defines whether messages are stored in a buffer before being sent. Memory buffering can increase transmission speeds.<br />`memory`: Messages are buffered in memory. They will be lost in the event of an EMQX node restart.<br />`disk`: Messages are buffered on disk, ensuring they can survive an EMQX node restart.<br />`hybrid`: Messages are initially buffered in memory. When they reach a certain limit (refer to the `segment_bytes` configuration for more details), they are gradually offloaded to disk. Similar to the memory mode, messages will be lost if the EMQX node restarts. | `memory` |
| Per-partition Buffer Limit (Sink) | Maximum allowed buffer size, in bytes, for each Kafka partition. When this limit is reached, older messages will be discarded to make room for new ones by reclaiming buffer space. <br />This option helps to balance memory usage and performance. | `2` GB |
| Segment File Bytes (Sink) | This setting is applicable when the buffer mode is configured as `disk` or `hybrid`. It controls the size of segmented files used to store messages, influencing the optimization level of disk storage. | `100` MB |
| Memory Overload Protection (Sink) | This setting applies when the buffer mode is configured as `memory`. EMQX will automatically discard older buffered messages when it encounters high memory pressure. It helps prevent system instability due to excessive memory usage, ensuring system reliability. <br />**Note**: The threshold for high memory usage is defined in the configuration parameter `sysmon.os.sysmem_high_watermark`. This configuration is effective only on Linux systems. | `Disabled` |
| Memory Overload Protection (Sink) | This setting applies when the buffer mode is configured as `memory`. EMQX will automatically discard older buffered messages when it encounters high memory pressure. It helps prevent system instability due to excessive memory usage, ensuring system reliability. <br />This configuration is effective only on Linux systems. | `Enabled` |
| Socket Send / Receive Buffer Size | Manages the size of socket buffers to optimize network transmission performance. | `1024` KB |
| TCP Keepalive | This configuration enables TCP keepalive mechanism for Kafka bridge connections to maintain ongoing connection validity, preventing connection disruptions caused by extended periods of inactivity. The value should be provided as a comma-separated list of three numbers in the format `Idle, Interval, Probes`:<br />Idle: This represents the number of seconds a connection must remain idle before the server initiates keep-alive probes. The default value on Linux is 7200 seconds.<br />Interval: The interval specifies the number of seconds between each TCP keep-alive probe. On Linux, the default is 75 seconds.<br />Probes: This parameter defines the maximum number of TCP keep-alive probes to send before considering the connection as closed if there's no response from the other end. The default on Linux is 9 probes.<br />For example, if you set the value to '240,30,5,' it means that TCP keepalive probes will be sent after 240 seconds of idle time, with subsequent probes sent every 30 seconds. If there are no responses for 5 consecutive probe attempts, the connection will be marked as closed. | `none` |
| Max Linger Time | Maximum duration for a per-partition producer to wait for messages in order to collect a batch to buffer. The default value `0` means no wait. For non-memory buffer mode, `5ms` will significantly reduce IOPS, though with the cost of increased latency. | `0` milliseconds |
Expand Down
2 changes: 1 addition & 1 deletion zh_CN/data-integration/data-bridge-kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -398,7 +398,7 @@ EMQX v5.7.2 引入了一项新功能,可以在 SQL 处理阶段将从设置的
| 缓存模式(生产者) | 定义在发送之前是否将消息存储在缓冲区中。内存缓冲可以提高传输速度。<br />`memory`:消息在内存中缓冲。如果 EMQX 节点重启,它们将会丢失。<br />`disk`:消息在磁盘上缓冲,确保它们可以在 EMQX 节点重启后存活。<br />`hybrid`:消息最初在内存中缓冲。当它们达到一定限制时(具体请参见 `segment_bytes` 配置),它们将逐渐转移到磁盘上。与内存模式类似,如果 EMQX 节点重启,消息将会丢失。 | `memory` |
| Kafka 分区缓存上限(生产者) | 每个 Kafka 分区允许的最大缓存大小(以字节为单位)。达到此限制时,将丢弃较旧的消息以为新消息腾出缓存空间。<br />此选项有助于平衡内存使用和性能。 | `2`GB |
| 缓存文件大小(生产者) | 当缓存模式配置为 `disk` 或 `hybrid` 时适用。它控制用于存储消息的分段文件的大小,影响磁盘存储的优化程度。 | `100`MB |
| 内存过载保护(生产者) | 当缓存模式配置为 `memory` 时适用。当 EMQX 遇到高内存压力时,将自动丢弃较旧的缓存消息。它有助于防止因过度内存使用而导致的系统不稳定,确保系统可靠性。<br />**注意**:高内存使用的阈值定义在配置参数 `sysmon.os.sysmem_high_watermark` 中。此配置仅在 Linux 系统上有效。 | `禁用` |
| 内存过载保护(生产者) | 当缓存模式配置为 `memory` 时适用。当 EMQX 遇到高内存压力时,将自动丢弃较旧的缓存消息。它有助于防止因过度内存使用而导致的系统不稳定,确保系统可靠性。<br />此配置仅在 Linux 系统上有效。 | `启用` |
| Socket 发送/收包缓存大小 | 管理 TCP socket 发送/收包缓存大小以优化网络传输性能。 | `1024`KB |
| TCP Keepalive | 此配置为 Kafka 连接器启用 TCP 保活机制,以维护持续连接的有效性,防止由长时间不活动导致的连接中断。该值应以逗号分隔的三个数字格式提供,格式为 `Idle, Interval, Probes`:<br />Idle:服务器发起保活探测前连接必须保持空闲的秒数。Linux 上的默认值是 7200 秒。<br />Interval:每个 TCP 保活探测之间的秒数。Linux 上的默认值是 75 秒。<br />Probes:在将连接视为关闭之前,发送的最大 TCP 保活探测次数(如果对端没有响应)。Linux 上的默认值是 9 次探测。<br />例如,如果您将值设置为 `240,30,5`,则意味着在 240 秒的空闲时间后将发送 TCP 保活探测,随后每 30 秒发送一次探测。如果连续 5 次探测尝试没有响应,连接将被标记为关闭。 | `none` |
| 最大延迟时间 | 每个分区生产者为了收集批量消息进行缓冲的最长等待时间。默认值为 0,表示不等待。 对于非内存缓冲模式,建议配置至少 5ms 以减少 IOPS(每秒输入输出操作次数)。 | `0` 毫秒 |
Expand Down
Loading