Zero downtime update

[daipom]

This changes the behavior when an update is performed while the service is active by using Fluentd's new feature: the zero downtime restart.
If you stop the service before the update, it will not be affected.

Change of specification:

• Changes auto restart conditions for update
• Adds FLUENT_PACKAGE_SERVICE_RESTART environmental variable
• Automate plugin install for update on demand

Changes auto restart conditions for update

RPM:

• Note: The specification of FROM-side version is applied. (It differs from DEB)
  • Ex. In case of v5.0.x -> v5.2.x, the spec of v5.0.x is applied.
• Before
  • The service automatically restarts only when it was active before update.
• After
  • The service automatically restarts with the zero downtime restart feature only when all of the following conditions are met.
    • It was active before update.
    • FLUENT_PACKAGE_SERVICE_RESTART setting is auto (default).
    • Both sides are version 5.2.0 or higher (Both sides support this feature).
  • Unless, the service does not automatically restart for update.

DEB:

• Note: The specification of TO-side version is applied. (It differs from RPM)
  • Ex. In case of v5.0.x -> v5.2.x, the spec of v5.2.x is applied.
• Before
  • The service automatically restarts when it was active or enabled before update
• After
  • The same as the RPM.

Adds FLUENT_PACKAGE_SERVICE_RESTART environmental variable

We can set this in the following file.

• RPM: /etc/sysconfig/fluentd
• DEB: /etc/default/fluentd

Value:

• auto (default):
  • The service automatically restarts with the zero downtime restart feature when all of the following conditions are met.
    • The service was active before update.
    • Both sides are version 5.2.0 or higher (Both sides support this feature).
  • Plugins are automatically reinstalled at that time.
    • Needs online envirionment.
    • Please See Automate plugin install for update on demand for details.
• manual:
  • If the service was active before update, the service does not restart automatically. The old processes continue to work after update.
  • You need to restart service manually as follows:
    • The zero downtime restart: Send SIGUSR2 to the supervisor process.
    • Normal restart: Restart service normally, such as systemctl restart.
  • Please use this when you want to manage plugins manually, or the environment is offline.

Automate plugin install for update on demand

When the service automatically restarts with the zero downtime restart feature, the missing plugins are automatically detected and reinstalled before restarting.

Previously, an automatic restart after the update was not recommended because it is problematic.
If you install some plugins manually, you need to reinstall plugins before restarting when the version of the embedded Ruby is updated.

So, for automatic restart, this feature provides automatic plugin installation.
This works as follows:

1. Collect the current plugin-list before udpate.
   • Recognize gems as plugins that begin with the prefix fluent-plugin-.
2. After installing new package, detect missing plugins by comparing the list with the default plugins.
3. Install the missing plugins.
4. Restart.

If you need to manage plugins manually, please set FLUENT_PACKAGE_SERVICE_RESTART to manual. (Ex. fixing the versions, offline environment, ...)
Then, automatic plugin installing and restarting are disabled.
You can manually install plugins and send SIGUSR2 signal to activate the zero downtime restart feature after update.

Caution: if you use a custom unit file, need to migrate it to use this feature safely

If you use a custom unit file, such as /etc/systemd/system/fluentd.service, please remove these 2 lines.

Environment=GEM_HOME=/opt/fluentd/lib/ruby/gems/...
Environment=GEM_PATH=/opt/fluentd/lib/ruby/gems/...

We don't need GEM_HOME and GEM_PATH.
They will be removed after v5.2.0, but if you use a custom unit file, you need to remove them manually.
If these variables are set, the zero downtime restart feature may not work correctly.
It is because the new process inherits the original environment variables if using this feature.

Mechanism of automatic restart with zero downtime

RPM

1. to-pre(2): Collect plugin-list.
2. Install TO-package
3. to-post(2): Do nothing.
4. from-preun(1):
   • Check auto or not.
   • Leave plugin-install flag and pid if need.
5. Uninstall FROM-package
6. from-postun(1): Disable %systemd_postun_with_restart.
7. to-posttrans: Install plugin and send SIGUSR2 if need.

DEB

1. from-prerm(upgrade):
   • Check auto or not.
   • Leave plugin-list and pid if need.
2. to-preinst(upgrade): Set tmp files for TO-package.
3. Install TO-package
4. from-postrm(upgrade): Clean tmp files of FROM-package.
5. Uninstall FROM-package
6. to-postinst(configure): Install plugin and send SIGUSR2 if need.

[daipom]

Waiting

• FLUENT_PACKAGE_SERVICE_RESTART: make empty or other values as auto #767

[kenhys]

Fixed DCO.

[kenhys]

LGTM.

[kenhys]

All checks has passed.