Erlang/OTP 23 Support

[maciozo]

Summary

Would it be possible to add support for OTP 23?

Desired Behaviour

Support for building/running CouchDB with OTP 23.

Possible Solution

I tried just adding the appropriate version numbers to rebar.config.script as was done in #2070, but the build fails when attempting to link b64url.

==> b64url (compile)
Compiled src/b64url.erl
Compiling c_src/b64url.c
/usr/bin/ld: cannot find -lerl_interface
collect2: error: ld returned 1 exit status
ERROR: sh(cc c_src/b64url.o $LDFLAGS -shared  -L"/usr/lib/erlang/lib/erl_interface-4.0/lib" -lerl_interface -lei -o priv/b64url.so)
failed with return code 1 and the following output:
/usr/bin/ld: cannot find -lerl_interface
collect2: error: ld returned 1 exit status

Some deprecated parts of erl_interface were removed in OTP 23 (https://www.erlang.org/news/140).

There may, or course, be further incompatibilities.

Additional context

Void Linux has moved on to OTP 23.

[jaydoane]

Another problem is that we currently have a lot of this kind of compiler warning:

/proj/couchdb-3/src/chttpd/src/chttpd.erl:279: Warning: erlang:get_stacktrace/0: deprecated; use the new try/catch syntax for retrieving the stack backtrace

But we can't replace those with the new try/catch syntax while we need to support versions previous to OTP 21. However, that same thread shows some workarounds that we could presumably use to patch all the existing incompatible code.

[maciozo]

Having to stick macros in all over the code does sound like a bit of a pain, unless it can be done with a find/replace without breaking everything.

I have basically zero knowledge of Erlang or how it works, but I do find it a bit strange that they'd remove existing syntax, and expect dependant projects to keep up. Unless it's common for Erlang projects to only support specific OTP versions?

[iilyak]

Another option is to deprecate erlang prior to 21. However it could be a tougher call because I think we have platforms which do not support 21 yet. Our current position is expressed here #2934 (comment).

[wohali]

@iilyak My understanding from IBM is that there have been issues in production with Erlang 21+, which is why the official binary packages we have on couchdb.org are built against 20.3.8.latest. If you or @rnewson or @davisp have updated info as to what's safe to move to, we could consider moving ahead, but the earliest would be 3.2.0. The impending 3.1.1 patch release would be too minor to change Erlang versions.

[davisp]

The only known bad versions I know of were explicitly excluded in the rebar.config.script.

…

On Mon, Aug 31, 2020 at 10:25 AM Joan Touzet ***@***.***> wrote: @iilyak <https://github.com/iilyak> My understanding from IBM is that there have been issues in production with Erlang 21+, which is why the official binary packages we have on couchdb.org are built against 20.3.8.latest. If you or @rnewson <https://github.com/rnewson> or @davisp <https://github.com/davisp> have updated info as to what's safe to move to, we could consider moving ahead, but the earliest would be *3.2.0*. The impending 3.1.1 patch release would be too minor to change Erlang versions. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3115 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAE3WLVBFJTVHIAXZCQXUDSDO6GZANCNFSM4QPXHR7A> .

[wohali]

FYI:

16:59 <+rnewson> couchdb goes quite wrong on 21/22
16:59 <+rnewson> the mochiweb modules thingy
16:59 <+rnewson> (at least)
16:59 <+rnewson> Wohali: I think it's all fixable but hard to keep 18/19 support
16:59 <+rnewson> I didn't get deep into it, but it bites me whenever I forget to add
                 erlang 20 to my path
17:00 <+rnewson> cloudant is on 20 all over afaik

[iilyak]

Keep in mind that couch_log need to be updated as well to match the changes in OTP logging infrastructure. I did a test and logging of crashes from gen_server doesn't work on otp 22 and I am sure wouldn't be working on 23 without changes.

[janl]

I’ve started digging through these.

• fixing -Ilerl_interface is relatively easy, we are not using it anywhere, so we can just hotwire our local rebar copy:

apache/couchdb-rebar#2

diff --git a/src/rebar_port_compiler.erl b/src/rebar_port_compiler.erl
index 9679c80..44f7b0d 100644
--- a/src/rebar_port_compiler.erl
+++ b/src/rebar_port_compiler.erl
@@ -673,7 +673,7 @@ default_env() ->
                        "\" "
                       ])},
      {"ERL_EI_LIBDIR", lists:concat(["\"", erl_interface_dir(lib), "\""])},
-     {"ERL_LDFLAGS"  , " -L$ERL_EI_LIBDIR -lerl_interface -lei"},
+     {"ERL_LDFLAGS"  , " -L$ERL_EI_LIBDIR -lei"},
      {"ERLANG_ARCH"  , rebar_utils:wordsize()},
      {"ERLANG_TARGET", rebar_utils:get_arch()},

• get_stacktrace() is not getting removed until 24, so we have one more release to go, but the diff is totally bearable IMHO: https://github.com/apache/couchdb/pull/new/feat/3.x/otp23

  • this needs some more work to support <23 properly, made a logic bubu that I can fix up tomorrow.

• snappy needs to allow 23 feat: support Erlang 23 couchdb-snappy#12

diff --git a/rebar.config b/rebar.config
index 1785295..250cd2a 100644
--- a/rebar.config
+++ b/rebar.config
@@ -1,4 +1,4 @@
-{require_otp_vsn, "R14|R15|R16|17|18|19|20|21|22"}.
+{require_otp_vsn, "R14|R15|R16|17|18|19|20|21|22|23"}.
 
 {erl_opts, [debug_info, warn_unused_vars, warn_shadow_vars, warn_unused_import]}.
 {port_sources, ["c_src/*.cc", 

• jiffy needs… I don’t know ;D @davisp help: h/t @rnewson jiffy main already supports 23,

—

That’s as far as I got this time around.

[janl]

I gave this a spin and with the changes outlined here on 3.x, I got make check to pass on Erlang 23.

@iilyak Do you have more pointers on the logging behaviour. I found this reference in the Erlang 22 release notes:

gen_* behaviours: If logging of the last N messages through sys:log/2,3 is active for the server, this log is included in the terminate report.

— https://www.erlang.org/news/132

But I’m not sure if that’s what you mean.

[janl]

Erlang 20 is now fixed: 70334b3#diff-a07a3d3b357a34c4acb80b2e178e9a19e6c2ed9d91cb0aa8d53cadabb1ae8bd0R95

This is mainly a draft to show how it can be done. I’m happy to see a bike shed about making this cleaner.

I’m a bit confused tho, this broke builds tagged with 21 and 22, but I didn’t dig too deep in it. Are we sure our per-version builds are actually using the versions they say they do?

[wohali]

I don't see it, our PRs currently build against 20.3.8.11, 20.3.8.25 and 22.2.3. We don't build against any 21 in Jenkins and we don't have 23 in Jenkins yet.

[janl]

@wohali agreed that the logs show that now, there was just some intermittent state there when they were failing. The UI there is also not very intuitive, so I may have turned myself around.

[wohali]

It looks like @janl fixed this with #3422, but I don't see those commits on main yet.

@nickva can you cherry-pick #3422 onto main, then we can close this? Thanks!

[nickva]

@wohali good idea. I will do that. Beforehand I think we may want to clean up some of the applications in main which only work in 3.x (rexi, mem3, etc). I have a head-start on that already and will see how it looks in the next few days, then will port Jan's Erlang 23 support to main.

[wohali]

@nickva Though this is blocking my CI work @ #3504 , I'm instead just merging that to 3.x and you can pull my CI changes forward to main when you do this - as discussed on IRC. You'll also need to pull forward #3505 .

Thanks!

[nickva]

We have OTP 23 support now in both main and 3.x branches. I think we can close this