Liquidsoap 2.1.4

This is the fourth follow-up release for the liquidsoap 2.1.x release cycle!

This release brings more bugfix and stability to liquidsoap 2.1.x users with a couple of new features as well.

New features are:

• More logging around streaming exceptions to help diagnostics of hard failures while streaming
• Support for umask to bring the full unix/posix file/directory permissions control the way everyone's used to
• Added buffer_length method to buffers to help report and monitor buffer user (overrun/underrun, slow drain etc.)

The bugfixes include patch to prevent several situations that made the application crash under certain specific circumstances. Overall, we believe that this release should be a pretty stable iteration and highly recommend migrating to it. Make sure to always use a staging environment before updating your production systems!

We expect this release to be the last of the 2.1.x release cycle. The current development branch is about to be forked as rolling-release-v2.2.x and we should begin the stabilization process to bring that branch to a releasable state. There might still be bugfix releases on the 2.1.x branch but only for critical issues.

Thanks for all y'all continuing use of liquidsoap!

Full Changelog:

2.1.4 (2022-03-02)

New:

• Added buffer_length method to buffer operator.
• Always display error backtrace when a fatal exception
  is raised in the streaming loop.
• Added umask() to get the current umask and umask.set(...)
  to set the current umask (#2840)

Changed:

• Add break when restarting the external process in input.external.{rawaudio,rawvideo}
  (#2860, #2872)
• Removed disconnect method on input.harbor. This method was doing the same as
  the stop method. Added shutdown method to properly shutdown the
  source even when not connected to an output.
• Made process a string getter in input.external.{rawaudio,rawvideo} (#2877)

Fixed:

• Fixed parameter type for stats_interval in SRT I/O.
• Fixed type generalization on variable and pattern bindings
  (#2782)
• Fixed memory leak in http requests (#2935)
• Make sure that exception raised in request.dynamic never crash
  the process (#2897)
• Fixed filename getter being called multiple time in
  output.file (#2842)
• Space trim in interactive variables set on telnet (#2785)
• Fixed internal streaming logic in max_duration and crossfade.
• Make sure that there's at most one metadata at any given
  frame position (#2786)
• Fixed metadata.json.parse always returns an empty list (#2816).
• Fixed icy_id being ignored in output.shoutcast (#2819)
• Fixed shutdown livelock with some ffmpeg inline encoder,
  decoder and filter operators.
• Fixed input polling stop (#2769)
• Fixed parsed error report in %include directives (#2775)
• Fixed crash in external processes when received a Unix.EINTR
  event (#2861)
• Fixed crash in string.interpolate (#2883)
• Cleaned up srt support.

Liquidsoap 2.1.3

This is the third and, hopefully, last bug fix release for the 2.1.x release branch!

Overall, this release cycle was thankfully pretty calm, hence the more than two months since the previous one. The release contains a couple of new features that may also be useful such as being able to select or list portaudio devices and some enhancements in input.harbor dynamic capabilities.

Anyone on one of the 2.1.x releases should be able to upgrade safely to this version.

2.1.3 (2022-11-04)

New:

• Added time.string.
• Added error.on_error to report any error raised during the
  script's execution. Enhanced reported error positions (#2712)
• Added device_id and latency options to input.portaudio
  and output.portaudio to be able to choose the requested
  device. Use liquidsoap --list-portaudio-devices to see the
  list of devices (#2733)
• Added disconnect method to input.harbor, making it possible
  to disconnect a source client programmatically, including when
  a new client is trying to connect.

Changed:

• Send data in-memory in http.{post,put}.file when input data
  is already in memory. This allows to use plain Content-Length
  instead of chunked transfer encoding in these case, though
  libcurl seems to always prefer chunked encoding for put
  requests.
• Better error message when an encoder is not available on windows
  (#2665)
• Create output directory in HLS outputs when it does not exist
  using newly introduced perms permission argument (#2725)
• Removed restart_on_error argument on output.url and added
  restart_delay which implements a delayed restart. Added
  on_error argument to be notified of errors (#2731)
• Changed default encoding parameter in string.{quote, escape}
  to be null. Fallback to "ascii" encoding when no encoding is
  specified and "utf8" fails. This prevents unexpected script
  failures but might not be backward-compatible if you used a
  custom escape_char or special_char function (#2738)

Fixed:

• Enhanced methods typing support (#2659)
• Add support for song metadata (mapped to title) and url (mapped to
  metadata_url) in input.harbor (#2676)
• Fixed blank.* operator types.
• Fixed request metadata escaping (#2732)
• Fixed input.external.rawadudio mono input (#2742)
• Fixed http response body on redirect (#2758)

Liquidsoap 2.1.2

This is the second follow-up release for the v2.1.x branch featuring bugfixes, updates and some new features!

Noticeable API changes:

• http.{post,put}.file has been added to be able to send a file content via HTTP without bloating the memory.
• Many new formats have been added to the taglib metadata resolver. This can change your script's behavior. If you are seeing issues, please report to us. To revert to the old behavior, use this settings:

settings.decoder.file_extensions.taglib.set(["mp3"])

Other than that, this release should bring more stability to your script so go for it. After some testings, as always! 🙂

2.1.2 (2022-09-26)

New:

• Added string.char, string.getter.flush and string.getter.concat.
• Added http.multipart_form_data and http.{post,put}.file.

Changed:

• Allow sub-second values in sleep() (#2610)
• Allowed many new format for taglib (#2605)
• Add settings.ffmpeg.content.copy.relaxed_compatibility_check.set settings to allow relaxed
  compatibility check for ffmpeg copy content, making it possible to encode
  streams with various audio samplerate or video size when the container
  supports it.

Fixed:

• Stop error loop when opening a listening ssl socket with non-existent certificate. (#2590)
• Youtube HLS upload for live streams.
• Fixed data:... uri scheme to conform to RFC 2397 (#2491)
• Fixed multiple issues related to empty ogg/opus metadata (#2605)
• Ensure that video.add_text fails when the source does (#2609)
• Fixed metadata parsing in server.insert_metadata (#2619)
• Fixed extract_replaygain path (#2624, @parnikkapore)
• Fixed crash when terminating the process (#2585)
• Fixed channels conversion when using input.rawaudio (#2602)
• Fixed memory leak when using ffmpeg streams (input.ffmpeg, input.http, ...)

Internal Change:

• ref() implementation switched to OCaml's Atomic to prevent race conditions,
  thread.mutexify and mutexify functions removed. (#2603)

Liquidsoap 2.1.1

This is the first bugfix release of the v2.1.x release cycle, fixing some minor and less-minor issues. So far, the release cycle seems to be holding up fairly well!

Worth noting:

• ocurl is now a mandatory dependency. Its functionalities, support for http.{get,put,post,..} providing, among other things, remote requests resolution, were already expected by most users.
• Windows build for v2.1.0 did not have the ffmpeg encoders and decoders enabled.. 🤦‍♂️
• playlist.next now returns the requests in the right order. This could be breaking some existing scripts!
• playlist.remaining has been renamed into playlist.remaining_files. This could be breaking some existing scripts!

Full changelog:

2.1.1 (2022-08-28)

New:

• Added process.quote.command to generate complex quoted command strings suitable
  for use with process.run and os-independent.

Changed:

• Renamed playlist.remaining into playlist.remaining_files (#2524)
• Added id argument to replaygain operator (#2537).
• Made ocurl dependency required, added uri as required dependency (#2551)

Fixed:

• Fixed missing ffmpeg features on windows build.
• Fixed sync issues with ffmpeg.encode.* inline encoders (#2584)
• Fixed http.get issues when user-agent was not set (#2517)
• Fixed order of playlist.next returned requests.
• Fixed infinite loop when reloading a failed playlist (#2576)
• Fixed http requests with urls containing spaces (#2551)
• Fixed on_connect type for srt inputs and outputs.
• Fixed parsing issues with functions/variables definitions
  beginning with rec or replaces (#2560)
• Fixed infinite parse error loop (#2527)
• Fixed empty initial mp4 HLS segment.
• Prevent initial start for autostart and fallible sources.

Liquidsoap 2.0.x rolling release

This release provides liquidsoap assets before they are published as a new versioned release.

You can use it to install the latest stable code before it is published and test/prepare your production environment for it.

Rolling releases can also be useful for us to quickly detect and report bugs before the final published release!

⚠️ Warning ⚠️
Assets in this release will be deleted. If you are looking for permanent links to release assets, please head over to https://github.com/savonet/liquidsoap-release-assets/releases

For more details about our release process, please checkout https://github.com/savonet/liquidsoap#release-details

2.0.x (permanent rolling release)

This development branch has reached the end of its release cycle and will only be
maintained on a best-effort basis. You are advised to switch to the 2.1.x development
branch for all new liquidsoap work.

Liquidsoap 2.1.0

This is the first release of the 2.1.x release branch! 🎉

This version comes with some major improvements and breaking changes. As usual, you can get a (mostly) complete list in the migration page

In particular, the following changes are noticeable:

JSON parsing is greatly improved

You should now be able to do:

let json.parse ({
  foo,
  bla,
  gni 
} : {
  foo: string,
  bla: float,
  gni: bool 
}) = '{ "foo": "aabbcc", "bla": 3.14, "gni": true }'

For any one who has ever tried to parse json in their liquidsoap scripts, this is gonna be a game changer. We have a detailed article here

Partial function application has been removed.

This is not so much an improvement but something we thought was an acceptable trade-off. Support for partial application is creating a lot of complexity in the language's implementation and not a lot of users are actively using it. This change means that instead of doing:

def f(x, y) =
...
end

g = f(0)

You will now have to do:

def f(x, y) =
...
end

def g(y) =
  f(0, y)
end

Regular expressions are now first-class entities.

This should be familiar to anyone used to working with Javascript's regular expression. So, now, instead of doing:

string.match(pattern="\\d+", s)

You will now do:

r/\d+/.test(s)

There's a detailed description of this new feature here.

Full Changelog:

2.1.0 (2022-07-15)

New:

• Added support for variables in encoders (#1858)
• Added support for regular expressions (#1881)
• Added generalized support for value extraction patterns (#1970)
• Added support for string getter for http.{post,put} operations (#1984)
• Added output.youtube.live.hls
• Rewrote out internal JSON parser/renderer (#2011). Breaking change values
  that cannot be represented as JSON will now raise error.json when
  converted to JSON. infinite and nan floats can be exported using the
  json5 export format.
• Added socket API (#2014).
• Added support for ffmpeg bitstream filters (#2387)
• Added liquidsoap.version.at_least.
• Added video.rectangle, video.persistence.
• Added video.vumeter.
• Added video.slideshow.
• Added video.add_text.camlimages (#2202).
• Added video.text.* and re-implemented video.add_text.* from those (#2226).
• Added irc.channel operator to retrieve the contents of an IRC channel
  (#2210).
• Added new in-house parsing of metadata for some image and video formats
  (#2236).
• Added file.download
• Added new options for %ffmpeg copy encoder: ignore_keyframes
  and wait_for_keyframe (#2382)

Changed:

• Removed support for partial application, which should avoid some type errors,
  improve performance and simplifies the code related to the reduction (#2204).
• Video dimensions (width and height) can now be specified per stream in the
  type and are then used instead of the default ones. For instance, you can now
  write

  s = (single("file.mp4") : source(video(width=300,height=200)))
  

  in order to force the decoding of a file to be performed at the 300×200
  resolution (#2212).
• Video images are now canvas, which means that they do not directly contain
  the images, but are constituted of multiple images placed at various
  positions. This should make much more efficient operations such as making
  videos from multiple ones, adding a logo, etc. (#2207)
• output.youtube.live renamed output.youtube.live.rtmp, remove bitrate and
  quality arguments and added a single encoder argument to allow stream copy
  and more.
• source.on_metadata and source.on_track now return a source as this was the
  case in previous versions, and associated handlers are triggered only when the
  returned source is pulled (#2103).
• Made streams_info parameter of output.file.hls a record (#2173).
• Disable scrolling by default in video.add_text. You can re-enable it by
  using video.add_text(speed=70, ...).
• Added "example" sections to operators documentation, we now need to populate
  those (#2227).
• Default implementation of video.testsrc is now builtin, previous
  implementation can be found under video.testsrc.ffmpeg.
• Images can now generate blank audio if needed, no need to add
  mux_audio(audio=blank(),image) anymore (#2230).
• Removed deprecated timeout argument in http.* operators.
• Deprecated request.ready in favor of request.resolved.

Fixed:

• Fixed typo in status command of the mix operator.
• Fixed performances issues with input.ffmpeg and input.http (#2475)
• Fixed list.shuffle which was used to randomize playlists in playlist
  operator (#2507, #2500).

Liquidsoap 2.0.7

This is the seventh bugfix release of the 2.0.x release branch of liquidsoap. We promised to only release new versions for this branch for important bug fixes and found two:

• A memory leak with the opus headers. Technically this is a bug in ocaml-opus but we also had to release new binary assets.
• An inefficient initialization of decoding buffer and samplerate converters

This release addresses those issues as well as a minor one with initial shoutcast/icecast metadata. It is intended as the last versioned released with potential follow-up being pushed as rolling releases on a best-effort basis.

The initial 2.1.0 liquidsoap release was also produced as a companion release to this one. All users are now advise to start exploring the 2.1.x release branch as we expect most if not all bug fixes to be only applied on this branch.

Happy liquidsoap hacking!

2.0.7 (2022-07-15)

Fixed:

• Fixed memory leaks with opus bindings.
• Make sure decoding buffer and samplerate converter are only created once. (#2475)
• Make sure first metadata is always sent in icecast/shoutcast output (#2506)

Liquidsoap 2.0.6

This is the 6th and last bugfix release for the 2.0.x branch. Following this release, the 2.0.x development branch will only be updated on a best-effort basis. New projects are now advised to work with the rolling-release-v2.1.x branch, will is set to be the next stable release.

This release contains a set of minor fixes. Production scripts using any 2.0.x version are advised to switch to it but, still, after testing that there is no unforeseen regression.

A lot of effort has been put into debugging memory-releated issues with regard to requests. If your script uses a lot of requests (playlist, request.dynamic etc.) you should benefit from upgrading to this version.

Full changelog:

2.0.6 (2022-06-20)

New:

• Added video/mp4 to list of recognized mime types for request resolutions.

Changed:

• Log errors when using process.read (##2420, @martinkirch)

Fixed:

• Memory leak when executing process.run (#2424)
• Delay harbor server endpoint registration until application has started (#1589)
• Print user-readable encoder parameter error report.
• Fixed m3u metadata parsing when artist has a comma in their name (#2449)
• Cleanup failed request in playlist operator.
• Make sure requests are always cleaned up, making request.destroy calls
  optionals.

Liquidsoap 2.0.5

This is the 5th bugfix release of the 2.0.x branch, fixing mostly minor issues. This is also the last stable release on this branch as 2.1.x is expected to be the next stable release cycle.

2.0.5 (24-05-2022)

New:

• Extended m3u EXTINF parser to support empty duration and annotations.

Changed:

• Brought back mix operator (#2401)

Fixed:

• Allow crossfade duration override of 0.
• Buffer synchronization issues.
• Drop methods from ffmpeg filter input source
  types to avoid unecessary conflicts.
• Fix evaluation of abstract values with methods.
• Prevent some sources from being consumed when not active,
  namely ffmpeg inline encoders, soundtouch, resample and all
  the muxing operators.
• Raise runtime exceptions in string.replace failures with
  useful message. (#2408)
• Prevent request.dynamic from raising exceptions when checking
  if the source is ready (#2381)

Liquidsoap 2.0.4

Release 2.0.4 is mostly focused on CPU and memory optimization. As 2.0.x was getting adopted, we started receiving reports of increased CPU and memory usage compared to 1.4.4. We investigated them and found a couple of places that needed to be optimized:

• Our new media content API, which was extended to support the new ffmpeg raw and encoded data, was accessing content in a way that was not optimal
• Our internal format for PCM audio content had been switched from OCaml native float array to Bigarrays holding C arrays of floats. However, performances when accessing Bigarray float data in our audio filters written in OCaml were problematic.

These two issues have been fixed and we've been able to observe performances on-par with liquidsoap 1.4.4 for this release.

These improvements are particularly relevant if:

• Your liquidsoap script is using a lof of internal filters and audio processing operators.
• Your liquidsoap is handling multiple stream/inputs/outputs.

The other breaking change for this release is related to source shutdown methods. We realized that only specific sources (outputs and active sources) should have a shutdown method so we removed it from the other ones. Sources without a shutdown method are cleaned up when all the sources depending on them have been shutdown.

While this release should be, for the most part, functionally equivalent to any previous 2.0.x release, we recommend trying it in a staging environment prior to deploying it to production. The changes for the underlying audio format were pretty invasive and, also, there is always the chance that your setup was relying on an incorrect behavior fixed in the release.

Full changelog for this release:

Liquidsoap 2.0.4 (04-23-2022)

New:

• Added settings.video.add_text to enforce consistent choice of
  video.add_text implementation (#2302)

Changed:

• Make sure source shutdown can only be called on sources that
  can actually be shutdown:
  • Remove generic source.shutdown
  • Keep s.shutdown() method only on sources that are active.
    Refs: #2259
• Optimized memory usage when accessing frame content (#2266)
• Optimized memory usage when accessing ground terms.
• Allow crossfade duration getter to override duration at the
  end of each track if duration isn't set via metadata.
• Make sure crossfade metadata are not duplicated (#2153)
• Renamed map_metadata into metadata.map, deprecated map_metadata.
• Deprecatdd list.mem_assoc
• Enhanced remaining time when using add (#2255)
• Added timeout_ms to http.* to provide time in milliseconds, deprecated
  timeout argument.
• Connect output.icecast when data is available instead of when
  operator starts to avoid useless connections when underlying source
  fails immediately.

Fixed:

• Prevent infinite loops when crossfade duration is negative (#2287)
• Prevent mutex deadlock when recursively locking mutexes (#2274)
• Mark method add() as internal in request.queue, fix method length() (#2274)
• Fixed retry_delay being ignored in some cases in request.dynamic.
• Prevent race condition in external process handler.
• Fixed A/V sync when streaming encoded data via ffmpeg encoder (#2159)
• Prevent stopped/iddle sources from being restarted when reseting
  clock(s) after too much latency (#2278)
• Fixed registration of video.add_text.ffmpeg as possible implementation
  for video.add_text (#2302)
• Fixed http.* calls preventing liquidsoap from shutting down.
• Fixed http protocol not returning an error when timing out (#2242)
• Reworked ffmpeg filters feeding mechanism.
• Fixed inconsistencies in playlist.parser (#2257)
• Fixed inconsistent reselect in rotate (#2300)
• Fixed special characters escaping in video.add_text.ffmpeg (#2324)
• Fixed input.rawaudio and input.rawvideo when handling non-stereo
  content.