Skip to content

Notes on Buffers

Jump to bottom
Alexander Dean edited this page Sep 8, 2015 · 2 revisions

LibEMP Buffers are an unassumingly important mechanism. In many systems this mechanism is abstracted away and the decisions on how to handle failure cases are wrapped up in the implementation without the ability to change them. One such example of a failure cases is how the system handles message overflow when the producer is generating more information than consumers can consume (see [1] for an example of solutions and what they mean, [2] for just the slides).

Part of the goal is to make this decision more transparent and plug-able, where the goals and requirements of the system are described by the LibEMP Buffer selection you make. If it comes to writing your own, the API should push you to make a decision for each failure case, but make it easy to do so.

Some decisions that must be made:

  • Synchonous or Asyncronous "take" and "give"?
  • When is a "give" considered done? Is it pushed to disk locally? Do we need back-pressure here?
  • Should the buffer keep track of events after "take"? (i.e. what happens on Sink failure?)
  • How strict is order of events on "take"?
  • Are we a distributed system?
  • What is our required throughput vs safety guarantees?
  • etc.

[1] - https://www.youtube.com/watch?v=cQohRGGqevo&list=UUQ7dFBzZGlBvtU2hCecsBBg

[2] - http://www.erlang-factory.com/static/upload/media/1411659923678189runningforever.pdf

Clone this wiki locally