High Level Design
LibEMP is a generic event processing framework. It eases the creation of distributed and fault-tolerant monitoring applications by providing easy to use abstractions and default implementations for many portions of an event processing stack. LibEMP is designed to be modular, so as to allow for mixing and matching functionality. It is split up into three types of components, each with different domains of functionality in mind, they are: The Monitor, the Buffer, and the Sink.
Events can come from many different sources, yet there are many attributes which are fairly common to keep track of: time the event was heard, the order in which they are generated, etc. In fact the types and structure of events you will see are typically well defined ahead of time, and will vary only a little or (more likely) not at all.
Event Monitors are LibEMP Components for listening for events from a specific source and letting the platform become aware of them in a generic way. Examples of Monitors include a Cron monitor which emits events every N milliseconds, or a RESTful monitor which pulls a REST service for events and emits them into the system.
LibEMP provides mechanisms for managing Event Monitors, including a plug-n-play like registration framework that applications can take advantage of. By registering Monitors with LibEMP, you tell the platform what types of Events can be expected, what kinds of Actions might be possible to take on that Event Stream, and the types of dependencies the Monitor requires of the system (i.e. requires unsandboxed networking).
TODO: Link to Event Monitor API.
Events need to be stored, either for later processing, visualization, or merely for logging. The Event Buffer encapsulates that storage. At bare minimum this could be an in-memory queue, however this does not preclude a more complex mechanism or even a remote database. In fact, a distributed Buffer is the mechanism by which LibEMP does distributed event processing.
LibEMP provides a number of buffer implementations with different goals and performance characteristics:
- Simple In-memory Queue: Fast, sequencing, in-memory queue; useful for multiple Monitors you wish to serialize events for. Requires a local Sink.
- Batching Multi-Queue: Fast, non-sequencing, in-memory queue; useful for multiple Monitors and a local Sink.
- Mnesia Queue: Eventually consistent, sequencing, potentially remote and on-disk, queue; useful for distributed systems with multiple nodes for Monitors and/or Sinks.
Buffers differ from Monitors in that only a single instance may be instantiated. Many more specialized buffers can be built on a per-application basis, but the modularity of LibEMP allows for easy testing and comparison.
TODO: Link to Event Buffer Definition.
To process events all one needs to do is implement an Event Sink. The EMP application provides an implementation of a dynamic, self-modifying, Event Sink which unifies over the domain of registered Events.
TODO: Link to Event Sink Definition.
