Blocking Bootstrap
When the system is being set up for the first time there will be a lot of data that has been accumulated since day one. This data needs to be transformed and stored in destination data store the way it would be done for events in real time.
However there is an issue in reusing the usual relay and client. Since the throughput of the client would be dependant on that of the destination data store, the clients would fall off the Relay. The buffer in the Relay does not help as the Relay throughput is far higher than that of Client and hence the buffer fills up and starts overwriting even before the client has been able to pull that event.
Here the Blocking Bootstrap is useful. It consists of two parts
- Blocking Producer : A producer similar to producer in Relay that can pull events from the Source or the events are pushed to it from the Source. Processing related to serialization is skipped.
- Partitioned Consumers : Consumers consume only the particular partition in order. Partitioned consumers use Event Mapper and Transformer as well as Data Layer similar to Aesop Clients.
Currently blocking bootstrap producer is implemented for MySql as source.
See sample for usage of Blocking Bootstrap