Skip to content

Relay Examples

Jump to bottom
Regunath B edited this page May 23, 2016 · 6 revisions

Example: Memory Relay in 5 minutes

This is the simplest example and requires no coding effort. Please refer to Maven Dependencies to setup the requisite repository location in your Maven build settings. The sample Memory Relay employs an Event producer that generates a series of events simulating fictitious data change events.

Follow these steps to setup and run your first Memory based Relay:

  • git clone aesop (ignore if done already)
  • Run mvn clean install in sample-memory-relay directory. This might take a while to download Aesop, Trooper and their dependencies from the various Maven repositories.
  • Start the Relay in sample-memory-relay directory :
java -Dlog4j.configurationFile=file:<absolute path to log4j2.xml> -cp "./target/*:./target/lib/*" \
org.trpr.platform.runtime.impl.bootstrap.BootstrapLauncher \
./src/main/resources/external/bootstrap.xml
  • A successful start will display a message like below on the console:
*************************************************************************
 Trooper __
      __/  \         Runtime Nature : SERVER
   __/  \__/         Component Container : com.flipkart.aesop.runtime.spring.RelayRuntimeComponentContainer
  /  \__/  \         Startup Time : 2,771 ms
  \__/  \__/         Host Name: Regu.local
     \__/
*************************************************************************
18:28:20.899 [main] INFO  o.t.p.r.i.bootstrap.spring.Bootstrap - ** Trooper Bootstrap complete **
  • Point your browser to http://localhost:9090. You should see the Aesop relays console: Aesop relays console

The next step is to run a Consumer for this Memory based Relay. Do the following:

  • Run mvn clean install in sample-client directory. This might take a while to download Aesop, Trooper and their dependencies from the various Maven repositories.
  • Start the Consumer in sample-client directory :
java -Dlog4j.configurationFile=file:<absolute path to log4j2.xml> -cp "./target/*:./target/lib/*" \
org.trpr.platform.runtime.impl.bootstrap.BootstrapLauncher \
./src/main/resources/external/bootstrap.xml
  • A successful start will display a message like below on the console:
*************************************************************************
 Trooper __
      __/  \         Runtime Nature : SERVER
   __/  \__/         Component Container : com.flipkart.aesop.runtime.spring.ClientRuntimeComponentContainer
  /  \__/  \         Startup Time : 2,278 ms
  \__/  \__/         Host Name: Regu.local
     \__/
*************************************************************************
17:23:46.443 [main] INFO  o.t.p.r.i.bootstrap.spring.Bootstrap - ** Trooper Bootstrap complete **
  • Now refresh the Aesop relays console at http://localhost:9090. You should see the newly registered localhost Consumer and the SCN of last consumed change event. The Metrics link in the dashboard shows live stats for event generation and consumption(this screenshot shows multiple consumers and highlights best and worst performing partition per consumer against producer SCN): Aesop Metrics

More Aesop Examples

This information is for more involved use cases of Aesop - for e.g. sourcing changes from a real database and applying it on downstream data sources.

Example : MySQL Relay

Follow these steps to run the MySql Relay:

  • This examples assumes a running MySql setup. It also assumes the user root with no password.
  • Add the following lines in my.cnf if not already present and restart mysql server
log-bin=mysql-bin
server-id=1
binlog_format=ROW
binlog_checksum=NONE (This last line to be added only for mysql 5.6 and above)

  • git clone aesop (ignore if done already)

  • Run mvn clean install in sample-mysql-relay directory. This might take a while to download Aesop, Trooper and their dependencies from the various Maven repositories.

  • A distribution directory will be created in the sample-mysql-relay directory. This directory is the distribution which contains the built sample.

  • Run the script create_person.sh present in distribution/sample-db directory. This will create the sample database, users, tables and data required for this sample to run.

  • Now run the script start.sh present in distribution directory.

  • Sample MySql relay will now start.

  • A successful start will display a message like below on the console:

01:36:07.548 [main] INFO  o.t.p.c.impl.logging.SLF4jLogWrapper.info 59 c.f.a.r.producer.MysqlEventProducer - Open Replicator has been started successfully for the file mysql-bin.000001
01:36:07.550 [main] INFO  c.l.d.c.c.netty.ServerContainer.start 372 c.l.d.c.c.netty.ServerContainer - Databus service started!
01:36:07.552 [binlog-parser-1] WARN  o.t.p.c.impl.logging.SLF4jLogWrapper.warn 77 c.f.a.r.p.e.OpenReplicationListener - Received Unsupported Event! FormatDescriptionEvent[header=BinlogEventV4HeaderImpl[timestamp=1406836599000,eventType=15,serverId=1,eventLength=116,nextPosition=120,flags=0,timestampOfReceipt=1406837167549],binlogVersion=4,serverVersion=5.6.16-log,createTimestamp=1406836599000,headerLength=19,eventTypes={56,13,0,8,0,18,0,4,4,4,4,18,0,0,92,0,4,26,8,0,0,0,8,8,8,2,0,0,0,10,10,10,25,25,0,0,-105,-47,-74,-20}]
01:36:07.553 [main] INFO  o.t.p.c.impl.logging.SLF4jLogWrapper.info 59 c.f.a.r.s.RuntimeComponentContainer - Initialized runtime registry: com.flipkart.aesop.runtime.impl.registry.ServerContainerRegistry
01:36:07.557 [main] INFO  o.t.p.c.impl.logging.SLF4jLogWrapper.info 59 o.t.p.r.i.bootstrap.spring.Bootstrap - 
*************************************************************************
 Trooper __
      __/  \         Runtime Nature : SERVER
   __/  \__/         Component Container : com.flipkart.aesop.runtime.spring.RelayRuntimeComponentContainer
  /  \__/  \         Startup Time : 3,587 ms
  \__/  \__/         Host Name: Jagadeesh-Huliyar-Mac.local
     \__/
*************************************************************************
01:36:07.557 [main] INFO  o.t.p.c.impl.logging.SLF4jLogWrapper.info 59 o.t.p.r.i.bootstrap.spring.Bootstrap - ** Trooper Bootstrap complete **
  • Now update any row in the person table in the or_test database.
  • You should be able to see the change event in the logger.
01:36:07.572 [binlog-parser-1] INFO  o.t.p.c.impl.logging.SLF4jLogWrapper.info 59 c.f.a.r.p.m.i.DefaultBinLogEventMapper - Mapped GenricRecord for schema Person_V1 : {"id": 5, "firstName": "jagadeesh", "lastName": "hv", "birthDate": 452543400000, "deleted": "false"}
01:36:07.583 [binlog-parser-1] INFO  o.t.p.c.impl.logging.SLF4jLogWrapper.info 59 c.f.a.r.p.m.i.DefaultBinLogEventMapper - Mapped GenricRecord for schema Person_V1 : {"id": 5, "firstName": "jagadeesh", "lastName": "huliyar", "birthDate": 452543400000, "deleted": "false"}

Example : HBase Relay

Refer to HBase change event producer for design and implementation details of the HBase Event Producer. Follow these steps to run the HBase Relay:

  • This example assumes a running HBase setup. For test purposes, you may install a single node HBase instance.
  • Enable HBase replication and configure the SEP custom replication source. This is done in the standard hbase-site.xml file of the HBase instance. A sample is provided here : Sample hbase-site.xml. Follow instructions in the NGData HBase-SEP demo : HBase SEP setup to configure your HBase cluster and start it.
  • git clone aesop (ignore if done already)
  • Run mvn clean install in sample-hbase-relay directory. This might take a while to download Aesop, Trooper and their dependencies from the various Maven repositories.
  • Start the Relay in sample-hbase-relay directory :
java -Dlog4j.configurationFile=file:<absolute path to log4j2.xml> -cp "./target/*:./target/lib/*" \
org.trpr.platform.runtime.impl.bootstrap.BootstrapLauncher \
./src/main/resources/external/bootstrap.xml

  • A successful start will display the startup message on the console, similar to that for the Memory Relay explained earlier in this page. You will also be able to access the Aesop relays console at http://localhost:9090.

  • The next step is to generate some change events. This requires the following steps:

    • Setup schema for the sample HBase table "aesop-person" - this is done by running SchemaSetupMain from the sample-hbase-relay Maven module. It may be run as follows:
    java -Dlog4j.configurationFile=file:<absolute path to log4j2.xml> -cp "./target/classes:./target/lib/*" 
com.flipkart.aesop.relay.hbase.SchemaSetupMain
    • Injest data into the sample HBase table through a series of HBase Put/Delete operations - this is done by running PersonDataInjesterMain from the sample-hbase-relay Maven module. It may be run as follows:
    java -Dlog4j.configurationFile=file:<absolute path to log4j2.xml> -cp "./target/classes:./target/lib/*" 
com.flipkart.aesop.relay.hbase.PersonDataInjesterMain

  • The next step is to run a Consumer for this Relay. Follow instructions described earlier on this page to run the sample consumer available in sample-client Maven module. On successfully running the consumer, you can see change propagation metrics reflecting on the Aesop relays console at http://localhost:9090.

Example : Diff Relay

Refer to Pull change event producer for design and implementation details of the Diff based Pull event Producer. Follow these steps to run the Diff Relay:

  • git clone aesop (ignore if done already)
  • Run mvn clean install in the following Maven module directories :
    • sample-snapshot-serializer : Creates snapshots periodically by calling the User data iterator API/Service
    • sample-diff-relay : Produces change events for the User data by comparing snapshots that were created by the sample-snapshot-serializer This might take a while to download Aesop, Trooper and their dependencies from the various Maven repositories.
  • This example expects the User data iterator API/Service to return data objects of the type UserInfo as defined in the sample Maven modules. Ensure you have the iterator API/service running at the location mentioned in UserInfoServiceScanReader which currently points to "http://localhost:25151/userservice/v0.1/customer/batch"
  • Edit spring-batch-config.xml in sample-snapshot-serializer Maven module to update the serializedDataLocation property value of this bean :
<bean id="stateTransitioner" class="com.flipkart.aesop.serializer.stateengine.DailyStateTransitioner">
  <property name="serializerFactory">
    <bean class="com.flipkart.aesop.serializer.serializers.RootSerializerFactory"/>
  </property>
  <property name="rootNFTypeSerializer">
    <bean class="com.flipkart.aesop.serializer.serializers.UserInfoSerializer"/>
  </property>
  <property name="serializedDataLocation" value="/Users/regunath.balasubramanian/Documents/junk/zeno" />				
</bean>

to point to any existing directory. This is used to store serialized Netflix Zeno snapshots.

  • Start the snapshot serializer batch job in sample-snapshot-serializer directory :
java -Dlog4j.configurationFile=file:<absolute path to log4j2.xml> -cp "./target/*:./target/lib/*" \
org.trpr.platform.runtime.impl.bootstrap.BootstrapLauncher \
./src/main/resources/external/bootstrap.xml
  • Post successful startup, you may access the deployed job at : http://localhost:8182/jobs. The user data snapshot creation job is set to fire every 5 minutes. Check the directory set in the serializedDataLocation above for snapshot files after the job runs.
  • You would also need to edit the data source of the User data iterator API/service to modify record values between runs of the snapshot creation job. These changes will be captured in the snapshots and deltas created by the job.
  • Next, in order to run the sample Diff producer in sample-diff-relay, edit the file spring-relay-config.xml to update the serializedDataLocation property value of this bean:
<bean id="diffInterpreter" class="com.flipkart.aesop.serializer.stateengine.DailyDiffInterpreter">
  <property name="serializedDataLocation" value="/Users/regunath.balasubramanian/Documents/junk/zeno" />    
    <property name="serializerFactory">
      <bean class="com.flipkart.aesop.serializer.serializers.RootSerializerFactory"/>
    </property>
  <property name="diffChangeEventMapper" ref="diffChangeEventMapper"/>
</bean>

to point to the directory containing Netflix Zeno snapshots of User data created by the batch job.

  • Start the Relay in sample-diff-relay directory :
java -Dlog4j.configurationFile=file:<absolute path to log4j2.xml> -cp "./target/*:./target/lib/*" \
org.trpr.platform.runtime.impl.bootstrap.BootstrapLauncher \
./src/main/resources/external/bootstrap.xml

  • A successful start will display the startup message on the console, similar to that for the Memory Relay explained earlier in this page. You will also be able to access the Aesop relays console at http://localhost:9090.

  • The next step is to run a Consumer for this Relay. Follow instructions described earlier on this page to run the sample consumer available in sample-client Maven module. On successfully running the consumer, you can see change propagation metrics reflecting on the Aesop relays console at http://localhost:9090.

Table of Contents

Clone this wiki locally