Proposal to combine with es loader

[rverma-nikiai]

Proposing to enhance the loader architecture in a more modular fashion. We have a core module at https://github.com/snowplow/snowplow-elasticsearch-loader/tree/develop/core/src. Maybe we can combine the overlapping abstract loaders to a core module, which can be distributed as compiled jar.

This will help in simplifying the loader code, will allow implementing/enhancing loader easier. As a java developer would be amazing for me if I can start with core jar and create a new loader extending StreamLoader directly.

Please give it a thought. Would be nice if we can publish the https://github.com/snowplow/snowplow-elasticsearch-loader/tree/develop/core as a jar in bintray also.

[alexanderdean]

I think the approach here would be to make a snowplow-loader-common library or framework.

[BenFradet]

Interesting idea! 👍

[rverma-nikiai]

Would be possible to provide the implementation for snowplow/snowplow-elasticsearch-loader#114

[rverma-nikiai]

@BenFradet any idea if I want to pick this up how am I suppose to proceed?

[BenFradet]

I think you would want a core module with everything that is common to both the s3 and es loaders.

And then have submodules for each.

Does that make sense?

[rverma-nikiai]

Actually, I was thinking of something totally opposite.
We have some client interface which can be implemented in any JVM language (java, scala, kotlin) and a core[engine] module, in which we can add these libraries. Something like spring-boot. The engine module detects and auto-registers the lib.

This may look an overkill, but we can go forward with a pluggable architecture, something like fluentd(https://www.fluentd.org/assets/img/architecture/pluggable.png).

We have implementations like snowplow-elasticsearch, which is responsible to both loading in and out of elasticsearch(https://github.com/fluent/fluent-plugin-kafka). Similarly for Kafka, kinesis, s3.

But your idea is also very good. I think if we can keep the application server within the core, it will give a similar result.

[BenFradet]

Makes a lot of sense 👍