Improvements to the library organization

[EinMByte]

Fragment from IRC:

<EinMByte> 1) Add an (actual) API to the core library.
<EinMByte> 2) Put I2PControl, Streaming etc in the client library.
<EinMByte> 3) Make the main application only use the client library and the API
<EinMByte> So the client library would use *only* the API to do core stuff
<EinMByte> (so the API is used instead of I2CP, since we don't have I2CP)

Comment from @fluffypony on the matter:
<fluffypon> I agree with that approach - core shouldn't be able to provide every-interface- ever when that can be done with stubby clients

This would mean that the application will consist of 3 rather than 4 components:

• The core library (libkovri-core). The only thing that changes here is the addition of an API that allows loose coupling with the client library, and possibly other applications that want to use the core directly.
• The client library (libkovri-client). This will also contain what is currently in libkovri-api. This library depends on libkovri-core, and its API to do whatever is necessary.
• The main application, which should contain the daemon and (command line) interface code. This is the front-end only,. My suggestion is that this repository should only contain a minimal user interface, as is the case right now. There can be separate repositories that use the client and core libraries to provide (various) graphical user interfaces.

Please discuss.

[anonimal]

Welcome back @EinMByte. A very warm welcome to you.

My decision for our current organization was based on present I2P specifications. The idea was to parallel the specifications because present APIs are not yet necessary to Kovri nor are they necessary to the Kovri client at this point (JFTR: I'm not opposed to reorganization).

@EinMByte I see where you are going with the joining of client and api, but that is under the assumption that we will even keep those APIs. I've given thought to a "kovri-lite" that works specifically with bitmonero and our present "kovri" which could be a more functional i2p client (that is, if we wish to keep everything and put in the work). With a current spec-appropriate organization, I imagine that maintaining the two repos would be easy as "kovri-lite" would only need core (with proposed API) and possibly a custom client bitmonero side.

@EinMByte @fluffypony I'd like to hear more about your ideas on this. @EinMByte can you provide more support for your proposal? What API do you propose for core? How many more API's do we really need? One of our objectives is to provide I2CP; do you disagree with that direction?

As I've said before, I feel that I2P has too many moving parts so I'd like to see some more benefits to this reorganization. None-the-less, I'm glad we're talking about this and I think we're moving in the right direction 😄

[fluffypony]

In discussions with zzz about the RNG it occurred to me that there were many design decisions (in I2P) that were good for the time, but in hindsight and given the landscape in 2016 they might be done differently. I don't think we should be afraid to push something out onto an interoperability layer, and implement our own standards internally.

Building on that, and on what you've said: let's think of it as kovri-core instead of kovri-lite, as that better describes its role. So kovri-core is effectively just the router, and that has some sort of API, be it 0MQ IPC or direct library hooks into functions.

Monero would use kovri-core, but so would Kovri (the full-featured router). Ideally kovri-core would be able to self-update, and also be able to service any applications that access it (so if I run Kovri and Monero on the same system there is only one instance of the running router).

Full featured Kovri is able to expose all sorts of interfaces: BOB, SAM, Lua, JSON RPC, whatever, and it's just transliterating that to kovri-core's API / IPC. If someone wants to provide native Python hooks into routing through the router and controlling it, they simply use kovri-core functions, no need to touch anything else.

Also this means there's no need to have two repos we keep in sync - Kovri just bundles kovri-core anyway.

[fluffypony]

Also to add: Kovri can add interfaces as plugins, which means we don't even have to develop SAM or BOB support; someone else could (ostensibly) create a kovri-sam plugin and we bundle it in to the release (or git submodule it)

[majestrate]

So kovri-core is effectively just the router, and that has some sort of API, be it 0MQ IPC or direct library hooks into functions.

I2PControl comes to mind but some things, like custom tunnel building algorithms, make more sense to be done via lua instead of I2PControl, IMO we should pick 1 and go with that instead of fragmenting everything up into a dozen different API components that end up highly coupled in the end.

Full featured Kovri is able to expose all sorts of interfaces: BOB, SAM, Lua, JSON RPC, whatever, and it's just transliterating that to kovri-core's API / IPC.

SAM and BOB are for clients to transport data while, lua / jsonrpc are for controlling the router. I think it would be best to explicitly separate these two types of API. A problem I see in the future is where I2CP fits in. I2CP is both data transport and a little bit of controlling the router, the control only applies to the destination belonging to the I2CP session but it's not simply data transport like SAM or BOB.

If someone wants to provide native Python hooks into routing through the router and controlling it, they simply use kovri-core functions, no need to touch anything else.

If we do that we're going to need to expose core to the world which means a lot of cleanup, too much to bite off for now, probably best to not think about that quite yet.

/end 2cents

[majestrate]

One of our objectives is to provide I2CP; do you disagree with that direction?

IMO I2CP is a requirement, it allows everything that exists made for i2p to "just work" (no really everything, SAM and BOB go over I2CP in the lower levels)

How many more API's do we really need?

Ideally 1, but realistically 2 (maybe 3 if we can get away with it).

• I2CP (exists+required, allows for all existing i2p applications to work out of the box with kovri, i2pd does not have this)
• I2PControl (exists+not required, works with java i2p, is the proposed Grand Unified Router Controller thing, i2pd and java i2p has this, kovri has this )
• lua ( not exists+not required, would be powerful tool for researchers, power users, hackers, tinkerers, no one has this)
• SAM (exists+not required, i2pd has it, java i2p has it, we don't anymore )
• BOB (exists+not required, i2pd has it, java i2p has it, we don't anymore )
• 0MQ (exists+not required, no one has this but it would allow people outside the i2p ecosystem to easily develop i2p applications without bullshit )

[EinMByte]

@anonimal I am not opposed to providing I2CP support, but I wish to be realistic: it's going to take a while to implement the full protocol.
As a first priority, I want to make sure people can start using the core as a library. Monero is good example.
However, in response to @fluffypony, the core is unlikely to be able to self-update (since it's not a stand-alone application). Then again, replacing the core should be a trivial matter and the API should facilitate this.
I2CP itself could be put into the client library. This is a fundamental difference with the Java router, but I see little reason to follow that design. We only need I2CP to allow easy coupling between Java and C++ components. The core does not take the form of an application, this is in contrast with the Java router (which is not intended to be bundled).

In response to "How many APIs?": for the time being we need only two APIs, but the client library/API requires no additional code. The interface to the core library is the new part. The components from the client library must be modified to use this API, and they can be used in a stand-alone manner.

When @majestrate is talking about plugins that allow custom tunnel building algorithms etc., you'll need direct access to the core in some cases. Some of these things cannot be done in the Java router today, and it would be nice to offer that amount of extensibility. To me this boils down to making sure that the API provided by the core is very flexible.

Of course I'm willing to write relevant code for all of the suggestions I have made above.

[majestrate]

(ftr my github handle is @majestrate , @psi is someone else )

[fluffypony]

@EinMByte we could publish "best practice" implementation that includes auto-updating to spec, or provide a small auto-updater library that can be called on startup + routinely or whatever. In terms of the actual auto-update, more discussion on #48, but effectively it's just stopping the router, replacing the .so / .dll for kovri-core, and starting it up again, right?

[EinMByte]

@fluffypony I agree, auto-update can be something in the client library or if that becomes too large it can be a separate library with some other basic stuff.

Something more specific:

As a first step, I propose to move the following things from client to the main kovri application:

• Daemon, DaemonLinux, DaemonWin32
• Config

This mainly requires ClientContext to be initialized in a different way (it shouldn't use the config directly). A builder pattern is needed there anyway.

[anonimal]

In terms of the actual auto-update, more discussion on #48, but effectively it's just stopping the router, replacing the .so / .dll for kovri-core, and starting it up again, right?

@fluffypony yes, but the trick is how to do that self-sufficiently without an external container or with as little co-dependency as possible - and all while reusing libcore functions to download/validate/install said update without moving libcore code around or redundantly copying libcore code into other libs. So, while it's been stated that yes libcore shouldn't nor can it (at the moment) auto-update, we'll have to think about how to reuse our code instead of reinvent - and that may take reorganization.

There are ways we can auto-update and our library layout will definitely effect those decisions. Yay #48.

How many more API's do we really need?

This was actually a rhetorical question. Available API's are well documented so the question was more of a response to this previous question:

What API do you propose for core?

@EinMByte, I agree with the reality of I2CP but what do you propose for an API (or is that TBD)?

I'm still not seeing enough reason to move libs around but I'm also not seeing enough reason to keep our current design. It also seems like we could throw out libapi all together instead of merging with libclient - or not. Daemon* + Config could move into libcore - or not (though, I can see the motive behind doing so (see below)). My point to @fluffypony is for reorganization - but there are other ways to approach the problem.

So, I see this ticket becoming a moot point. Can anyone provide more concrete relationships between the need to reorganize (staying on-topic to the ticket) and the need to implement a core API + everything else proposed? Regardless, we should open a new ticket for at least the core API proposal.

As a first priority, I want to make sure people can start using the core as a library.

I agree, and I see this as a good motive for reorganization. More motives would be appreciated though.

[anonimal]

It also seems like we could throw out libapi all together instead of merging with libclient - or not.

Edit: I didn't mean all of libapi, just the non-essentials after reorganizing.

[anonimal]

From Sunday's meeting:

2016-01-31 19:58:42     +EinMByte       anonimal: You were OK with merging libapi and libclient right? Forgot to ask since that was part of my short-term TODO for #98
2016-01-31 20:00:55     &anonimal        Yes, as long as the reasoning is: A) part of an overarching *essential* design decision and not just aesthetics (which we are good on because we've talked about it) and B) we use everything within t
he two libs and anything else is pruned or moved elsewhere.
2016-01-31 20:01:38     +EinMByte       Well "we" use it or someone else does, since it's a library
2016-01-31 20:02:30     &anonimal        By "we", e.g., if SAM/BOB was still there, B) would ensure that it's not until it is needed.

In addition, overall great discussion and things are moving along as planned. Related comments in #107.

[anonimal]

NOTICE: THIS ISSUE HAS BEEN MOVED TO GitLab. Please continue the discussion there. See #1013 for details.