EventHandler coresystem upgrade

[MaGaMeGa]

Dew to the growing number of questions around the possible upgrade of event handler core system, I opened this issue to collect discussion points.

Some points to start:

• Event authorization:
  The current version is using a workaround to avoid the complete lack of authorization of delivery of events.
  The current version is assuming that a subscriber is authorized to all events of a provider if this subscriber as an authorized consumer
  to any of the given providers services.
  Once the NGAC gets implemented in the arrowhead framework, it would be good to have a way to define subscriber-eventtype-publisher
  or subscriber-service-eventtype-publisher or subscriber-eventtype-metadata-publisher or subscriber-service-eventtype-metadata-publisher, etc. authorization rules.

• Event/Service relation:
  In the arrowhead framework could there be events allowed to publish without a generating service? (system events should be allowed?)
  What would be the best practice to document the events that a service/system must/may produce?

• Event delivery guaranties:
  At the current implementation in a scenario when a publisher publish an event and the eventhandler find a subscriber with valid authorization for that event and try to deliver the event to that subscriber and the subscriber is not available (busy, offline, etc. ) than that subscriber will never get notified about that event.

• Permanent Event store:
  A permanent event store could be a way to approach the event delivery guarantee issue.
  In a constrained cloud deployment where every system/service has to produce some unified mandatory events ( service request started, service request fulfilled, service request received, service started, service finished, etc. ) a permanent event store could be a base of further monitoring/accounting systems.

[emanuelpalm]

@MaGaMeGa Thank you for initiating this discussion!

In the arrowhead framework could there be events allowed to publish without a generating service? (system events should be allowed?)

I believe this should be the case, yes. Possible solutions would be to use HTTP long polling, which would allow us to preserve the documentation format in use for existing HTTP services. Another alternative is to use any existing message queuing standard, such as AMQP, MQTT or XMPP. All of these standards have plenty of existing implementations. Using any of them would also help with delivery guarantees and event storage. The hard part, however, is that I'm unaware of any rigorous attempts at documenting an Arrowhead solution using any of them, even though I have heard about MQTT being used in particular.

What would be the best practice to document the events that a service/system must/may produce?

As far as I know, there is no normative way of documenting any non-request/response messaging paradigm. I believe we should, however, just use whatever solution fits our purposes best and then figure out how to document it afterwards.

[MaGaMeGa]

@emanuelpalm Thank you for your prompt contribution!

The effort you showed at other sections of the arrowhead project to formalize the planning/documentation/development processes was a major inspiration for me to open the issue above, where the goal of collection the discussion points is to create open, self-documented, trackable discussions about each point - which process, hopefully, would lead to a transparent, conversation-based requirements document, something formally similar to the well-presented document at: https://github.com/arrowhead-f/x509-profiles/blob/main/requirements.md

Then, with such a document in hand, any interested party could implement its own,
Arrowhead compliant, solutions.

I would also hope that some of the points, I listed or would be listed by others, would be quickly discarded by someone who has a much wider/deeper view of the theoretic base of the framework, than the lister of the point, by pointing out the articles or baselines were the foundation of the clarifications for the cloudy understandings could be found at.

Based on the above, I would wonder if we should discuss the technical details here, or should we postpone it to another time/space and base it on a detailed, mutually understood set of requirements.

However, I would very much like to learn about the base/background of your belief in non-service-produced events and also the possible pros and cons of allowing it.

• Would you allow device events?
• Would you allow cloud events?
• Would you see any benefits of mandatory events?

I believe we should, however, just use whatever solution fits our purposes best and then figure out how to document it afterwards.

I think you are quite right that there is a phrase of every project when creating a proof of concept, a minimal viable product, etc. could be parallel or even a prior to the specification of the detailed requirements, as a boost of conceptualisation, but I think the EventHandler core system has passed that phrase now, and would benefit from well-defined requirements/acceptance criteria before starting the upgrade of its architecture. I do not say, that if any requirements/acceptance criteria are once defined it should be final, but I think if such change occurs it should be well reasoned, and mutually agreed by each participant.

As far as I know, there is no normative way of documenting any non-request/response messaging paradigm.

As you pointed out elsewhere eclipse-arrowhead/core-java-spring#269 there are viable solutions, but again, my intent here was not to find the technical solutions, more to play around with possible options.

Should we create individual eventType description documents?
Should we include eventType descriptions in serviceDescriptions?

[emanuelpalm]

Based on the above, I would wonder if we should discuss the technical details here, or should we postpone it to another time/space and base it on a detailed, mutually understood set of requirements.

GitHub is excellent for (1) getting ideas out there and (2) making tangible proposals that need criticism/refinement. This issue would fall in camp 1, as far as I can tell. People can see that an issue exists with Arrowhead and that some people are thinking about it. At this stage, we need to know who has stake in this issue and who could contribute to fixing it, when it should be fixed, if backwards compatibility is a priority, etc.

By the way, I believe Arrowhead really needs a document containing normative use case descriptions. The document would list use cases with a lot of emphasis on non-functional requirements (throughput, latency, redundancy, portability, etc.). The use cases would be selected such that they are as different from each other as possible, while still being within the scope of what Arrowhead is set to establish. The point of the document would be to have something to test ideas and decisions against. Right now, we are all working with a rather vague picture of how Arrowhead will be used. @jerkerdelsing @jenseliasson What do you think?

[emanuelpalm]

• Would you allow device events?

Such as an event, published via something like XMPP, directed to all systems running on a particular device? Sure. Why not?

• Would you allow cloud events?

By which I assume you mean events that are published, via XMPP or otherwise, that any system belonging to the same local cloud is allowed to see? Again, why not? It is just a matter of enough flexibility in whatever system is used to define authorization rules.

• Would you see any benefits of mandatory events?

Such as in an event that all systems of a local cloud must subscribe to and react on? I don't believe that the Arrowhead project should make decisions on reactions to certain events being mandatory. We could, however, define something like a Local Cloud Operation Service that would allow for individual systems to subscribe to operation and maintenance events, such as requests to shutdown, restart, enter low-power mode, and so on. It would be up to the users of that service to ensure that all systems in their local clouds subscribe to and react on those events.

[emanuelpalm]

Should we create individual eventType description documents?
Should we include eventType descriptions in serviceDescriptions?

Good questions. I would assume both to be suitable. If an event is reused a lot by many different services, then maybe it should go into its own document. Perhaps the Semantics Profile (SP) document could be used for that? Events tailored for specific services should probably be defined in their SD and IDD documents.

[MaGaMeGa]

Such as in an event that all systems of a local cloud must subscribe to and react on?

Actually I was thinking about quite the opposite:
In a constrained Arrowhead Cloud if a provider wants to be complaint with the local rules,
like if it wants to fall under the monitor able category,
then each of its services must publish some mandatory service lifecycle events.
For example each service must publish request received, service process started, service process finished events,
each denied request must include a concrete value from a predefined error code set and a self claimed root cause, etc..

[emanuelpalm]

@MaGaMeGa That's an interesting suggestion. I like it.

(A) It requires that systems are able to describe what events they will emit to the service/system registry when they register, right? For an event to be emitted, it must adhere to some protocol, which means that a service must be involved. Could, perhaps, a service be described that, if consumed, implies that the consumer binds itself to publish certain events? The system interested in those events would provide the service, while all publishers would publish them to it.

(B) Would that be enough? Or did you mean that a message broker sits between publishers and subscribers? If that is the case, then there should be some way for the broker to indicate what services it "forwards". I'll try to explain it with a picture.

On the left side you can see the A case and on the right the B case. Note how in case A, the consumer pushes events to the service provider. Without mediation, this is the only way for a service to exist that establishes what events can be published directly between systems (unmediated).

In the B case, a mediator facilitates N consumers (event publishers) and M service providers (event subscribers), where all consumers and providers are interested in the same events (i.e. all service providers provide the same service. How do you think that a service provider should indicate that its service is being provided through the event broker? Should the event broker register all services its subscribers are offering? Other ideas?

[emanuelpalm]

Actually, now that I think about it, there is a third way of doing it. Look at the following:

In the first case, the service provider publishes the event, in the second the service provider subscribes to the events. What are the consequences for the mediated situation that it can go both ways? A service provider can be both a publisher and a subscriber. I guess a distinction need to be made between published events and observed events in service definitions?

[jerkerdelsing]

@palvarga to setup a meeting to close the issue.

[emanuelpalm]

Event propagation is something that could become easier to express for Arrowhead via something like what I propose here:

• Initial proposal for how to document event types #46