distributed sequencer algorithm

[hosie]

This PR introduces the detail documentation of the distributed sequencer protocol.

In domains ( such as Pente) where the spending rules for states allow any one of a group of parties to spend the state, then we need to coordinate the assembly of transactions across multiple nodes so that we can maximize the throughput by speculative spending new states and avoid transactions being reverted due to double concurrent spending / state contention. The distributed sequencer protocol described in this PR is the formal specification of how that operates and the responsibilities and expectations of each node involved.

This goes beyond describing what happens to be implemented in code at this moment in time and aims to be comprehensive in the specification of the algorithm. Further code changes PRs will be opened to bring the code inline with this architecture once it has been agreed.

Includes explicit (but concise) write up of the key architectural decisions made and the consequences of those compared to the alternatives.

Remaining TODOs before moving out of draft
Tweaks to protocol

• remove the dispatchNotification protocol point and replace with heartbeat messages from submitter
  • as a consequence of this, dispatchConfirmation has little value being a persistence point on the sender node so update the diagrams to not assume persistence here
• heartbeat messages should go to everyone in the group to help with detection of active groups in e.g. in cases of coordinator revival
• Delegation requests should include a field that states whether this is the first choice or not to help the proposed delegate decide if it agrees.
• Remove any mention of DelegationReturned message or possibly include it in a "potential future optimizations" section. The protocol should not rely on this.

Fill in gaps / corrections in writeup

• add Transaction lifecycle state diagram including persistence points
• add details on message exchange for domain context and how it helps with a smooth natural (happy path) coordiantor switchover and how the protocol allows the system to operate in certain error cases where that is not passed
• add timeout cases to TransactionDelegation and Endorsement message exchanges
• add section on configuration ( timeouts, range size etc...) and how they can be changed
• fill out more details and fix diagram for REQUEST DISPATCH CONFIRMATION
• complete details on Check availability sub process
• Endorsement request add flowchart and include where the startup message fits in
• Gather endorsement add flowchart
• Select available coordinator fix diagram and complete description
• add section for message-exchange-transaction-confirmation
• complete Terminology section

Formatting

• some of the sequence diagrams have so many lanes, they get scaled down so far that they are difficult to read without zoom. Either break them down to smaller sequencers, or combine lanes ( e.g. fold the database layers in) and/or pre-build the svg and include it as and image so that the reader can at least open the image in a separate tab
• dark mode need to tweak the css so that the images look good in dark model. This will require all images to be inline SVG or to have 2 copies of each image. I think I prefer 2 copies because inline SVG has the downside that chrome does not give the option to "open image in new tab" and sometimes that is useful.
• In various high level discussions (e.g. scenario walkthroughs) we make reference to message types and message exchange handshakes but do not currently include hyperlinks.
• could possibly break down the "Senders Responsibility" section. Maybe pull out failover to a separate section so that it shows up in the right nav.
• the actual protocol points ( message exchanges and sub-processes) do not appear in the right nav. Restructure the hierarchy levels so that they do

[peterbroadhurst]

It's a pedantic words thing, but leader election is the outcome of the distributed algorithm, rather than the thing that is performed directly by Coordinator selection on each given node.

e.g. leader election is not performed using coordinator selection on a single node in isolation. Rather leader election is a/the property that emerges from every node following the full specification in this document, and one important component of that specification is a deterministic decision made independently by each node to come to the same conclusion about a set of data (including a block number that changes over time inconsistently across the nodes, and static configuration including the committee that is agreed deterministically by recording it to the blockchain at contract deployment time).

[peterbroadhurst]

See note above about needing to be really specific about if it is node name or committee member identifier

[peterbroadhurst]

I don't think we have to solve for it in the first version of this specification, but I note that the ability to change n over time is something that might be necessary for tuning the algorithm on long-running privacy groups.

It probably fits into a wider note on "changing configuration"

[peterbroadhurst]

What is the intended behavior of the algorithm in this scenario?

• Highest ranked coordinator is B
• D delegates transaction to B
• B stops for a period
  • Longer than the heartbeat interval
  • Shorter than it takes for the block-range to change
• A is the second-highest scored coordinator
• D delegates transaction to A (because it gives up on B)
• B becomes available again
• C delegates transaction to B

At this point D has delegated a TX to A, but C has delegated to B.
I haven't (yet at this point in the spec) understood what would cause A and B to notice that they are both active (because at the time A became the coordinator, B was not available to ask).

[peterbroadhurst]

Ah, ok found this statement below:

If Node A comes back online, then it will begin transmitting heartbeat messages again because it has selected itself as coordinator given the current block height.

This seems complex to implement - what would cause A (B in my above sorry) to evaluate that it selected itself as the coordinator? It does not have any transactions after the the restart. Why would this contract (of the infinite potential contracts) be one after restart that it would know to select itself as a coordinator for?

[peterbroadhurst]

I did not understand this sentence.

This implies some future inference model and/or state tracking that seems very complex to achieve, and it's unclear to me the justification for this.

I note I believe this different statement is true, but it's currently the only thing I understand to be true with respect to evaluation of block height:

When each node in the group is notified that the block height of the chain has reached 4 (happens at different times on each node) they will recognize node B as the highest scoring coordinator when they have reason to run the scoring algorithm

[peterbroadhurst]

Ah, ok - I think I understand.
You are saying that the the combination of these two things will cause the scoring algorithm to occur:

• Detection of block height moving to 4
• The fact that transactions are unresolved for this contract address

This implies the algorithm re-evaluates the local "sender state" on detection of each block.

However, I'm worried I'm missing some detail as you say:

receive receipts of block 3

I'm not clear what receiving receipts means in this case, how it relates to block confirmations, or why it's block 3 (rather than 4) that's relevant here.

[hosie]

The block numbers in the diagram start at 0 therefore block 3 is the last block in the first range. Early versions of these diagrams had x+0,x+1 etc... where x was the block height when the privacy group was created but I felt that the x+.. was adding noise to the diagrams with no added value so I simplified it to be an absolute numbering and it happened to start at 0.

So when we see that block 3 has been mined, we can assume that any further transactions should come into the second range. Given that block 0 is the genesis block, it is probably not very helpful for me to be including it in the first range so I'll change it the numbering system to start at 1.

[peterbroadhurst]

The indexing wasn't the problem. Starting from zero was fine for me - we were referring to the same block.

I think you just spelt "Detection of block height moving to 4" as "receive receipts of block 3".

If instead you had said "confirmation of block 3 being mined, so the next block is now block 4" I'd have been all square.

[peterbroadhurst]

I'm struggling to understand the assumptions for this "meanwhile".

What makes it safe to send anything to B, if you are not absolutely certain that A has transferred your previous transactions to B?

The message from C->B that is sent by C because DelegationReturned was received by C could (in fact in the above sequence is reasonably likely) to reach B, before the message from A->B that provides information about B's previous transactions.

I wonder, given these facts:

• DelegationReturned will be sent by A->C after any message (sorry couldn't see the name) for I've taken ownership of submission for C-2 is sent from A->C
• The sender is responsible for their transactions
• The sender thus will know at the point it sends C-3 to B, that C-2 has reached the point of confirmed delegation on A

... whether C should just package up the information about which confirmed delegations it has on A, up to B.

[peterbroadhurst]

If we start to use domain context as a specification level thing (vs. just an implementation detail to meet the requirements of the spec) we will need to define what it is and how it works.

[peterbroadhurst]

Some code to experiment with behavior is great... we will need to come back and formally address the behavior of the specification afterwards.

[peterbroadhurst]

Some nesting of sub-bullets would help readability here

[peterbroadhurst]

Also, see comments above on my questions around "delegates all current inflight transactions to the new coordinator" being between coordinators. e.g. in the algorithm as described sometimes its the sender that delegates, and sometimes its another coordinator, and I'm not sure we've worked through that fully yet (or why we've chosen that approach vs. it's always the sender that delegates).

[peterbroadhurst]

This seems like an important part of the algorithm for sure, but I'm not sure why the spec for it is under variation in block height

Ok - understood now with the below bullet

• if not, then it rejects the endorsement and includes its view of the current block number in the rejection message

Maybe I'm wondering if this fact should have been mentioned earlier.

I did a search and endorsement is mentioned a lot earlier as something that comes out of the spec, but not really in these concrete terms.

[peterbroadhurst]

Very cool 🚀

[peterbroadhurst]

or re-delegation

Again we have the inconsistency here. This paragraph states that re-delegation is the sender's responsibility, but as discussed in comments above we seem to have split this responsibility between sender and coordinator.

As I've mentioned above, I think it would be a simpler to understand spec if it was always the sender's responsibility as this paragraph indicates.

[peterbroadhurst]

I'm not clear so far whether if a node has A-1 and A-2, if it is responsible for waiting for the ack of A-1 being delegated, before delegating A-2. This is important in edge cases such as failover, as discussed above

[hosie]

yes. I need to add some details generally to explain how explicit dependencies are handled.

[peterbroadhurst]

This seems to answer some of my earlier questions, but it also seems inconsistent again. This implies it's the sender that re-delegates (whereas the above diagrams implied some direct old-coordinator <-> new-coordinator comms)

[peterbroadhurst]

A detail point - but no transaction can be sent to a coordinator, until any referenced dependencies have been confirmed.

[peterbroadhurst]

I don't understand the difference between detection of "becomes unavailable" or "seems to have forgotten about it". I need a bit of help to understand.

I know in the spec, when a coordinator has a transaction it is responsible for regularly sending an "I'm still coordinating transaction X for contract Y" (referred to as "heartbeat") message. So one clear thing a sender can do is detect that it hasn't received an expected "heartbeat". But in that situation how would it know the difference between the coordinator having restarted and forgotten, vs. being unavailable.

[peterbroadhurst]

Covered in a detail question earlier this exact scenario. It was unclear to me what would cause a sender to find out "the preferred coordinator becomes available again"

[peterbroadhurst]

This is great. I assume this is only the transaction IDs that have been delegated by the sender this heartbeat is targeted to?

[hosie]

I am proposing that it may include other transactions and those should be ignored by the receiver of the message.

[peterbroadhurst]

Ok - think this is confirmation of my above point - the algorithm above is working to "is compatible block height?" (not "same") semantics.

[peterbroadhurst]

Digressing to an implementation point here... how do we think code could reasonably be constructed for this?

It seems like a sender node will have:

• A database with zero-or-more (maybe 1million) transactions it wishes to get submitted
• Each transaction will be on the same of different contract address (maybe 1million contract addresses)
• Each contract address has a different coordinator committee
• Each transaction might be in one of a number of states (a state machine diagram would be awesome for this)
  • Initial state - candidate to be delegated to someone
  • Delegated to self - I'm the coordinator so I don't need to healthcheck myself
  • Delegated to remote, but not yet requested submission
  • Delegated to remote, submission requested, but submission not occurred
  • Delegated to remote, submission requested, submission approved, but not yet confirmed
  • Delegated to remote, submission confirmed
    • While in any of these states, we might detect a the block range changing means this is the "wrong" coordinator
  • Done. No need to continue tracking
• The decision on whether an action should be performed for any transaction in this DB could be triggered by:
  • A message from the active coordinator for that txn (we have specs for these starting to finalize in this doc)
  • The lack of a message from the active coordinator fro that txn (the subject of this section)
  • The arrival of a new confirmed block
  • The arrival of an on-chain event (from a confirmed block) confirming a transaction
  • The completion of an earlier transaction that directly (through dependency) or indirectly (through memory-management of "in-fight" transactions for that contract) unlocks processing

[peterbroadhurst]

In our code in Paladin we've chosen to have evaluation loops against the DB and in-memory state, working at various levels. I guess I'm writing this question, because I'm not 100% sure how to correlate those code modules to the state machine(s) implied by this specification that I can see how it fully scales to "mempool management" of a large sea of transactions with efficient DB query and in-memory management.

Potentially this whole question should be moved out of this issue on refinement of a formal specification, into a separate one that is about updating our code to be an complete high-performance implementation of the spec.

[peterbroadhurst]

Going to pause my review here @hosie because I think there's a lot of outstanding conversations in what I've done so far that is affecting my ability to understand some of the more detailed items like the flow charts.

[hosie]

Thanks @peterbroadhurst these are a great points. I feel bad that that you had trudge through some basic inconsistencies in the writing resulting from in flight course adjustment but hopefully that does eventually help to share the journey in the thinking rather than just the final destination.

I'll focus on fixing the inconsistencies first so that the current draft is at least a coherent strawman within itself and then we can pick up the following key conversations points to decide what significant changes are needed:

Identity Vs Node as committee members. I think this is a short conversation. I don't have any strong objections to moving completely to Identity as committee members. Just need to work that through.

How much responsibility is actually delegated. At one point I was of the thinking that when a sender node delegates to the coordinator then it is up to the coordinator to make sure the transaction eventually gets submitted, even if that means delegating onwards to yet another coordinator. I really struggled with this idea as we started to consider fault tolerance. It requires the coordinator to persist the delegation and if the coordinator becomes permanently unavailable all in-flight transactions are now stranded). I now prefer the model where the sender remains responsible for ensuring that all of its transactions are actively being coordinated and it is the senders responsibility to delegate to a different coordinator if necessary. If a node becomes permanently unavailable, the only transactions that suffer are its own transaction. This re-think is the cause for a lot of the inconsistencies that you found so I'll clean that up.

A consequence of this decision is that it creates a possibility where the sender has delegated to a new coordinator but has lost connection to the old coordinator which thinks it is still the coordinator. So this increases the probably of double submission and I have added a sync point into the protocol to mitigate that. This protocol point ( dispatchConfirmation ) could be a big conversation in itself but I do think it is worth tracking as a sub conversation of the delegate vs abdicate discussion.

Another consequence of this decision is that it is now the responsibility of the sender to detect when the preferred coordinator for the current range comes back online. That hasn't been worked through in the write up yet and I think we need a better strawman before we can have a useful convo on this.

Message ID / Correl ID etc
Do we need these? does the DSP care about them or are they just a utility for the transport plugin?

Feasibility of efficient, scalable, implementation
I do think your points about implementation are useful here. Agree at some point, the detail implementation convo should move to another issue but if we can not convince ourselves (at least on paper) that the first reference implementation could be a high performing solution, then I think the spec would be broken.

Hot reconfiguration
This is not something I have given a ton of thought but agree that it is valuable to think about, at least as a thought experiment to test out the spec before we finalize it.

[github-actions]

This PR is stale because it has been open 30 days with no activity.

[github-actions]

This PR is stale because it has been open 30 days with no activity.