Weekly update 22.5.

[chimp1984]

Here is the current state of the p2p network work:

• Support of I2P (next to Tor and clearnet)
• Support multiple network layers and parallel redundant distribution of messages
• Encrypted messages
• Simple file based persistence
• Data storage for protected data (only owner can remove) and mailbox messages
• Inventory request (WIP)
• Investigating persistence options: https://github.com/chimp1984/misq/discussions/14
• Data storage discussion: https://github.com/chimp1984/misq/discussions/12
• Multiple network support discussion: https://github.com/chimp1984/misq/discussions/10

Next steps:

• Complete GetInventory protocol
• AppendOnly datastore
• PeerManagement
• Access control layer (pow)
• Metrics for system load (used for access control / dos protection)
• Best practice for modular protobuf usage

Notes:
I think I found a solution to support protected storage as persisted data. In Bisq that failed (TempProposal) as the node would miss RemoveData messages while offline and when loading the persisted data it would be inconsistent to the network state.
I use a different model to keep all AddDataRequests and RemoveDataRequests and embed the sequence number into those. By that a user would receive from the peers the missing RemoveDataRequests at the GetInventory protocol.

Beside that it will allow to verify the received data from the peer, which is also not possible in Bisq. As we usually request that data from seednodes only it is less problematic but as we want to avoid any special node in Misq that would be bad. So now when a user gets an AddDataRequest from a peer at the GetInventory request they still perform the verification if the data is valid to get added.

To persist by default all data (those data are kept in ram only in Bisq) will help for the intended use case of offline-offers (e.g. if user went offline but they keep their offers online for a while). The number of offers will likely be much higher then now and loading all at startup would come with higher bandwidth costs.

The GetInventory requests are limited to about 1 MB to avoid that connections get timeouts. If there are more data the response contains the number of truncated items, so the receiver can decide to repeat requests until that number is close to zero. It would be good to parallelize that process but I have no clear idea yet how to do that without introducing another request for asking for the total data, which would be hard as it would involve the delta handling which is not that cheap.

But I guess it should not be that hard, e.g. passing the strategy how to truncate the data, so the requester can make 4 requests with telling the receivers to use the first, second, third or forth 25% as priority for the filtering. That way they get pretty good chances to get most of the data after the 4 parallel requests are completed.

Also I would like to look into other filter approaches (Bloomfilters) beside sending the hashes (and sequence number) as the requests are getting larger with that.

Each data type will get its own storage file and service. This allows more fine grained control and applying different strategies/priotities. E..g User starts to load most important data, then continue with lower prio data.

There will be also no concept of "all initial data received". We will treat the GetInventory request like we would receive any data from the gossip protocol. So the app starts without data and get streamed in the data.

As Tor hidden service creation is much faster with v3 as it was with v2 the preliminary and updated data request concept is dropped as well. A Tor node is ready after the HS is ready.

We also will support different TTL and other metrics for mailbox messages. In Bisq that does not work as that data is encrypted and we don't know the payload, so all mailbox message have same TTL. In Misq we pack the sealed data into a wrapper which carries that metadata of the unencrypted message so its not lost. E.g. an AckMessage as mailbox message should have shorter TTL than a trade message.
Currently I also use dedicated files/services for each message type which got into the mailbox message. It would help to see which messages are mostly used and finetune parameters. But I am not sure yet if that would be too much...

[ghubstan]

I looked at the GraalVM last week. If graal's native-binary feature can build new Misq modules into usable native apps, the shipped packages would be much smaller, and startup times much quicker. I did some performance comparisons between jdk 16 and graalvm 16 on some EC and hash-2-address functions, but the runtime differences were small. Only the hash-2-address functions were faster, by about 8%.

Currently Bisq cannot be be run in the latest GraalVM; I haven't tried the older version. I did manage to build a native bisq app that would not start due to reflection related problems, and the binary size was 38Mb.

I'm now working on SQ's atomic tx project.

[ghost]

Update (27-05-2021) :

Spent a lot of time fighting to get electrum to be able to sign a combined transaction between Alice & Bob for funding a multisig trade. Electrum's transaction building routines are a bit weird and nowhere near as good as bitcoin core. Had to resort to deugging electrum code to find out how it works and eventually come up with something barely acceptable. Electrum 4 has rudimentary PSBT support and so alice and bob can each separately sign the same transaction then the PSBT can be combined into a finialized transaction. I did not try the payout part but it should work just the same as the deposit part - they are both two entities signing the same Tx.

https://github.com/jmacxx/misq_prj52_WIP/blob/master/misqWalletRpc4/README.md

I'll leave the Electrum demo at this stage and move on to LND.