Embed Trusted Default Seed Nodes into src/config.ts for bootstrapping the P2P network

[CMCDragonkai]

Specification

The seed nodes in the Polykey network provide the first point of entry for bootstrapping new keynodes into the wider Polykey network.

We've now got support for deterministic root keypair generation according to a 24-word recovery key mnemonic, as per #202. We will be using this deterministic generation to generate our own set of seed nodes. This generation of seed nodes will be done in #285.

We'll require two sets of keynodes: one for testnet.polykey.io and another for mainnet.polykey.io. At least for the time being, we'll just be considering a single, shared seed node for both testnet and mainnet. An additional environment variable (PK_ENV) or config flag will be required to switch between which nodes to use (i.e. either mainnet in production, or testnet for debugging/testing).

Specifying the list of seed nodes

The list of seed nodes is required to be a static list of mappings from NodeId to NodeAddress (in this case, containing a hostname and port). The port for all of these seed nodes should be pre-defined (not defaulted) to be 1314.

Recall that the canonical version of a node ID is a 32-byte structure. Discussions of this can be found in #261. Therefore, this 32-byte structure would ideally be used for this static storage. Alternatively, we need to use some other string-encoded version of the node ID.

This list will be able to be supplied from 3 sources:

1. Environment variable: PK_SEED_NODES
   • can use a semi-colon separated structure: [email protected]:1314;...
   • parse with new URL(): e.g. new URL('pk://[email protected]:80'), new URL('pk://sdof8er@[2001:0db8:85a3:0000:0000:8a2e:0370:7334]:80'). Note the need for the prepended pk:// when using new URL
   • seed nodes specified in this environment variable would be expected to be used for both mainnet and testnet
2. Configuration file:
   • dynamically read from file by the Polykey agent at the path specified by nodePath
   • similar to the password file
   • structure of this? Most likely sufficient
3. src/config.ts:
   • statically define the seed nodes in the source code. e.g.:

   config = {
     seedNodesTest: {
       "ENCODED-NODEID1": { host: "testnet.polykey.io", port: 1314 },
     },
     seedNodesMain: {
       "ENCODED-NODEID1": { host: "mainnet.polykey.io", port: 1314 },
     }
   };

   • can specify both mainnet and testnet separately
   • switch selection based on config flag/environment variable

DNS resolution

There's a couple of aspects to supporting DNS resolution.

1. Performing the actual resolution from hostname to IP address
   • if necessary, may need to use the DNS module to resolve to the appropriate IP - this should be wrapped as a promisify (if already at an IP, no need to call this)
   • all of this resolution should be done at the NodeGraph level (before creating a NodeConnection)
   • resolution should occur dynamically - we should never store a resolved IP address over a hostname. The underlying IP address can be dynamic. Therefore, always dynamically resolve from hostname to IP when creating a NodeConnection. This should also occur for any reconnections on dropped connections (see Propagate networking connection error handling into NodeConnection error handling #224 for this)
2. Supporting the same IP address mapping to different node IDs.
   • if a hostname resolves to multiple IP addresses, then we choose 1 randomly. Use the all parameter in the dns.lookup options. For this, no need to do crypto random. Just use Math.random and select an index (https://stackoverflow.com/a/38448710/582917)
   • as a consequence of this, this means that a hostname could resolve to a different node ID
   • verification of the keynode should then succeed if the node matches any of the potential node IDs of the seed nodes
   • this may require some customisation of the current verification process
   • see Embed Trusted Default Seed Nodes into src/config.ts for bootstrapping the P2P network #269 (comment)

Node reconnection

There was also discussion of needing to resolve #224 in this issue too, such that we can implement some kind of reconnection logic when a previously initialised connection is dropped. This will need to be further discussed, and if required, may take another day or so.

I wonder if we actually care about reconnecting at all. Would it not be sufficient to simply propagate the error up, terminate the connection, and only re-establish connection if another agent to agent call is made? This would fit the current structure of NodeConnection (where we have a getConnectionToNode function that either uses an existing connection, or creates a new one).

Additional context

• Testnet Node Deployment (testnet.polykey.io) #194
• Propagate networking connection error handling into NodeConnection error handling #224
• Enable un-attended bootstrapping for pk agent start and pk bootstrap with PK_RECOVERY_CODE and PK_PASSWORD #202
• Canonicalise node ID representation #261

Tasks

1. Investigate 32-byte canonical representation of node ID
   1. Can we use this byte structure in our static storage of node IDs?
   2. Alternatively, what base encoding to use instead? base58btc? base64 is dangerous because there's unsafe URL characters in the encoding. base64url is also a potential one here (this is used for the JWS claims in the sigchain too, as this removes the unsafe URL characters.
   3. What does this mean for the Kademlia implementation in NodeGraph? From memory in Canonicalise node ID representation #261, we've refactored this such that we use the 32-byte representation. If we use a string encoding for the node ID, we need to firstly convert back to the 32-byte UInt8Array.
2. Refactor nodes domain:
   1. Refactor all notions of broker into seed nodes (as per Update testnet.polykey.io to point to the list of IPs running seed keynodes #177 (comment))
   2. Remove separation of brokerNodeConnections - the seed nodes should be treated as regular nodes, that get added to the NodeGraph (Kademlia system) (as per Update testnet.polykey.io to point to the list of IPs running seed keynodes #177 (comment))
      • A note on this though. This could mean that eventually the seed nodes are removed from a bucket (if the bucket were to become full, and they were considered the "least active" node. Is this desired? We'd still have their static information in our config, but not in the internal NodeGraph system. Perhaps this is okay, given that we should reduce reliance on these nodes as the node becomes aware of other nodes in the Polykey network
      • We'll be embedding them just like any other node into the Kademlia system. Any further complexity can be added later
   3. Ensure that we support both IP address and hostname in the NodeGraph system (i.e. where NodeId can then map to a NodeAddress containing either [ IP address, port ] or [ hostname, port ])
      • prototype this with NodeConnection and the GRPCClient to see what happens
      • if necessary, use the
   4. Ensure we support the chosen encoding/representation for the seed node IDs in `NodeGraph (from above point 1.3)
3. Provide support for the sources of seed nodes:
   1. Environment variable: PK_SEED_NODES
      • write parser to extract the components from the semicolon separated structure
   2. Configuration file
   3. Source code: src/config.ts

Old specification

Block 3 from #194 (comment)

Specification

Once we deploy the test keynodes #194, and make use of recovery codes #202 to generate deterministic root keys and root certificates, this will mean we will have a deterministic Node ID.

We should have a set of Node Ids to be trusted, but we can start with just 1.

The Node ID has to be embedded into our default trusted seed list in the source code.

This can be done in 3 tiers of configuration:

1. Environment variable PK_SEED_NODES
2. Configuration file, this would be a configuration used by the PK agent, and should be edited in the node path.
3. The default src/config.ts

This configuration needs to be something like:

const config = {
  seedNodes: {
    "ENCODED-NODEID1": { host: "testnet.polykey.io", port: 1314 },
    "ENCODED-NODEID2": { host: "testnet.polykey.io", port: 1314 },
  }
};

I forgot why I chose 1314, or if we had registered anything here before, but this is suitable as the well known port we use by default.

The ENCODED-NODEID should be a multi-base encoding of the Node ids. This means we should also resolve #261, as decoding the node ids will result in a Uint8Array to be used as the canonical node ID.

In the case of an environment variable, where one has to specify this configuration as a string, we should use a standardised env variable format involving : colon separation, unforunately the standard way of separating domains from ports is to use the :, therefore we use ; as the separation.

export PK_SEED_NODES='ENCODED-NODEID1=testnet.polykey.io:1314;ENCODED-NODEID2=testnet.polykey.io:1314

And a parser should be written to do that. Note that we require the port setting at all times. There is no defaulting of the port.

The resulting value should be referred to from src/config.ts. All other code should be doing import config from '../config'.

Now the NodeGraph that reads this, needs to support dns hostnames as well as IP addresses. At this point we have not tested what happens if we just use hostnames. The hostnames will be established in the NodeConnection which uses GRPC client and also the networking domain which uses uTP. You can prototype this and see what happens first.

To be safe, we can directly use the dns module: https://nodejs.org/api/dns.html to resolve hostnames if we have a hostname (https://nodejs.org/api/dns.html#dnslookuphostname-options-callback). If you have a IP address, don't call the DNS module, it's not necessary. The call will need to be wrapped as a promise, so use the appropriate promisify wrapper in the src/utils.ts.

The DNS module will resolve to the IP addresses that we want, and we would then use these. Because the underlying IP addresses may be dynamic, do not cache the IP addresses. Instead you resolve every time you want to establish a running NodeConnection. This allows us to change the IP addresses due to migration/redeployment/crashes... etc, and any PK agents connected to the seeds, will just re-connect (there is no re-connection logic built into NodeManager yet). Consult the #224 regarding propagating asynchronous errors from networking connections to NodeConnection. And when you re-connect, you resolve the DNS again to get the appropriate IP address.

At this point there's a problem with uTP not supporting IPv6, see #224 for that. So the testnet.polykey.io will only return A records which are IPv4 addresses.

Multiple A records may be resolved. When multiple are resolved, you must choose 1 randomly to be used. Use the all parameter in the dns.lookup options. For this, no need to do crypto random. Just use Math.random and select an index (https://stackoverflow.com/a/38448710/582917).

The encoded node ids will need to be generated via the #194 procedure, we will keep the recovery keys secret in our own private git repo. But also dog-food PK when appropriate here.

To accompolish this, these 2 additional things need to be done as well:

• Re-connection logic to node connections should be in NodeManager, the networking domain doesn't know when to reconnect, so this responsibility is in NodeManager.
• Need to resolve Propagate networking connection error handling into NodeConnection error handling #224 to ensure that connection closes in networking is propagated to nodes

Reference documentation should be done in the configuration of the PK (should have a configuration page) as a well as the testnode component architecture.

Additional context

• Testnet Node Deployment (testnet.polykey.io) #194
• Propagate networking connection error handling into NodeConnection error handling #224
• Enable un-attended bootstrapping for pk agent start and pk bootstrap with PK_RECOVERY_CODE and PK_PASSWORD #202
• Canonicalise node ID representation #261

Tasks

1. ...
2. ...
3. ...

[CMCDragonkai]

Some changes necessary here:

1. There is both testnet.polykey.io and mainnet.polykey.io both of these will have their own set of trusted nodes, and will have to encoded into src/config.ts. This might mean:

   config = {
     seedNodesTest: {
       "ENCODED-NODEID1": { host: "testnet.polykey.io", port: 1314 },
     }
     seedNodesMain: {
       "ENCODED-NODEID1": { host: "mainnet.polykey.io", port: 1314 },
     }
   };
   

   We would need to change to using main or test depending on an environment variable like PK_ENV.
2. The PK_SEED_NODES could use [email protected]:1314;.... This can be split on ; and parsed by new URL()

   new URL('pk://[email protected]:80')
   new URL('pk://sdof8er@[2001:0db8:85a3:0000:0000:8a2e:0370:7334]:80')
   

   Note the usage of pk:// used when used with new URL.
   When using PK_SEED_NODES it would have to override main and test being used.
3. Note that because testnet.polykey.io may answer with multiple IP addresses, which IP address could resolve to a different node ID. That would mean that verification would check that any of the possible node ids is allowed. This may require some customisation on the current verification logic in our P2P system. Because currently when you contact the IP with GRPCClient, you're expecting that to resolve to a specific Node Id, not any number of node ids. If the NLB can point to any number of nodes, that isn't a good idea, cause then the GRPCClient may complain that it doesn't verify.

Regarding point 3... normally TLS works here because multiple hosts can claim the same certificate. But right now it's not possible to have multiple nodes claim the same node ID. It seems either we have to change it so that when connecting to given IP, we can accept multiple potential node IDs, or we have to ensure that there's 1 public IP (and therefore EIP) per node. We have some code that can deal with the former already, especially since we can accept multiple node ids over a root certificate chain. And the latter is also possible if we were to investigate AWS more deeply and figure out how to connect an EIP to specific container processes on a 1 to 1 basis. I'm leaning towards the former atm.

[joshuakarp]

Before I start playing around with this, I'll need to resolve #282 (currently being fixed in #284).

[CMCDragonkai]

Changed from starting on 4th Nov to 8th Nov due to #282.

[joshuakarp]

Changed from starting on 8th Nov to 10th Nov due to #282.

[joshuakarp]

Specification updated.

[joshuakarp]

Most of the refactoring work here is done to support the seed nodes.

Last bit of work is adding the different sources of seed nodes. From discussions with Roger, I'm going to be adjusting the approach here slightly to instead use the Commander options specified in src/bin/options.ts. This means that all sources of the seed nodes will need to be specified as the node ID -> node address mappings themselves (i.e. no paths/filenames for configuration files to seed nodes). As such, we'll have the following (in order of priority):

1. CLI options: pk bootstrap -sn <[email protected]:1314> etc
   • like the ingress host option, should be able to specify multiple
2. Environment variable: PK_SEED_NODES
   • semi-colon separated structure: [email protected]:1314;...
3. src/config.ts

   config = {
     seedNodesTest: {
       "ENCODED-NODEID1": { host: "testnet.polykey.io", port: 1314 },
     },
     seedNodesMain: {
       "ENCODED-NODEID1": { host: "mainnet.polykey.io", port: 1314 },
     }
   };

Given that there's some slight differences in format here:

• CLI options: sequence of strings
• Environment variable: semi-colon separated structure
• src/config.ts: POJO structure

I'll need to figure out if/how Commander.Option can be used here.

[joshuakarp]

After the above, will then need to begin looking into supporting the propagation of errors from networking, and garbage collecting NodeConnections that are no longer valid. See #224 for discussion of this.

As such, I'd estimate another day or 2 for this. Adjusting this work to conclude on Thursday 18th Nov.

[joshuakarp]

So after discussion with Roger, we can simplify this.

We should have the following:

1. CLI flag for agent start and bootstrap: --seed-nodes -sn <nodeId@host:port;nodeId@host:port;...> (i.e. not variadic - simply a list of semicolon separated mappings)
   • this is an optional flag - no flag specified means read from environment variable, or if none defined there, take the default seed nodes from src/config.ts
2. Environment variable PK_SEED_NODES: nodeId@host:port;nodeId@host:port;...
   • note that an empty environment variable PK_SEED_NODES='' should be an explicit indication to start a node with 0 seed nodes
3. Default: src/config.ts: object-map structure

    config = {
      seedNodesTest: {
        "ENCODED-NODEID1": { host: "testnet.polykey.io", port: 1314 },
      },
      seedNodesMain: {
        "ENCODED-NODEID1": { host: "mainnet.polykey.io", port: 1314 },
      }
    };

By default, the CLI flag should overwrite any existing seed nodes (specified in either the environment variable or src/config.ts). Similarly, the environment variable should overwrite any seed nodes specified in src/config.ts.

If the seed nodes specified in the CLI flag or the environment variable are desired to be appended to the default seed nodes specified in src/config.ts, then we can borrow from the nix ecosystem and use an angle-bracket flag (<seed-nodes>) in the list of semicolon separated seed nodes. For example: nodeId@host:port;nodeId@host:port;<seed-nodes> (note, this flag can appear anywhere in the list).

We can also potentially extend this for a variadic CLI flag option:

1. --seed-node <nodeId@host:port> <nodeId@host:port>
   • that is, the default action of this flag is to append (instead of overwrite)
   • if this is done, we'd need to change the short-forms of the commander options such that there's no ambiguity. We propose --seed-nodes -sns and --seed-node -sn
   • this is a low priority though, given that the only time the seed nodes are injected is on pk agent start/pk boostrap

[joshuakarp]

Adjusting end date to Wednesday December 1 (from November 18 - delayed from refactoring work in #283), to give time for review and merge.