From 040d20eba487831742f3e2fc1aca0f2165350375 Mon Sep 17 00:00:00 2001
From: Philipp Schlarb
Date: Mon, 24 Apr 2023 11:53:46 +0200
Subject: [PATCH 1/7] Merge pull request #1689 from pSchlarb/newNetworkDoc
Added Documentation for creating a new Network
(cherry picked from commit fe1632ed4e1db9d40f3e69fb932c1f5544e89e1a)
Signed-off-by: Philipp Schlarb
---
docs/source/NewNetwork/CLIInstall.md | 116 +++++
docs/source/NewNetwork/CreateDID.md | 33 ++
docs/source/NewNetwork/NewNetwork.md | 144 ++++++
docs/source/configuring-2nics.md | 148 +++++++
docs/source/index.rst | 7 +-
docs/source/installation-and-configuration.md | 418 ++++++++++++++++++
docs/source/start-nodes.md | 2 +-
sample/Network/DIDs.txt | 71 +++
sample/Network/README.md | 132 ++++++
sample/Network/Stewards.csv | 5 +
sample/Network/Trustees.csv | 4 +
sample/Network/domain_transactions_genesis | 7 +
sample/Network/pool_transactions_genesis | 4 +
13 files changed, 1089 insertions(+), 2 deletions(-)
create mode 100644 docs/source/NewNetwork/CLIInstall.md
create mode 100644 docs/source/NewNetwork/CreateDID.md
create mode 100644 docs/source/NewNetwork/NewNetwork.md
create mode 100644 docs/source/configuring-2nics.md
create mode 100644 docs/source/installation-and-configuration.md
create mode 100644 sample/Network/DIDs.txt
create mode 100644 sample/Network/README.md
create mode 100644 sample/Network/Stewards.csv
create mode 100644 sample/Network/Trustees.csv
create mode 100644 sample/Network/domain_transactions_genesis
create mode 100644 sample/Network/pool_transactions_genesis
diff --git a/docs/source/NewNetwork/CLIInstall.md b/docs/source/NewNetwork/CLIInstall.md
new file mode 100644
index 000000000..4ce980c75
--- /dev/null
+++ b/docs/source/NewNetwork/CLIInstall.md
@@ -0,0 +1,116 @@
+# Installing the `indy-cli`
+
+The `indy-cli` is developed under the [indy-sdk](https://github.com/hyperledger/indy-sdk). This documentation may be not up to date.
+
+You will need to perform the following once for each `indy-cli` machine you would like to set up (only 1 is required).
+It is recommended that you install the `indy-cli` in your native work environment if possible, so you always have it available even when traveling.
+
+## Containerized `indy-cli` Environment
+
+The following sections describe how to install and configure the `indy-cli` directly on a machine or VM. However, possibly the most convenient option is to use a containerized `indy-cli` environment like the one included with [von-network](https://github.com/bcgov/von-network). For information on how to use the containerized `indy-cli` in `von-network`, refer to [Using the containerized indy-cli](https://github.com/bcgov/von-network/blob/main/docs/Indy-CLI.md)
+
+## Windows:
+To install the `indy-cli` on Windows 10 perform the following steps:
+1. Download https://repo.sovrin.org/windows/indy-cli/stable/1.16.0/indy-cli_1.16.0.zip and unzip it.
+ If there is a newer version under https://repo.sovrin.org/windows/indy-cli/stable/ it instead.
+2. Open a command prompt. (This will work differently if you use Windows Terminal).
+3. `cd` to the directory where you unzipped the `indy-cli` package. For example, if you unzipped directly in your ‘downloads’ directory like I did you would type: `cd \Users\\Downloads\indy-cli_1.14.2`
+4. Create a JSON Config file containing your taaAcceptanceMechanism in the directory where indy-cli.exe resides (I created \Users\\Downloads\indy-cli_1.14.2\cliconfig.json on my machine)
+ ```json
+ {
+ "taaAcceptanceMechanism": "for_session"
+ }
+ ```
+5. Run `indy-cli.exe --config cliconfig.json` to verify proper installation. You should see a new window appear with an `indy>` prompt, (If you are double clicking to start `indy-cli`, you need to right click on the .exe in your window and add the --config parameter first.) If you get an error stating that it is missing vcruntime140.dll then do the following:
+6. Download and install vc_redist.x64.exe from the Visual Studio 2017 section on the https://support.microsoft.com/en-ae/help/2977003/the-latest-supported-visual-c-downloads page, and then rerun indy-cli.exe to see if it works as described in previous step.
+7. Type ‘exit’ in the `indy-cli`
+
+## Ubuntu:
+To install the `indy-cli` on Ubuntu, perform the following steps from the ubuntu command line:
+
+1. `sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CE7709D068DB5E88`
+2. `sudo add-apt-repository "deb https://repo.sovrin.org/sdk/deb xenial stable"`
+3. `sudo add-apt-repository "deb https://repo.sovrin.org/deb xenial stable"`
+4. `sudo apt-get update -y`
+5. `sudo apt-get upgrade -y `
+6. `sudo apt-get install -y indy-cli`
+7. `cd ~`
+8. Create a JSON Config file containing your taaAcceptanceMechanism in your home directory:
+`vim ~/cliconfig.json`
+
+ Press the “i” key and paste the following into the file:
+ ```json
+ {
+ "taaAcceptanceMechanism": "for_session"
+ }
+ ```
+ Press the “esc” key then the following characters to write the file and quit
+`:wq`
+9. Run `indy-cli --config ~/cliconfig.json` to start the `indy-cli`
+
+## Mac:
+
+Since there is not a prepackaged version of the `indy-cli` prepared for the Mac, the following steps will help you to create an environment, build, and run the `indy-cli` in a Mac terminal.
+
+Open a Terminal
+Run the following commands in the terminal:
+
+1. `cd ~`
+2. `mkdir github`
+3. `cd github`
+4. `git clone https://github.com/hyperledger/indy-sdk.git`(might need xcode-select --install if error occurs)
+5. `/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"`
+6. `curl https://sh.rustup.rs -sSf | sh`
+7. Follow onscreen instructions to install rust
+8. `brew install pkg-config libsodium automake autoconf cmake openssl zeromq zmq`
+ NOTE: the openssl path needs to match what you currently have on your system
+
+9. Run > `ls /usr/local/Cellar/openssl/`
+ Note the name of the directory shown (the example below shows 1.0.2p but the latest version is 1.1.1l)
+
+ Use this directory in place of the one listed below in your .profile file
+
+10. Add the following lines to your` ~/.profile file `(making the correction shown in the previous step if needed)
+ ```
+ export PATH="$HOME/.cargo/bin:$PATH:~/github/indy-sdk/libindy/target/debug:~/github/indy-sdk/cli/target/debug"
+ export PKG_CONFIG_ALLOW_CROSS=1
+ export CARGO_INCREMENTAL=1
+ export RUST_LOG=indy=trace
+ export RUST_TEST_THREADS=1
+ export OPENSSL_DIR=/usr/local/Cellar/openssl/1.0.2p #use your path
+ export LIBRARY_PATH=~/github/indy-sdk/libindy/target/debug/
+ export LIBINDY_DIR=~/github/indy-sdk/libindy/target/debug/
+ ```
+11. Run the following commands from your terminal to build the `indy-cli`:
+ ```
+ source ~/.profile
+ cd ~/github/indy-sdk/libindy
+ cargo build
+ cd ../cli
+ cargo build
+ ```
+12. Create a JSON Config file containing your taaAcceptanceMechanism in your home directory:
+ `vim ~/cliconfig.json`
+
+ Press the “i” key and paste the following into the file:
+ ```json
+ {
+ "taaAcceptanceMechanism": "for_session"
+ }
+ ```
+ Press the “esc” key then the following characters to write the file and quit
+ `:wq`
+13. You can now run `indy-cli` from within a terminal by typing
+
+ `indy-cli --config ~/cliconfig.json`
+
+ `indy> exit` (To exit from the `indy-cli` prompt when you ar done)
+
+ If the above gives error regarding library not loaded libssl.1.0.0, you will probably need to run the following command (all in one line should work) to revert your version:
+
+ ```
+ brew uninstall --ignore-dependencies openssl; brew uninstall openssl;
+ brew install https://github.com/tebelorg/Tump/releases/download/v1.0.0/openssl.rb
+ ```
+
+
diff --git a/docs/source/NewNetwork/CreateDID.md b/docs/source/NewNetwork/CreateDID.md
new file mode 100644
index 000000000..eb8033d12
--- /dev/null
+++ b/docs/source/NewNetwork/CreateDID.md
@@ -0,0 +1,33 @@
+# Creating a DID using the `indy-cli`
+
+You will need to perform the following commands once for each `indy-cli` machine that you want to run on. The following commands contain suggestions to save certain values in a secure place. Please do not share those values or that place with anyone.
+
+_If you just need to quickly generate a set of secrets (Seed and wallet key), or a Seed, DID, and Verkey and do not have an `indy-cli` environment already setup, you can use the `indy-cli` features integrated into `von-network`. Refer to [Generate a set of Secrets](https://github.com/bcgov/von-network/blob/main/docs/Indy-CLI.md#generate-a-set-of-secrets), and [Generate your DID](https://github.com/bcgov/von-network/blob/main/docs/Indy-CLI.md#generate-your-did) for details._
+
+
+1. Start your `indy-cli` using the instructions from [Installing the `indy-cli`](./CLIInstall.md) for your platform.
+
+ All following commands are executing inside the `indy-cli`.
+
+2. Create a wallet with:
+
+ `wallet create key`
+
+ You will be prompted for a wallet key. What you type will not be displayed on the console. Your wallet key is a secure key that only you should know, and it should be randomly generated. Save it in a secure place for later use. You will use it every time you need to send transactions to the ledger from the `indy-cli`.
+
+3. `wallet open wallet_name key`
+
+ You will be prompted for your wallet key. What you type will not be displayed on the console.
+
+4. `did new seed`
+
+ You will be prompted for a seed. What you type will not be displayed on the console.
+
+ If you have lost your original seed or have never created one, then create a new one. This seed is used to regenerate your DID and to add your DID to your wallet(s).
+
+ The seed is a 32 character string that only you can know.
+
+ > WARNING: Whoever knows your Seed can recreate your exact DID in their own wallet and use it to manage the ledger.
+
+ Save your Seed in a secure place so that only you can recreate your DID whenever needed.
+ Also save the public DID and verkey generated from this step so that you will know and can verify your public DID.
\ No newline at end of file
diff --git a/docs/source/NewNetwork/NewNetwork.md b/docs/source/NewNetwork/NewNetwork.md
new file mode 100644
index 000000000..821b03a5a
--- /dev/null
+++ b/docs/source/NewNetwork/NewNetwork.md
@@ -0,0 +1,144 @@
+# Setting up a New Network
+
+## Introduction
+
+ The purpose of this document is to describe in some detail the process of building a brand-new Indy Node Network (Network) using 4 Stewards on their own separate nodes.
+ It goes into more details than [Starting a Network](../start-nodes.md).
+ These instructions are intended to be used for a distributed or “production” level environment but can be adapted to be used to build a private network.
+
+ This document is heavily based on [Create New Indy Network](https://docs.google.com/document/d/1XE2QOiGWuRzWdlxiI9LrG9Am9dCfPXBXnv52wGHorNE) and the [Steward Validator Preparation Guide v3](https://docs.google.com/document/d/18MNB7nEKerlcyZKof5AvGMy0GP9T82c4SWaxZkPzya4).
+
+## I. Create Network Governance documents (Optional)
+
+ Network Governance describes the policies and procedures by which your new network will run and be maintained. Here’s an example: [Sovrin Governance Framework](https://docs.google.com/document/d/1K8l5MfXQWQtpT49-FHuYn_ZnRC5m0Nwk)
+
+
+## II. Assign Network Trustees
+
+ Trustees are the people who manage the network and protect the integrity of the Network Governance. This includes managing the `auth_rules`.
+
+ For a production Network, at least 3 Trustees representing three different persons are required and more are preferred. For a test Network one Trustee is required and 3 or more are preferred (all Trustee DID’s may belong to the same user on a test network if needed).
+
+ Initial Trustees (3 preferred) must create and submit a Trustee DID and Verkey so that the domain genesis file can be built.
+
+ Each trustee has to [instal the `indy-cli`](./CLIInstall.md) and [create a Trustee DID](./CreateDID.md).
+
+ Once the Trustees have created their DID and Verkey give the Trustees access to a spreadsheet like [this one](https://docs.google.com/spreadsheets/d/1LDduIeZp7pansd9deXeVSqGgdf0VdAHNMc7xYli3QAY/edit#gid=0) and have them fill out their own row of the Trustees sheet. The completed sheet will be used to generate the genesis transaction files for the network.
+
+
+## III. Genesis Stewards
+
+ A Steward is an organization responsible for running a Node on the Network
+
+ Exactly 4 “Genesis” Stewards are needed to establish the network, more Stewards can be added later.
+
+ Each Genesis Steward’s node information will be included in the Genesis Pool file, so they should be willing to install and maintain a Node on the new Network for an extended period of time.
+
+ The Stewards must:
+ 1. Generate Steward DIDs as described in [Creating DID](./CreateDID.md).
+ 1. Install their node as described in [Installation and configuration of Indy-Node](../installation-and-configuration.md) (with some small adjustments):
+ 1. Determine a name for the new network and have the stewards substitute it in the appropriate places in the guide, such as when setting the network name and creating the directory when creating the keys for the node.
+ 1. They all need to stop at the normal place ([3.5. Add Node to a Pool](../installation-and-configuration.md#3.5.-Add-Node-to-a-Pool)) as instructed in the guide as the steps that follow differ when creating a new network. The following sections of this guide describe the steps required to start the new network.
+
+ Once the Stewards have created their DID and Verkey, and performed the initial setup of they node, give the Stewards access to a spreadsheet like [this one](https://docs.google.com/spreadsheets/d/1LDduIeZp7pansd9deXeVSqGgdf0VdAHNMc7xYli3QAY/edit#gid=0) and have them fill out their own row of the Stewards sheet. The completed sheet will be used to generate the genesis transaction files for the network.
+
+## IV. Create and Distribute genesis transaction files
+
+ Save the sheets filled out by the Trustees and Stewards as separate files in csv format, and use the [genesis_from_files.py](https://github.com/sovrin-foundation/steward-tools/tree/master/create_genesis) script to generate the `pool_transactions_genesis` and `domain_transactions_genesis` files for the network.
+
+ >Tip: The `generategenesisfiles` in `von-network` provides a convenient wrapper around the `genesis_from_files.py` and runs it in a container including all of the dependencies. For more information refer to [Generate Genesis Files](https://github.com/bcgov/von-network/blob/main/docs/Indy-CLI.md#generate-genesis-files).
+
+ Double check the files contain the correct information:
+ - The `domain_transactions_genesis` file should contain all of the DIDs and Verkeys for the Trustees (`"role":"0"`) and the Stewards (`"role":"2"`).
+ - The `pool_transactions_genesis` file should contain each of the nodes with all their unique information.
+
+ Publish the genesis files to a public location, such as a GitHub repository associated with your network. The Stewards and end users will need this information.
+
+ Inform the Stewards and Trustees where they can download the genesis files.
+
+ - The Trustees and Stewards will need to register the `pool_transactions_genesis` with their `indy-cli` to complete the setup and to be able to connect to the network once it's running. How and where they need to register the `pool_transactions_genesis` depends on how they setup their `indy-cli` environment; [Installing the `indy-cli`](./CLIInstall.md)
+
+ - The Stewards will also need to download the genesis files onto their nodes while completing the setup. All of the following steps are to be completed on the node.
+ 1. Set the network name in `/etc/indy/indy_config.py`, replacing `` in the following command with the actual network name;
+
+ `sudo sed -i -re "s/(NETWORK_NAME = ')\w+/\1/" /etc/indy/indy_config.py`
+
+ 1. Create a network directory and download the genesis files into it. _The directory name must be the same on all of the nodes and it must match the name of the network._
+ 1. `sudo -i -u indy mkdir /var/lib/indy/`
+ 1. `cd /var/lib/indy/`
+ 1. `sudo curl -o domain_transactions_genesis `
+ 1. `sudo curl -o pool_transactions_genesis `
+ 1. `sudo chown indy:indy *`
+ - It is important the files are owned by `indy:indy`.
+
+## V. Schedule a meeting to instantiate the new network
+
+ Invite all Genesis Stewards to a meeting where they can execute commands and share their screens for both an `indy-cli` and for their Validator Nodes being added to the Network.
+
+ > NOTE: It is very useful to go through some checks for each node to verify their setup before continuing. Some large amounts of debug and recovery work can be avoided by 5-10 minutes of checking configs of each node at the beginning of the meeting.
+ > - `/etc/indy/indy_config.py`
+ > - all nodes need to have the same network name.
+ > - the name of the network should correspond to the `/var/lib/indy/` directory on each node which contains the genesis files for the network, and the files in the directory should be owned by `indy:indy`.
+ > - `/etc/indy/indy.env`
+ > - all nodes should have local ip addresses in this file and be pointing at the correct ports.
+ > - Genesis files
+ > - Ensure both `pool_transactions_genesis` and `domain_transactions_genesis` files contain the expected content.
+ > - Verify the software version on all the nodes match
+ > ```
+ > dpkg -l | grep indy
+ > dpkg -l | grep sovrin
+ > ```
+ > - Network Connectivity
+ > - Use `nc -l ` (on the host), and `nc -vz ` (on the remote) to test the following.
+ > - Check the network connectivity between nodes using the `node_ip:port` combinations. Ensure that each node can communicate with all of the other nodes.
+ > - Check the network connectivity between the nodes and a client using the `client_ip:port` combinations. Ensure each node is accessible to client machines.
+
+ Once all of the checks are complete have the Stewards simultaneously start their nodes as described in section [3.5.2. Enable the Service](../installation-and-configuration.md#3.5.2.-Enable-the-Service) of the Installation and configuration of Indy-Node guide, and walk though the remainder of that guide.
+
+## VI. Configure the Indy Network
+
+### `auth_rules`
+ Update the network's `auth_rules` to help enforce the governance rules for the network.
+
+ For more information on `auth_rules` refer to:
+ - [Default AUTH_MAP Rules](../auth_rules.md)
+ - [auth_rules Walkthough](https://docs.google.com/document/d/1xk0A5FljKOZ2Fazri6J5mAfnYWXdOMl2LwrFK16MJIY)
+
+### `TAA` (Transaction Author Agreements)
+ Add a `TAA` to the network.
+
+ For more information on `TAA`s refer to:
+ - [Transaction Author Agreement - `indy-sdk`](https://github.com/hyperledger/indy-sdk/blob/master/docs/how-tos/transaction-author-agreement.md)
+ - [Transaction Author Agreement - `indy-plenum`](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/transaction_author_agreement.md)
+ - [Transaction Author Agreement Design](../../../design/txn_author_agreement.md)
+ - [TAA for CLI Walkthrough](https://docs.google.com/document/d/1Ma-EJkYpRfPOZApyEvcWrkb4EKn71XrIFd9KvZL0Whg)
+
+## Where to go from here?
+
+ ### Add more Nodes
+
+ For the network to remain in write consensus in the event of node failures the network needs to be comprised of `3f+1` nodes, where `f` is the number of failed nodes.
+
+ For a network of 4 nodes the network can remain in write consensus if a single node at a time fails, however if more than a single node fails at a time the network will loose write consensus and go into a read-only state. Similarly, a network comprised of 7 nodes can withstand up to 2 nodes failing at any given time. Therefore, it's recommended to have at least 7 nodes running in your network.
+
+ Examples:
+
+ | Failures to Withstand | 3f+1 | Required Nodes |
+ |--|--|--|
+ | 1 | 3(1)+1 | 4 |
+ | 2 | 3(2)+1 | 7 |
+ | 3 | 3(3)+1 | 10 |
+
+ ### Network Monitoring
+
+ [hyperledger/indy-node-monitor](https://github.com/hyperledger/indy-node-monitor) is the community supported and maintained tool for network monitoring.
+
+ #### Manual
+ - Run `indy-node-monitor` at least three times a day to detect any issues with the network.
+
+ #### Automated
+ - Run `indy-node-monitor` on a schedule (every 15-30 minutes) and add a notification plugin to alert you to any issues. _Please consider contributing your work back to the project._
+
+## Hands On Walkthrough
+
+ An example walkthrough of the above mentioned steps can be found in the `sample/Network` [folder](../../../sample/Network/README.md).
diff --git a/docs/source/configuring-2nics.md b/docs/source/configuring-2nics.md
new file mode 100644
index 000000000..38f7faa22
--- /dev/null
+++ b/docs/source/configuring-2nics.md
@@ -0,0 +1,148 @@
+# Configuring a 2 NIC Node
+First some caveats and warnings. These are notes based on setting up 2 NICs on an AWS VM. It might be possible to adapt them for other environments as well, particularly the "Configure Network Interfaces in Ubuntu" section.
+
+>WARNING:
+When you are doing network configuration, it is very possible to put your VM into a state where you are no longer able to log into it over the network. This may be difficult or impossible to recover from. Be very careful. If you have questions, doubts, or just need help, reach out prior to following these instructions.
+
+## Initial networking steps in an AWS console
+Create security group "validator client"
+- Port 22 for ssh
+- Port 9702 for Validator client connections
+
+
+Create security group "validator inter-node"
+- Port 9701 for Validator inter-node connections
+- Initially set up your Validator IP address to accept connections from anywhere, but later modify it as follows to only allow connections from specific IP addresses.
+ - To generate an allow list, run the following command on a Validator Node:
+
+ `current_validators.py --writeJson | node_address_list.py --outFormat aws`
+
+Setup Validator instance
+1. Provision VM
+ - Use security group "validator client" for the default network interface
+ - make note of the instance ID when completed
+2. Add and configure a 2nd network interface in AWS.
+ - On EC2 left side menu - Network & Security -> Network Interfaces -> Create Network Interface
+ 1. Subnet -> Select a different subnet in the same zone as your instance
+ 2. Private IP -> auto assign
+ 3. Security groups -> validator inter-node
+ - On the main screen, select the new interface and click the Attach button
+ Find and select the instance ID (recorded in step 1)
+3. Note the Network Interface ID of each network interface
+ - On EC2 left side menu - INSTANCES -> Instances
+ - Select your instance
+ - At the bottom of the screen select the description tab and scroll down to ‘Network interfaces’
+ - Click on each interface and then record the ‘Interface ID’ and the ‘Private IP Address’ for later use.
+4. Create 2 Elastic IP’s, 1 for each NIC, and associate them with the network interfaces
+ - On EC2 left side menu - Network & Security ->Elastic IPs
+ 1. Click Allocate New Address
+ 1. Give your new addresses appropriate names so that you can identify them later. (i.e. BuilderNet Client and BuilderNet Inter-Node)
+ 2. I used Amazon IP addresses, but you can use your own if you like
+ 3. Repeat steps 1 and 2 to create a second Elastic IP
+ 2. For each new Elastic IP do the following:
+ 1. Select one of the Elastic IP’s you just created
+ 2. Click Actions -> Associate address
+ - Resource type -> ‘Network interface’
+ - Network Interface ->