From cb650effeb25c286755f33d3489d337b1ba0d0ff Mon Sep 17 00:00:00 2001
From: Enzo Cioppettini <48031343+ecioppettini@users.noreply.github.com>
Date: Fri, 12 Jul 2024 13:41:51 -0300
Subject: [PATCH 1/4] document avail batcher settings

and other undocumented variables
---
 docs/home/1-setup/20-paima-bacher.md          | 77 ++++++++++++++++++-
 .../1-setup/4-environment-config-values.md    |  5 +-
 2 files changed, 76 insertions(+), 6 deletions(-)

diff --git a/docs/home/1-setup/20-paima-bacher.md b/docs/home/1-setup/20-paima-bacher.md
index a7adc38a..fd1bf2a0 100644
--- a/docs/home/1-setup/20-paima-bacher.md
+++ b/docs/home/1-setup/20-paima-bacher.md
@@ -49,7 +49,78 @@ First, ensure you have a config file `.env.*` ready in the root directory of you
 cat .env.mainnet >> ../.env.mainnet
 ```
 
-Of note, one key variable that needs to be set manually to use the batcher is the `BATCHER_PRIVATE_KEY`. This needs to be set as the private key of the wallet intended to be used for creating and posting the batched transactions (note, the wallet needs sufficient funds for posting to the contract). The expected format of the variable is a hex string without the `0x` prefix (ie. exactly what you get from MetaMask under Account details -> Export private key).
+## Configuration
+
+Many of the environment variables used by the batcher are shared with the
+engine, and these are already documented [here](4-environment-config-values.md).
+
+### Required variables
+
+Of note, one key variable that needs to be set manually to use the batcher is
+the `BATCHER_PRIVATE_KEY`.  For evm based chains this needs to be set as the
+private key of the wallet intended to be used for creating and posting the
+batched transactions (note, the wallet needs sufficient funds for posting to the
+contract). The expected format of the variable is a hex string without the `0x`
+prefix (ie. exactly what you get from MetaMask under Account details -> Export
+private key). For Avail the format is that of a [Substrate
+uri](https://polkadot.js.org/docs/keyring/start/suri/).
+
+** Note **: For Avail this is only used for self signed inputs, otherwise it's
+handled by the light client.
+
+- `CHAIN_URI`: The URI of the chain (rpc endpoint).
+- `BATCHER_PORT`: The port to listen for submissions from the frontend.
+- `BATCHER_DB_HOST`: Host of the batcher's database.
+- `BATCHER_DB_USER`: User for the batcher's database.
+- `BATCHER_DB_PW`: Password for the batcher's database.
+- `BATCHER_DB_NAME`: Name of the database.
+- `ENABLE_RECAPTCHA_V3`: See [this section](#batcher-security-recaptcha).
+- `RECAPTCHA_V3_BACKEND`: See [this section](#batcher-security-recaptcha)
+- `BATCHED_MESSAGE_SIZE_LIMIT`: Max size limit (in characters) for batched
+transactions in a single round.
+- `MAX_USER_INPUTS_PER_MINUTE`: Per user input submission limit.
+- `MAX_USER_INPUTS_PER_DAY`: Per user input submission limit.
+- `SECURITY_NAMESPACE`: [See this section for details](../200-read-write-L2-state/600-autosign.md#defining-your-namespace).
+
+## Network dependent settings
+
+- `BATCHER_NETWORK` can be used to choose the type of network the batcher submits
+to. Possible values are: `evm` and `avail`. If not set it will default to `evm`.
+
+### Avail
+
+When `BATCHER_NETWORK` is set to `avail`, the following variables are also required:
+
+- `BATCHER_AVAIL_LIGHT_CLIENT`: for the Avail's light client rest api. This is
+used for the transaction poster. [Reference](https://docs.availproject.org/docs/operate-a-node/run-a-light-client/light-client-api-reference#v2submit).
+- `SECURITY_NAMESPACE`: There is no contract address in this case, so this is
+required for the batcher. [See this section for
+details](../200-read-write-L2-state/600-autosign.md#defining-your-namespace).
+
+### Evm
+
+When `BATCHER_NETWORK` is set to `evm`, the following variables are also required:
+
+- `CONTRACT_ADDRESS`: The address of your Paima L2 contract.
+- `DEFAULT_FEE`: The fee for the contract call.
+
+## Optional variables
+
+- `CONTRACT_ADDRESS`: The address of your Paima L2 contract.
+- `MAX_BASE_GAS`: For gas estimation. Defaults to 50000.
+- `MAX_GAS_PER_BYTE`: For gas estimation: 32.
+- `BATCHED_TRANSACTION_POSTER_PERIOD`: Delay for the transaction poster loop
+when there are no inputs in the queue.
+- `CARP_URL`: The URL of a Carp instance. This is required if
+`BATCHER_CARDANO_ENABLED_POOLS` is used.
+- `BATCHER_CARDANO_ENABLED_POOLS`: A comma separated list of pool credentials,
+only users delegating to one of these pools will be able to post to the batcher.
+The expected format is the public key hash (28 bytes) as a hexadecimal string
+(56 characters).
+- `GAME_NODE_URI`: Used for input validation.
+- `DEFAULT_VALIDATION_ACTIVE`: Whether input validation is active or not. This
+requires `GAME_NODE_URI` to also be set.
+- `GAME_INPUT_VALIDATOR_PERIOD`: Throttle for the input validator.
 
 If you plan to use the batcher in web 2.5 environment, you also need to turn on the self signing feature by setting `SELF_SIGNING_ENABLED="true"` and filling in `API_KEY` value of your choice in `SELF_SIGNING_API_KEY` variable. You'll use this key afterwards on the server communicating with the batcher.
 
@@ -75,7 +146,7 @@ sh ./shutdown.sh
 
 ## Batcher Security (reCAPTCHA)
 
-As the Paima Batcher posts user submissions, you might want only to allow human users to submit data and avoid bots or malicious agents. This is a difficult task, but Paima Batcher can leverage Google's reCAPTCHA V3 and easily be integrated into games. 
+As the Paima Batcher posts user submissions, you might want only to allow human users to submit data and avoid bots or malicious agents. This is a difficult task, but Paima Batcher can leverage Google's reCAPTCHA V3 and easily be integrated into games.
 
 1. Create a reCAPTCHA V3 account and get the `site-key` and `secret-key`. (https://www.google.com/recaptcha)
   * Set `RECAPTCHA_V3_BACKEND` in the `.env.<NETWORK>` file with your `secret-key`.
@@ -84,7 +155,7 @@ As the Paima Batcher posts user submissions, you might want only to allow human
   * (OPTIONAL) Set `RECAPTCHA_V3_SCORE` in the `.env.<NETWORK>` file as a value between `0.0` and `1.0` (default `0.5` is used if not set). reCAPTCHA V3 returns a value between 0.0 and 1.0, where 0.0 is most likely a bot, and 1.0 is most likely a human. Scores < RECAPTCHA_V3_SCORE will be rejected.
 
 2. Add the reCAPTCHA code to your project
-  * Add the reCaptcha3 script `<script src="https://www.google.com/recaptcha/api.js?render=${site_key}" />` into your main HTML. 
+  * Add the reCaptcha3 script `<script src="https://www.google.com/recaptcha/api.js?render=${site_key}" />` into your main HTML.
   * Or call `injectReCaptchaToHTML()` in your frontend through the middleware.
 
 
diff --git a/docs/home/1-setup/4-environment-config-values.md b/docs/home/1-setup/4-environment-config-values.md
index c0fe2486..2abe760a 100644
--- a/docs/home/1-setup/4-environment-config-values.md
+++ b/docs/home/1-setup/4-environment-config-values.md
@@ -27,10 +27,9 @@ These these variables are used if you only need to synchronize a single network
   - `DEFAULT_PRESYNC_STEP_SIZE`: number of blocks to process in each step during initial presync phase. If not provided, a value of 1000 is used. Generally no need to change this value.
   - `CDE_CONFIG_PATH`: allows you to specify a custom location for your `extensions.yml` that is used to initialize primitive catalogue entries
 - Cardano extensions
-  - `CARP_URL`: The URL of a Carp instace, required when using Cardano primitives.
+  - `CARP_URL`: The URL of a Carp instance, required when using Cardano primitives.
   - `CARDANO_NETWORK`: One of `preview`, `preprod`, `mainnet`. It needs to match the network indexed by the Carp instance.
-  - `BATCHER_CARDANO_ENABLED_POOLS`: A comma separated list of pool credentials, only users delegating to one of these pools will be able to post to the batcher. The expected format is the public key hash (28 bytes) as a hexadecimal string (56 characters).
-- `DEFAULT_FUNNEL_GROUP_SIZE`: The number of blocks queried in one funnel sync step. If not set, a value of 100 is used. Generally no need to change this value.
+  - `DEFAULT_FUNNEL_GROUP_SIZE`: The number of blocks queried in one funnel sync step. If not set, a value of 100 is used. Generally no need to change this value.
 
 ## Others
 

From 95973b5db87c7cc0f3ca91909755e2a3d0f5198a Mon Sep 17 00:00:00 2001
From: Enzo Cioppettini <48031343+ecioppettini@users.noreply.github.com>
Date: Wed, 17 Jul 2024 21:37:42 -0300
Subject: [PATCH 2/4] improve the docs for the avail block funnel

---
 .../3-funnel-types/200-configuration.md             |  3 ++-
 .../3-funnel-types/700-parallel-avail-funnel.md     |  5 ++---
 .../3-funnel-types/800-avail-block-funnel.md        | 13 +++++++++++--
 3 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/docs/home/300-react-to-events/3-funnel-types/200-configuration.md b/docs/home/300-react-to-events/3-funnel-types/200-configuration.md
index 2cc19e0a..073e6309 100644
--- a/docs/home/300-react-to-events/3-funnel-types/200-configuration.md
+++ b/docs/home/300-react-to-events/3-funnel-types/200-configuration.md
@@ -20,7 +20,7 @@ Each entry should have a `type` variable, which can be one of
 - `avail-other`
 
 There can be only one entry of the `main` type. In the case of `evm-main`, this
-network is the one that has the Paima contract deployed. 
+network is the one that has the Paima contract deployed.
 
 Currently also there should only be one entry of type `cardano` and `mina`.
 
@@ -108,6 +108,7 @@ Avail:
   confirmationDepth: 10
   funnelBlockGroupSize: 100
   presyncStepSize: 1000
+  genesisHash: "0xdd60847daa1119ecc9bdd006b140087737ac03d260ce110ecd7cb33cf721be35"
 ```
 
 ## Extensions
diff --git a/docs/home/300-react-to-events/3-funnel-types/700-parallel-avail-funnel.md b/docs/home/300-react-to-events/3-funnel-types/700-parallel-avail-funnel.md
index a8e62f33..ec0f5d80 100644
--- a/docs/home/300-react-to-events/3-funnel-types/700-parallel-avail-funnel.md
+++ b/docs/home/300-react-to-events/3-funnel-types/700-parallel-avail-funnel.md
@@ -9,10 +9,9 @@ Avail:
   type: avail-other
   lightClient: 'http://localhost:7007'
   rpc: ws://127.0.0.1:9944
-  delay: 200 
-  confirmationDepth: 10
+  delay: 60 
   funnelBlockGroupSize: 100
-  presyncStepSize: 1000
+  genesisHash: "0xdd60847daa1119ecc9bdd006b140087737ac03d260ce110ecd7cb33cf721be35"
 ```
 
 ## Conceptually
diff --git a/docs/home/300-react-to-events/3-funnel-types/800-avail-block-funnel.md b/docs/home/300-react-to-events/3-funnel-types/800-avail-block-funnel.md
index 87367741..7f901d98 100644
--- a/docs/home/300-react-to-events/3-funnel-types/800-avail-block-funnel.md
+++ b/docs/home/300-react-to-events/3-funnel-types/800-avail-block-funnel.md
@@ -1,5 +1,14 @@
 # Avail block funnel
 
+An analogous to the [evm block funnel](./300-block-funnel.md) but for Avail.
+
+This funnel will do the following:
+
+1. Get the latest block number with verified data availability from the Avail light client through the `v2/status` endpoint. [Reference](https://docs.availproject.org/docs/operate-a-node/run-a-light-client/light-client-api-reference#v2status).
+2. Fetch a group of `funnelBlockGroupSize` headers (or less if we're already at the tip).
+3. Fetch all the submitted data from the Avail light client through the `v2/blocks/{block_number}/data` endpoint ([reference](https://docs.availproject.org/docs/operate-a-node/run-a-light-client/light-client-api-reference#v2blocksblock_numberdatafieldsdataextrinsic)). This means full blocks are never processed, because the light client can filter by app id.
+4. Use `TextDecoder` to read the binary input as text, and continue with the Paima concise format processing.
+
 ## Configuration
 
 ```yaml
@@ -7,8 +16,8 @@ Avail:
   type: avail-main
   lightClient: 'http://localhost:7007'
   rpc: ws://127.0.0.1:9944
-  delay: 200
-  confirmationDepth: 10
+  delay: 60
   funnelBlockGroupSize: 100
   presyncStepSize: 1000
+  genesisHash: "0xdd60847daa1119ecc9bdd006b140087737ac03d260ce110ecd7cb33cf721be35"
 ```
\ No newline at end of file

From c647115d81bb044683b9e0c902b9d70f088e39d1 Mon Sep 17 00:00:00 2001
From: Enzo Cioppettini <48031343+ecioppettini@users.noreply.github.com>
Date: Thu, 18 Jul 2024 00:29:17 -0300
Subject: [PATCH 3/4] reorganize the sections based on feedback

---
 docs/home/1-setup/20-paima-bacher.md | 80 ++++++++++++++++------------
 1 file changed, 46 insertions(+), 34 deletions(-)

diff --git a/docs/home/1-setup/20-paima-bacher.md b/docs/home/1-setup/20-paima-bacher.md
index fd1bf2a0..c5e5a430 100644
--- a/docs/home/1-setup/20-paima-bacher.md
+++ b/docs/home/1-setup/20-paima-bacher.md
@@ -54,19 +54,27 @@ cat .env.mainnet >> ../.env.mainnet
 Many of the environment variables used by the batcher are shared with the
 engine, and these are already documented [here](4-environment-config-values.md).
 
-### Required variables
+### Common settings
+
+The batcher supports submitting transactions to more than one network.  This
+section talks about settings that apply in both cases, and then follow the
+specific ones.
+
+#### Required settings
 
 Of note, one key variable that needs to be set manually to use the batcher is
-the `BATCHER_PRIVATE_KEY`.  For evm based chains this needs to be set as the
+the `BATCHER_PRIVATE_KEY`.
+
+For evm based chains this needs to be set as the
 private key of the wallet intended to be used for creating and posting the
 batched transactions (note, the wallet needs sufficient funds for posting to the
 contract). The expected format of the variable is a hex string without the `0x`
 prefix (ie. exactly what you get from MetaMask under Account details -> Export
-private key). For Avail the format is that of a [Substrate
-uri](https://polkadot.js.org/docs/keyring/start/suri/).
+private key).
 
-** Note **: For Avail this is only used for self signed inputs, otherwise it's
-handled by the light client.
+For Avail the format is that of a [Substrate
+uri](https://polkadot.js.org/docs/keyring/start/suri/).  Note this is only used
+for self signed inputs, otherwise it's handled by the light client.
 
 - `CHAIN_URI`: The URI of the chain (rpc endpoint).
 - `BATCHER_PORT`: The port to listen for submissions from the frontend.
@@ -82,47 +90,51 @@ transactions in a single round.
 - `MAX_USER_INPUTS_PER_DAY`: Per user input submission limit.
 - `SECURITY_NAMESPACE`: [See this section for details](../200-read-write-L2-state/600-autosign.md#defining-your-namespace).
 
-## Network dependent settings
+#### Optional settings
 
-- `BATCHER_NETWORK` can be used to choose the type of network the batcher submits
-to. Possible values are: `evm` and `avail`. If not set it will default to `evm`.
+- `BATCHED_TRANSACTION_POSTER_PERIOD`: Delay for the transaction poster loop
+when there are no inputs in the queue.
+- `GAME_NODE_URI`: Used for input validation.
+- `DEFAULT_VALIDATION_ACTIVE`: Whether input validation is active or not. This
+requires `GAME_NODE_URI` to also be set.
+- `GAME_INPUT_VALIDATOR_PERIOD`: Throttle for the input validator.
 
-### Avail
+If you plan to use the batcher in web 2.5 environment, you also need to turn on the self signing feature by setting `SELF_SIGNING_ENABLED="true"` and filling in `API_KEY` value of your choice in `SELF_SIGNING_API_KEY` variable. You'll use this key afterwards on the server communicating with the batcher.
 
-When `BATCHER_NETWORK` is set to `avail`, the following variables are also required:
+- `BATCHER_CARDANO_ENABLED_POOLS`: A comma separated list of pool credentials,
+only users delegating to one of these pools will be able to post to the batcher.
+The expected format is the public key hash (28 bytes) as a hexadecimal string
+(56 characters).
+- `CARP_URL`: The URL of a Carp instance. This is required if
+`BATCHER_CARDANO_ENABLED_POOLS` is used.
 
-- `BATCHER_AVAIL_LIGHT_CLIENT`: for the Avail's light client rest api. This is
-used for the transaction poster. [Reference](https://docs.availproject.org/docs/operate-a-node/run-a-light-client/light-client-api-reference#v2submit).
-- `SECURITY_NAMESPACE`: There is no contract address in this case, so this is
-required for the batcher. [See this section for
-details](../200-read-write-L2-state/600-autosign.md#defining-your-namespace).
+### Network dependent settings
+
+- `BATCHER_NETWORK` can be used to choose the type of network the batcher submits
+to. Possible values are: `evm` and `avail`. If not set it will default to `evm`.
 
-### Evm
+#### Evm
 
-When `BATCHER_NETWORK` is set to `evm`, the following variables are also required:
+##### Required settings
 
 - `CONTRACT_ADDRESS`: The address of your Paima L2 contract.
 - `DEFAULT_FEE`: The fee for the contract call.
 
-## Optional variables
+##### Optional settings
 
-- `CONTRACT_ADDRESS`: The address of your Paima L2 contract.
 - `MAX_BASE_GAS`: For gas estimation. Defaults to 50000.
-- `MAX_GAS_PER_BYTE`: For gas estimation: 32.
-- `BATCHED_TRANSACTION_POSTER_PERIOD`: Delay for the transaction poster loop
-when there are no inputs in the queue.
-- `CARP_URL`: The URL of a Carp instance. This is required if
-`BATCHER_CARDANO_ENABLED_POOLS` is used.
-- `BATCHER_CARDANO_ENABLED_POOLS`: A comma separated list of pool credentials,
-only users delegating to one of these pools will be able to post to the batcher.
-The expected format is the public key hash (28 bytes) as a hexadecimal string
-(56 characters).
-- `GAME_NODE_URI`: Used for input validation.
-- `DEFAULT_VALIDATION_ACTIVE`: Whether input validation is active or not. This
-requires `GAME_NODE_URI` to also be set.
-- `GAME_INPUT_VALIDATOR_PERIOD`: Throttle for the input validator.
+- `MAX_GAS_PER_BYTE`: For gas estimation: Defaults to 32.
+
+#### Avail
+
+##### Required
+
+- `BATCHER_AVAIL_LIGHT_CLIENT`: for the Avail's light client rest api. This is
+used for the transaction poster. [Reference](https://docs.availproject.org/docs/operate-a-node/run-a-light-client/light-client-api-reference#v2submit).
+- `SECURITY_NAMESPACE`: There is no contract address in this case, so this is
+required in this case. [See this section for
+details](../200-read-write-L2-state/600-autosign.md#defining-your-namespace).
 
-If you plan to use the batcher in web 2.5 environment, you also need to turn on the self signing feature by setting `SELF_SIGNING_ENABLED="true"` and filling in `API_KEY` value of your choice in `SELF_SIGNING_API_KEY` variable. You'll use this key afterwards on the server communicating with the batcher.
 
 ## Usage
 

From 1e9de7b29f7d7fc600254ac726a016d40e6bcbdd Mon Sep 17 00:00:00 2001
From: Enzo Cioppettini <48031343+ecioppettini@users.noreply.github.com>
Date: Thu, 18 Jul 2024 02:03:07 -0300
Subject: [PATCH 4/4] remove delay from avail block funnel example

---
 .../300-react-to-events/3-funnel-types/800-avail-block-funnel.md | 1 -
 1 file changed, 1 deletion(-)

diff --git a/docs/home/300-react-to-events/3-funnel-types/800-avail-block-funnel.md b/docs/home/300-react-to-events/3-funnel-types/800-avail-block-funnel.md
index 7f901d98..6a180aa8 100644
--- a/docs/home/300-react-to-events/3-funnel-types/800-avail-block-funnel.md
+++ b/docs/home/300-react-to-events/3-funnel-types/800-avail-block-funnel.md
@@ -16,7 +16,6 @@ Avail:
   type: avail-main
   lightClient: 'http://localhost:7007'
   rpc: ws://127.0.0.1:9944
-  delay: 60
   funnelBlockGroupSize: 100
   presyncStepSize: 1000
   genesisHash: "0xdd60847daa1119ecc9bdd006b140087737ac03d260ce110ecd7cb33cf721be35"