reciprocity checks

[andymck]

proto pr: helium/proto#388

This adds a reciprocity or bidirectional validation to POC validations in order to satisfy https://github.com/helium/HIP/blob/main/0106-hotspot-bidirectional-coverage-requirement.md

The flow is as follows:

1. Run regular validations on POC ( per beacon and any associated witnesses )

2. After regular validations update DB to track the 'last beacon' and 'last witness' timestamps per relevant gateway

• If regular validations declared the beacon valid then record last valid beacon timestamp ( 1 update per beacon)
• If regular validations declared the witness valid then record last valid witness timestamp ( 1 update per witness )

3. After updating the DB then run the reciprocity check:

• For each previously declared valid witness, render the final status of each as invalid where the witnessing gateway does not have a valid beacon within the previous N hours

• Render the beacon invalid if beaconer's gateway does not have a valid witness within the previous N hours

4. Output POC reports to S3 as normal

TODO:

• determine perf hit from additional DB access
• add last witness read cache
• add batch updates for last witnesses
• mock last witness cache

[jeffgrunewald]

what's the size of the db thread pool we're working with in IoT Verifier? this seems risky to be cloning per-beacon

[andymck]

the pool size is configurable...not sure what it is on mainnet currently.

As per the TODO above, i plan on refactoring db handling in here and wrapping all in a txn. The db access here is far from ideal