Failed to rollback during fork

[yarkinwho]

Observed error:

... (Normal logs) ...
  INFO [12-31|20:13:30.937 UTC] ExecPipeline                       Forward done ---------------------------
  WARN [12-31|20:13:31.374 UTC] Can't link new block #412'853'649 (id:189ba59190b94070d213924c0ef14277f79a50b019cc19b5fa52f44f6d760902,prev:189ba59073866869a6d599d6420ea45248344eb6a25ae7cb391c54954a84c9ef)
  WARN [12-31|20:13:31.374 UTC] Fork at Block #412'853'648 (id:189ba59073866869a6d599d6420ea45248344eb6a25ae7cb391c54954a84c9ef,prev:189ba58f326fc01e92caee2e004b97c62ec1b2465eb2912e060b433b3a408926)
  WARN [12-31|20:13:31.374 UTC] Removing forked native block #412'853'649 (id:189ba591450660eb7a460093810c6be2676a00d9c6e2e45fa067ca32a15a2b50,prev:189ba59073866869a6d599d6420ea45248344eb6a25ae7cb391c54954a84c9ef)
  WARN [12-31|20:13:31.374 UTC] Reset upper bound for EVM Block #7'215'531, txs:0, hash:4660229b930d8c39da2aed9df5dbd3a456965b4d00fa8defb40e9853c6d19d1a to: #412'853'648 (id:189ba59073866869a6d599d6420ea45248344eb6a25ae7cb391c54954a84c9ef,prev:189ba58f326fc01e92caee2e004b97c62ec1b2465eb2912e060b433b3a408926)
  WARN [12-31|20:13:31.829 UTC] Can't link new block #412'853'649 (id:189ba591450660eb7a460093810c6be2676a00d9c6e2e45fa067ca32a15a2b50,prev:189ba59073866869a6d599d6420ea45248344eb6a25ae7cb391c54954a84c9ef)
  WARN [12-31|20:13:31.829 UTC] Fork at Block #412'853'648 (id:189ba59073866869a6d599d6420ea45248344eb6a25ae7cb391c54954a84c9ef,prev:189ba58f326fc01e92caee2e004b97c62ec1b2465eb2912e060b433b3a408926)
  WARN [12-31|20:13:31.829 UTC] Removing transactions in forked native block  #412'853'649 (id:189ba59190b94070d213924c0ef14277f79a50b019cc19b5fa52f44f6d760902,prev:189ba59073866869a6d599d6420ea45248344eb6a25ae7cb391c54954a84c9ef)
  CRIT [12-31|20:13:31.829 UTC] Unable to remove transaction ,txid_a:95fca302bab2aff9a40041d303a473bfc9dec22169ac3dd63a54587c06cda59e,txid_b:4b1fa8880479a8a6090850abffac15c9d510f5d9ae89960157127c55716ace58
  WARN [12-31|20:13:31.836 UTC] Can't link new block #412'853'650 (id:189ba5927912be47c78e3289dcda54d7a1f4334c526b81db7f6900571d926ce1,prev:189ba591450660eb7a460093810c6be2676a00d9c6e2e45fa067ca32a15a2b50)
  CRIT [12-31|20:13:31.837 UTC] Unable to find fork block 189ba591450660eb7a460093810c6be2676a00d9c6e2e45fa067ca32a15a2b50
  WARN [12-31|20:13:32.281 UTC] Can't link new block #412'853'651
... (Repeating Can't link errors) ...

What we should do:

1 try to handle this corner case
2 in the case of failing to process the fork, exit the program instead of entering a hopeless loop trying to resolve the error. Restarting the process will cause the system to restart from last irreversible block and can easily recover the system.

[elmato]

We believe that a possible explanation for this error is that the order in which we revert transactions from a forked native block differs from the order in which they are laid out in this native block.

Upon further inspection, the blocks received by the block_conversion_plugin (which handles native fork operations) already contain the transactions in the same order originally laid out in the ship_receiver_plugin. Therefore, the for_each_reverse_action function is indeed traversing them in reverse in the correct order.

Additionally, a minor improvement would be to shut down the node whenever an error occurs while removing a forked transaction from the EVM block, since we cannot recover from that state and must restart from the last irreversible block (which we already do upon startup).

WARN [12-31|20:13:31.829 UTC] Removing transactions in forked native block ...