Optimise handling of duplicate (miner_id, piece_cid) pairs

[bajtos]

There may be multiple deals with the same combination of (miner_id, piece_cid) values, that's how the UNIQUE constraint proposed in #31 works.

From what I've seen in fil-deal-ingester, it's very very likely there will be many such duplicates.

Let's implement more efficient handling of such duplicates, so that we reduce the number of PieceIndexer/IPNI calls.

I see two situations to consider:

• When we list all deals missing the payload CID, there will be duplicate (miner_id, piece_cid) pairs. I think this is unlikely to happen in practice after we complete the initial run.
• When a new deal is created from a claim event, there is already an older deal with the same (miner_id, piece_cid) and with a resolved payload_cid.

[NikolasHaimerl]

@bajtos is there any way to distinguish between the older and never deal when asking the piece indexer for the payload given a certain combination of (minder_id, piece_cid)?

[NikolasHaimerl]

When a new deal is created from a claim event, there is already an older deal with the same (miner_id, piece_cid) and with a resolved payload_cid.

Do you mean to exclude new claim events that have an already existin (miner_id,piece_cid) combination in the deal-observer database?

When we list all deals missing the payload CID, there will be duplicate (miner_id, piece_cid) pairs. I think this is unlikely to happen in practice after we complete the initial run.

After the initial run, those deals would be flagged with payload_unretrievable. Is this not a sufficient terminal state for these states or do you propose a further action here?