feat: check for missing payload

[NikolasHaimerl]

This PR adds the following changes:

1. Add a cache for payloads that could not be retrieved
2. Retrieve payloads up to 4 times with a time delay of 8 hours in between calls
3. Mark deals as unretrievable if all retries fail

See also this issue.

[NikolasHaimerl]

3. IIUC your design, on every loop iteration, you fetch all deals missing PayloadCID from the database. That will eventually mean fetching tens of millions of records.

I miscalculated the number of deals here. IIUC, this PR will mark deals as not advertised after ~1 day. fil-deal-ingester is adding 40-60k daily these days. Around 33% of deals are advertised to IPNI, according to what I see in the Spark Internal Dashboard. That means around 77% * 40k = ~30k deals to recheck after the service (re)starts.

I apologise for exaggerating the number in my initial comment.

I still prefer to keep our service as stateless as possible. It's a principle we have been following in other services, too. Please correct me if I am wrong, @juliangruber.

I created an implementation plan which makes the service stateless in exchange for an additional field in the database to keep track of when the last time was when we tried to fetch a payload. Please let me know what you think.

[pyropy]

Good job, I think we're almost there.

[juliangruber]

Suggested change

	payload_retrievability_state,
	payload_retrievability,

(everything stored in Postgres is a state)

If you want to annotate the type, maybe instead this?

Suggested change

	payload_retrievability_state,
	payload_retrievability_enum,

I suggest we wait for @bajtos with this one

[juliangruber]

Suggested change

	CREATE TYPE PAYLOAD_RETRIEVABILITY_STATE AS ENUM ('PAYLOAD_CID_NOT_QUERIED_YET', 'PAYLOAD_CID_UNRESOLVED', 'PAYLOAD_CID_RESOLVED','PAYLOAD_CID_TERMINALLY_UNRETRIEVABLE');
	CREATE TYPE PAYLOAD_RETRIEVABILITY_STATE AS ENUM ('PAYLOAD_CID_RESOLUTION_NOT_ATTEMPTED_YET', 'PAYLOAD_CID_UNRESOLVED', 'PAYLOAD_CID_RESOLVED','PAYLOAD_CID_TERMINALLY_UNRETRIEVABLE');

[juliangruber]

So as not to introduce another concept of "querying"

[juliangruber]

Suggested change

	const PayloadRetrievabilityState = {
	NotQueried: 'PAYLOAD_CID_NOT_QUERIED_YET',
	Unresolved: 'PAYLOAD_CID_UNRESOLVED',
	Resolved: 'PAYLOAD_CID_RESOLVED',
	TerminallyUnretrievable: 'PAYLOAD_CID_TERMINALLY_UNRETRIEVABLE'
	}
	const PayloadRetrievabilityStateType = Type.Enum(PayloadRetrievabilityState)
	const PayloadRetrievabilityStates = {
	NotQueried: 'PAYLOAD_CID_NOT_QUERIED_YET',
	Unresolved: 'PAYLOAD_CID_UNRESOLVED',
	Resolved: 'PAYLOAD_CID_RESOLVED',
	TerminallyUnretrievable: 'PAYLOAD_CID_TERMINALLY_UNRETRIEVABLE'
	}
	const PayloadRetrievabilityState = Type.Enum(PayloadRetrievabilityStates)

Does this make more sense? Payload Retrievability States lists all possible states. Payload Retrievability State is one of these states at a time.

We might want to wait for @bajtos with this one

[juliangruber]

This is so nice to read :)

[juliangruber]

Suggested change

	if (deal.last_payload_retrieval_attempt) {
	if (deal.payload_retrievability_state === PayloadRetrievabilityState.Unresolved) {

[juliangruber]

@bajtos I'm merging now so that we can hopefully get more payloads assigned over the weekend, and thereby improve Spark DDO data

[sentry-io]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ Error: Connection terminated unexpectedly loadDeals(deal-observer) View Issue
• ‼️ Error: Error updating payload of deal: { updatePayloadCidInActiveDeal(look-up-payload-cids) View Issue

_{Did you find this useful? React with a 👍 or 👎}