RHCLOUD-37430 | fix: integrations are not being returned sometimes

[MikelAlejoBR]

We were getting reports of integrations not showing up in stage when Kessel Relations was enabled. More specifically, the issue was that after trying to filter integrations by name, some of them did simply not show up in the results even though they were recently created.

After investigating the issue, it turned out that the problem was that we were getting an incomplete set of authorized integration IDs from Kessel, which in turn made us filter the database query results incorrectly, which again, in turn, made us return an inaccurate list of integrations for an organization.

The QE organization which reported the issue had around 2,500 integrations created, and when a request was made to look up for a very specific integration with name "X", Notifications first had to ask Kessel for the principal's authorized integrations. Since Kessel only returned a batch of a 1,000 we would use just those to apply a "integration ID in" filter in the query, so if the specified name in the request was in that batch, the user was lucky and the integration was returned.

These changes introduce a mechanism that keeps firing requests to Kessel whenever there is a continuation token available. That makes sure that we are fetching all the integrations that the user has access to, so that we don't miss any when querying our database.

Jira ticket

[RHCLOUD-37430]

[MikelAlejoBR]

These are the default values when launching the Kessel's Inventory API with the authentication enabled: https://github.com/project-kessel/inventory-api/blob/e0fc00b00af6af4977f8b4c66230a34f850da4a2/scripts/get-token.sh

In order to simplify things, I think it's just better to leave the credentials like this by default. It eases development and debugging, and since the credentials are public facing, I think it is just safe to have them here.

[MikelAlejoBR]

Even though it is not used right now, I thought that we might want to change our approach in the future depending on the usage we make of the Kessel integration. That way we'll have access to both "pre-filtering" and "post-filtering" methods in case we want to get creative.

[g-duval]

Great work!

[g-duval]

Suggested change

	String continuationToken;
	String newContinuationToken = "";
	do {
	// Replace the old continuation token with the new one. This way
	// the token gets used
	continuationToken = newContinuationToken;
	// Build the lookup request for Kessel.
	final LookupResourcesRequest request = this.buildLookupResourcesRequest(identity, integrationPermission, continuationToken);
	Log.tracef("[identity: %s][permission: %s][resource_type: %s] Payload for the resource lookup: %s", identity, integrationPermission, ResourceType.INTEGRATION, request);
	// Make the request to Kessel.
	final Iterator<LookupResourcesResponse> responses;
	try {
	responses = this.lookupClient.lookupResources(request);
	} catch (final Exception e) {
	Log.errorf(
	e,
	"[identity: %s][permission: %s][resource_type: %s] Runtime error when querying Kessel for integration resources with request payload: %s",
	identity, integrationPermission, ResourceType.INTEGRATION, request
	);
	// Increment the errors counter and stop the timer in case of
	// an error.
	this.meterRegistry.counter(KESSEL_METRICS_LOOKUP_RESOURCES_COUNTER_NAME, Tags.of(COUNTER_TAG_REQUEST_RESULT, COUNTER_TAG_FAILURES)).increment();
	lookupTimer.stop(this.meterRegistry.timer(KESSEL_METRICS_LOOKUP_RESOURCES_TIMER_NAME, Tags.of(KESSEL_METRICS_TAG_PERMISSION_KEY, integrationPermission.getKesselPermissionName(), Constants.KESSEL_METRICS_TAG_RESOURCE_TYPE_KEY, ResourceType.INTEGRATION.name())));
	throw e;
	}
	// Iterate over the incoming results.
	while (responses.hasNext()) {
	final LookupResourcesResponse response = responses.next();
	Log.tracef("[identity: %s][permission: %s][resource_type: %s] Received payload for the resource lookup: %s", identity, integrationPermission, ResourceType.INTEGRATION, response);
	uuids.add(UUID.fromString(response.getResource().getId()));
	// Update the continuation token every time, to make sure we
	// grab the last streamed element's continuation token.
	newContinuationToken = response.getPagination().getContinuationToken();
	}
	String continuationToken;
	do {
	// Build the lookup request for Kessel.
	final LookupResourcesRequest request = this.buildLookupResourcesRequest(identity, integrationPermission, continuationToken);
	Log.tracef("[identity: %s][permission: %s][resource_type: %s] Payload for the resource lookup: %s", identity, integrationPermission, ResourceType.INTEGRATION, request);
	// Make the request to Kessel.
	final Iterator<LookupResourcesResponse> responses;
	try {
	responses = this.lookupClient.lookupResources(request);
	} catch (final Exception e) {
	Log.errorf(
	e,
	"[identity: %s][permission: %s][resource_type: %s] Runtime error when querying Kessel for integration resources with request payload: %s",
	identity, integrationPermission, ResourceType.INTEGRATION, request
	);
	// Increment the errors counter and stop the timer in case of
	// an error.
	this.meterRegistry.counter(KESSEL_METRICS_LOOKUP_RESOURCES_COUNTER_NAME, Tags.of(COUNTER_TAG_REQUEST_RESULT, COUNTER_TAG_FAILURES)).increment();
	lookupTimer.stop(this.meterRegistry.timer(KESSEL_METRICS_LOOKUP_RESOURCES_TIMER_NAME, Tags.of(KESSEL_METRICS_TAG_PERMISSION_KEY, integrationPermission.getKesselPermissionName(), Constants.KESSEL_METRICS_TAG_RESOURCE_TYPE_KEY, ResourceType.INTEGRATION.name())));
	throw e;
	}
	// Iterate over the incoming results.
	while (responses.hasNext()) {
	final LookupResourcesResponse response = responses.next();
	Log.tracef("[identity: %s][permission: %s][resource_type: %s] Received payload for the resource lookup: %s", identity, integrationPermission, ResourceType.INTEGRATION, response);
	uuids.add(UUID.fromString(response.getResource().getId()));
	// Update the continuation token every time, to make sure we
	// grab the last streamed element's continuation token.
	continuationToken = response.getPagination().getContinuationToken();
	}

Can't we be more direct by removing newContinuationToken?

[g-duval]

Shouldn't we align this limit with the current Kessel's limitation of 1000?
BTW even if it should never happend, requesting 2147483647 results at a time could be little bit risky :)