Data removal not working as expected

[wkerckho]

Data removal is implemented based on Clickhouse ALTER TABLE x DELETE queries, which can only delete records after shards have been merged. Because there are no timing guarantees with respect to merges in Clickhouse, Obelisk also cannot guarantee when data be deleted after performing a delete request. This is an issue and a different approach is required!

[wkerckho]

The solution is to use the new DELETE FROM statement, which marks record as deleted immediately, automatically filtering out these records of all subsequent queries.

[MisterTimn]

Should note that this feature is still experimental in clickhouse 22.10, using the new statement for deletes will mark a minimum supported version for clickhouse and further custom config of CH (to allow the experimental lightweight delete).

Makes me think that we should look into a dependency version matrix and form a policy how we handle that with updates of said dependencies.