[core] Introduce bitmap index record reader #4502
Conversation
| new FileRecordReader( | ||
| bulkFormatMapping.getReaderFactory(), | ||
| formatReaderContext, | ||
| bulkFormatMapping.getIndexMapping(), | ||
| bulkFormatMapping.getCastMapping(), | ||
| PartitionUtils.create(bulkFormatMapping.getPartitionPair(), partition)); | ||
|
|
||
| if (fileIndexResult instanceof BitmapIndexResult) { |
There was a problem hiding this comment.
Here should be bug, bitmap in ApplyBitmapIndexRecordReader should be merged with ApplyDeletionVectorReader, otherwise the latter will encounter problems.
Can you create a test for this?
There was a problem hiding this comment.
Could you be more specific for the bug?
As I understand it
The bitmap index reader would filter all the mismatched data
The DV reader would filter all the deleted data
Is there a conflict?
Can this test case be covered?
org.apache.paimon.table.PrimaryKeyFileStoreTableTest#testDeletionVectorsWithFileIndexInMeta
org.apache.paimon.table.PrimaryKeyFileStoreTableTest#testDeletionVectorsWithBitmapFileIndexInFile
There was a problem hiding this comment.
You are right, the magic is in the returned position, it makes chain processing possible.
Purpose
Introduction of bitmap index record reader to filter data in advance with bitmap index result.
Tests
org.apache.paimon.table.AppendOnlyFileStoreTableTest#testBSIAndBitmapIndexInMemory
org.apache.paimon.table.AppendOnlyFileStoreTableTest#testBSIAndBitmapIndexInDisk
org.apache.paimon.table.PrimaryKeyFileStoreTableTest#testDeletionVectorsWithFileIndexInMeta
org.apache.paimon.table.PrimaryKeyFileStoreTableTest#testDeletionVectorsWithBitmapFileIndexInFile
API and Format
No.
Documentation
No.