PrefetchTail ignores max_sectors_kb

[mdcallag]

Expected behavior

While debugging performance with issue 12038 I added printfs to PosixRandomAccessFile::Prefetch to see what the prefetch sizes are. Almost all of them are slightly larger than the adjusted value of compaction_readahead_size because the block size is added to the compaction_readahead_size limit. However, this issue isn't open for that.

This issue is open because PrefetchTail calls Prefetch and it doesn't adjust the prefetch size based on max_sectors_kb. AFAIK, doing large requests to readahead isn't useful. And in a quick debugging session I see readahead calls for up to 500kb when PrefetchTail calls Prefetch and max_sectors_kb = 128.

An example stack trace is:

Thread 2 "rocksdb:low" hit Breakpoint 1, rocksdb::PosixRandomAccessFile::Prefetch (this=0x7ffff58170c0, offset=67109985, n=405299) at env/io_posix.cc:828
828                                              IODebugContext* /*dbg*/) {
(gdb) where
#0  rocksdb::PosixRandomAccessFile::Prefetch (this=0x7ffff58170c0, offset=67109985, n=405299) at env/io_posix.cc:828
#1  0x0000555555a07e05 in rocksdb::RandomAccessFileReader::Prefetch (n=<optimized out>, offset=<optimized out>, opts=..., this=0x7ffff586a1c0) at ./file/random_access_file_reader.h:179
#2  rocksdb::BlockBasedTable::PrefetchTail (ro=..., file=0x7ffff586a1c0, file_size=<optimized out>, force_direct_prefetch=<optimized out>, tail_prefetch_stats=<optimized out>, prefetch_all=true, preload_all=true, prefetch_buffer=0x7ffff6df9ee0, stats=0x0, tail_size=405299,
    logger=0x7ffff7262190) at table/block_based/block_based_table_reader.cc:941
#3  0x0000555555a18f3c in rocksdb::BlockBasedTable::Open (read_options=..., ioptions=..., env_options=..., table_options=..., internal_comparator=..., file=..., file_size=67515284, block_protection_bytes_per_key=0 '\000', table_reader=0x7ffff6dfa7a8, tail_size=405299,
    table_reader_cache_res_mgr=std::shared_ptr<rocksdb::CacheReservationManager> (empty) = {...}, prefix_extractor=std::shared_ptr<const rocksdb::SliceTransform> (use count 8, weak count 0) = {...}, prefetch_index_and_filter_in_cache=true, skip_filters=false, level=1,
    immortal_table=false, largest_seqno=0, force_direct_prefetch=false, tail_prefetch_stats=0x7ffff7262320, block_cache_tracer=0x7ffff7370710, max_file_size_for_l0_meta_pin=100663296, cur_db_session_id="2T3HKP7ZGTQD63EPTZ7L", cur_file_num=26, expected_unique_id=...,
    user_defined_timestamps_persisted=true) at table/block_based/block_based_table_reader.cc:663
#4  0x00005555559f7430 in rocksdb::BlockBasedTableFactory::NewTableReader (this=this@entry=0x7ffff7224140, ro=..., table_reader_options=..., file=..., file_size=file_size@entry=67515284, table_reader=0x7ffff6dfa7a8, prefetch_index_and_filter_in_cache=true)
    at table/block_based/block_based_table_factory.cc:568
#5  0x000055555588151d in rocksdb::TableCache::GetTableReader (this=<optimized out>, ro=..., file_options=..., internal_comparator=..., file_meta=..., sequential_mode=false, file_read_hist=0x7ffff7371fc8, table_reader=0x7ffff6dfa7a8, mutable_cf_options=..., skip_filters=false, level=1,
    prefetch_index_and_filter_in_cache=true, max_file_size_for_l0_meta_pin=100663296, file_temperature=<optimized out>) at db/table_cache.cc:148
#6  0x000055555588229f in rocksdb::TableCache::FindTable (this=0x7ffff722b320, ro=..., file_options=..., internal_comparator=..., file_meta=..., handle=0x7ffff6dfa928, mutable_cf_options=..., no_io=<optimized out>, file_read_hist=0x7ffff7371fc8, skip_filters=false, level=1,
    prefetch_index_and_filter_in_cache=true, max_file_size_for_l0_meta_pin=100663296, file_temperature=rocksdb::Temperature::kUnknown) at db/table_cache.cc:200
#7  0x0000555555882bdc in rocksdb::TableCache::NewIterator (this=this@entry=0x7ffff722b320, options=..., file_options=..., icomparator=..., file_meta=..., range_del_agg=range_del_agg@entry=0x0, mutable_cf_options=..., table_reader_ptr=0x0, file_read_hist=0x7ffff7371fc8,
    caller=rocksdb::kCompactionRefill, arena=0x0, skip_filters=false, level=1, max_file_size_for_l0_meta_pin=100663296, smallest_compaction_key=0x0, largest_compaction_key=0x0, allow_unprepared_value=false, read_seqno=0x0, range_del_iter=0x0) at db/table_cache.cc:245
#8  0x0000555555bd8462 in operator() (__closure=0x7ffff6dfad50, output_status=...) at db/compaction/compaction_job.cc:729
#9  0x0000555555bdfc24 in rocksdb::CompactionJob::Run (this=this@entry=0x7ffff6dfbce0) at db/compaction/compaction_job.cc:776
#10 0x000055555577b61f in rocksdb::DBImpl::BackgroundCompaction (this=0x7ffff7370000, made_progress=0x7ffff6dfc347, job_context=0x7ffff6dfc3b0, log_buffer=0x7ffff6dfc620, prepicked_compaction=<optimized out>, thread_pri=rocksdb::Env::LOW) at db/db_impl/db_impl_compaction_flush.cc:3954
#11 0x000055555577d913 in rocksdb::DBImpl::BackgroundCallCompaction (this=0x7ffff7370000, prepicked_compaction=0x0, bg_thread_pri=rocksdb::Env::LOW) at db/db_impl/db_impl_compaction_flush.cc:3427
#12 0x000055555577e234 in rocksdb::DBImpl::BGWorkCompaction (arg=<optimized out>) at db/db_impl/db_impl_compaction_flush.cc:3100
#13 0x0000555555ae2b5e in std::function<void ()>::operator()() const (this=0x7ffff6dfd0e0) at /usr/include/c++/11/bits/std_function.h:590
#14 rocksdb::ThreadPoolImpl::Impl::BGThread (this=0x7ffff7234200, thread_id=0) at util/threadpool_imp.cc:311
#15 0x0000555555ae30f3 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7ffff728a100) at util/threadpool_imp.cc:352
#16 0x00007ffff7a3b253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#17 0x00007ffff76c3ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#18 0x00007ffff7755850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81