RSPduo "Dual Tuner (diversity receiption)" Large Data Set Coherent Processing

[gozillah]

I am reading data from gr-sdrplay3 RSPduo in "Dual Tuner (diversity receiption)" mode for large data set correlation processing (typically 2^20 samples per batch or even more). To maintain both processing batch size as well as correlation my processing needs vector input (vector size 2^20 samples or even more). So far I have been using two GRC "Stream to Vector" blocks to convert RSPduo's stream outputs into my processing's vector inputs. This is working quite well with a correlation S/N of rougly 30 dB at 2^20 samples per batch (or less dB with lower batch sizes), measured while feeding both tuners from the same antenna.

Collecting and processing 2^20 samples is creating several seconds of latency down my processing chain. In order to reduce latency introduced by the GRC "Stream to Vector" blocks I integrated the "Stream to Vector" conversion into the first block of my GRC processing chain (written in Python, sorry), but so far without success. Since doing this I have completely lost correlation. My (unsuccessful) approach is as follows:

• Instead of a sync block (as when using the GRC "Stream to Vector" blocks) I am using a basic block for my processing.
• "input_items[]" stream input from the RSPduo seems to be available in chunks of much less than 2^20 samples, typically variable between about 2^8 and 2^13 maximum.
• So in the general_work() function I created two buffers into which I am accumulating stream input samples of the respective inputs until they reach my processing batch (= former vector) size of 2^20. I am doing this in a Python while-loop where I am reading (per loop) all currently available input from one input and then all available input form the other input. After each step I am calling self.consume(which_input, how_many_items) to signal GRC the consumed samples.
• I then present these buffers to my block's correlation processing.

I suspect that I am loosing coherence in the abovementioned while-loop. Could it be that my Python implementation is so slow that it misses some samples on one input while processing the other? What could be a successful strategy to accumulate the stream samples without loosing coherence?

[gozillah]

After hours of trial and error I finally got the point:

• gr-sdrplay RSPduo is streaming to my block chunks of up to about 2^13 samples per input, but often in smaller chunks. Each such chunk invokes my block whereupon I will have to accumulate the chunks of the two inputs into two static buffers until their individual contents will reach the desired vector size. While accumulating across several invocations of my block, its general_work() function will have to return zero (i.e. my block does not produce any vector processing or output yet).
• Only after both buffers will have accumulated the desired number of samples (e.g. 2^20 each) constituting the desired vectors' sizes I will invoke my block's vector signal processing and produce vector output to the next block in my processing chain. I also reset the buffers in preparation for the next round.

To make the two buffers and their indexes static they are implemented in the init() function:

        self.buffer = np.zeros((2,self.vector_size), dtype=np.complex64)
        self.index  = np.zeros(2, dtype=int)

The essential input processing in the general_work() function is:

        if self.index[0] < self.vector_size:
            len0 = len(input_items[0])
            if len0 > 0:
                maxlen0 = min(len0, self.vector_size - self.index[0])
                self.buffer[0][self.index[0]:self.index[0]+maxlen0] = input_items[0][0:maxlen0]
                self.consume(0, maxlen0)
                self.index[0] += maxlen0
            
        if self.index[1] < self.vector_size:
            len1 = len(input_items[1])
            if len1 > 0:
                maxlen1 = min(len1, self.vector_size - self.index[1])
                self.buffer[1][self.index[1]:self.index[1]+maxlen1] = input_items[1][0:maxlen1]
                self.consume(1, maxlen1)
                self.index[1] += maxlen1
               
        if self.index[0] < self.vector_size or self.index[1] < self.vector_size:
            return 0

Followed by my block's signal processing and output and ending in:

        self.buffer = np.zeros((2,self.sample_size), dtype=np.complex64)
        self.index  = np.zeros(2, dtype=int)
        return len(output_items[0])

You may close this issue or even delete it. Or you may leave it for others having the same problem. In any case it is not an issue of your gr-sdrplay3 driver but rather an issue of understanding how GNU Radio wants stream to vector processing to be implemented.

[fventuri]

@gozillah - I am glad to hear you figured it out.

A good resource for questions related to GNU Radio is their discussion mailing list: https://wiki.gnuradio.org/index.php/MailingLists - there are a lot of experts there.
I also know they have a live chat on Matrix, where you can ask for help, but I have never used it.

Franco

[gozillah]

Thanks for the link. I may register there and migrate my observations to the discussion forum.

I am somewhat worried about losing samples when the GNU Radio scheduler or Windows are running the thread of my block in intervals so large that the RSPduo's sampling output will overfill a buffer in your gr-sdrplay3 driver or somewhere in GNU Radio between your gr-sdrplay3 driver and my block. Taking statistics about the "stream chunks" delivered to my block's "Stream to Vector" processing I am observing the following behavior:

...
pagesize :debug: Setting pagesize to 4096 B
top_block_impl :debug: Using default scheduler "TPB"
Stream chunk size = 8191 , min so far = 8191 , max so far =  8191 , 1 times
Stream chunk size = 4096 , min so far = 4096 , max so far =  8191 , 1 times
Stream chunk size = 4095 , min so far = 4095 , max so far =  8191 , 1 times
Stream chunk size = 3777 , min so far = 3777 , max so far =  8191 , 1 times
Stream chunk size = 4414 , min so far = 3777 , max so far =  8191 , 1 times
Stream chunk size = 2638 , min so far = 2638 , max so far =  8191 , 1 times
Stream chunk size = 5553 , min so far = 2638 , max so far =  8191 , 1 times
Stream chunk size = 2744 , min so far = 2638 , max so far =  8191 , 1 times
Stream chunk size = 1764 , min so far = 1764 , max so far =  8191 , 1 times
Stream chunk size = 3276 , min so far = 1764 , max so far =  8191 , 1 times
Stream chunk size = 1260 , min so far = 1260 , max so far =  8191 , 1 times
Stream chunk size = 3024 , min so far = 1260 , max so far =  8191 , 1 times

...

The maximum chunk size seems to be 8191 samples. A current stream chunk size of 8191 samples raises the suspicion that samples may have been lost due to buffer overrun, spoiling the coherence of my vector data.

Is this limit of approximately 8k (= 2^13) a limitation in your gr-sdrplay3 driver or possibly in GNU Radio?

[gozillah]

Note: The above list contains only newly occurring chunks sizes, i.e. a listet chunk size may occur many times.

[fventuri]

@gozillah - you may want to enable 'Sample gaps check' in the RSPduo source block under 'Other Options'.
With that option enabled, the source block will print a warning message every time there's a gap in the sample count (i.e. samples are lost): https://github.com/fventuri/gr-sdrplay3/blob/main/lib/rsp_impl.cc#L1068-L1071

If you see those messages with your GNU Radio Companion flowgraph, it means that the work() function in the source block is not called often enough to transfer all the samples. In that case you may want to either reduce the sample rate (possibly via decimation) or make changes to the blocks downstream to alleviate the backpressure on the source block (or both).

Franco

[gozillah]

My processing cycle is as follows:

• A sampling period of say one second where I collect all samples of both gr-sdrplay3 output streams coherently into my first block. It is important that I do not lose samples here.
• Followed by a processing period of maybe another second or more down my processing chain during which I may (and certainly will) lose all samples collected by gr-sdrplay3 from RSPduo and streamed to my temporarily deaf first block. This is no problem.
• Repeat

When activating the 'Sample gaps check' I see a lot of missed samples warnings, but I am unable to specifically attribute them to either my sampling or my processing phase. So for the time being I cannot tell whether I am actually losing samples during my sampling phase. My processing output does not look bad, but my observation is that correllation S/N is varying from cycle to cycle by up to about +/- 5dB. I do not know whether this is due to loss of samples or antenna signal variations.

Decimation would greatly reduce processing quality. Translating all of my blocks from Python to C++ would improve downstream processing speed by maybe a factor of two at the cost of a lot of work (I do most of the maths with numpy, so the gain may be even less than two).

According to my Windows' Task manager, gr-sdrplay3 and the blocks of my application are running in a single process as separate threads. Running the workload of my blocks in separate processes on several CPUs may speed them up (I could start these processes from within the block's thread and communicate with them via shared memory), but this seems complex and I have no practical experience in this.

Assuming that gr-sdrplay3 has a mechanism warranting not to miss any samples from the RSPduo, and futher assuming that GNU Radio warrants that a vector output reliaby transfers the entire vector to the next block irrespective of GNU Radio or Windows scheduling, then the following concept may assure that no samples are lost during my sampling phase:

• gr-sdrplay3 has a vector output(s) option (i.e. gr-sdrplay3 output is properties-selectable either as streams or vectors) with configurable vector size.
• When a vector is full but no downstream block has been waiting to get it, then gr-sdrplay3 discards the vector(s). In Diversity Mode this would have to be done for both vectors in parallel to maintain coherence.
• When a downstream block has been waiting for the vector(s) then gr-sdrplay3 delivers it (them). During this delivery gr-sdrplay3 may discard or ignore new samples because the downstream block will require some processing time anyway. gr-sdrplay3 will resume collection of samples into vector(s) only after completion of the transfer.

But as I said above, I am not sure whether I am regularly losing samples during my sampling period.

Uff, this has now been TL;DR ;-).

[fventuri]

@gozillah - thanks for the detailed and interesting explanation.

A couple of things you can try to make sure this RSPduo source block doesn't drop samples under normal conditions:

1. create a simple GNU Radio Companion flowgraph with just the source block and a null sink, and enable the Sample gaps check option. In theory in this flowgraph you should see no dropped samples. If I remember correctly, under Windows you'll samples dropped at the very beginning, but once the the flowgraph has been running for a few seconds, you shouldn't see any more dropped samples. I don't see that happening under Linux, so it might be something related to the way the GNU Radio scheduler runs under Windows (to be honest, I know very little about Windows)
2. after that, try with another very simple flowgraph with the source block, then the GNU Radio block that converts from stream to vector, and finally the null sink block. See how things go in this case.

Then, little by little, you can add other blocks downstream (one by one so you can see the differences) to see how it goes in terms of dropped samples.

Some of the questions you are asking me require an in-depth knowledge of how GNU Radio and its scheduler work, and I honestly don't have that level of expertise. If I were you, I would post a message with what you are trying to do to the GNU Radio discuss mailing list to see what they suggest.

Franco

[gozillah]

Just to repeat it once in a while: I very much appreciate your responsiveness and help!

I created that RSPduo+Nothingelse+Nullsink graph, and yes, under Windows it initially drops samples:

pagesize :debug: Setting pagesize to 4096 B
top_block_impl :debug: Using default scheduler "TPB"
rspduo :warning: sample num gap in stream 0: 8535 [1092:9627] -> 33+219
rspduo :warning: sample num gap in stream 1: 8535 [1092:9627] -> 33+219
rspduo :warning: sample num gap in stream 0: 1 [9879:9880] -> 0+1
rspduo :warning: sample num gap in stream 1: 1 [9879:9880] -> 0+1
rspduo :warning: sample num gap in stream 0: 9616 [58012:67628] -> 38+40
rspduo :warning: sample num gap in stream 1: 9616 [58012:67628] -> 38+40
rspduo :warning: sample num gap in stream 0: 1 [67880:67881] -> 0+1
rspduo :warning: sample num gap in stream 1: 1 [67880:67881] -> 0+1
rspduo :warning: sample num gap in stream 0: 3 [68889:68892] -> 0+3
rspduo :warning: sample num gap in stream 1: 3 [68889:68892] -> 0+3
rspduo :warning: sample num gap in stream 0: 9886 [83760:93646] -> 39+58
rspduo :warning: sample num gap in stream 1: 9886 [83760:93646] -> 39+58
rspduo :warning: sample num gap in stream 0: 9870 [109774:119644] -> 39+42
rspduo :warning: sample num gap in stream 1: 9870 [109774:119644] -> 39+42
rspduo :warning: sample num gap in stream 0: 9858 [135772:145630] -> 39+30
rspduo :warning: sample num gap in stream 1: 9858 [135772:145630] -> 39+30
rspduo :warning: sample num gap in stream 0: 1 [145882:145883] -> 0+1
rspduo :warning: sample num gap in stream 1: 1 [145882:145883] -> 0+1
rspduo :warning: sample num gap in stream 0: 18 [147395:147413] -> 0+18
rspduo :warning: sample num gap in stream 1: 18 [147395:147413] -> 0+18
rspduo :warning: sample num gap in stream 0: 9853 [161777:171630] -> 39+25
rspduo :warning: sample num gap in stream 1: 9853 [161777:171630] -> 39+25
rspduo :warning: sample num gap in stream 0: 1 [171882:171883] -> 0+1
rspduo :warning: sample num gap in stream 1: 1 [171882:171883] -> 0+1

But then the scheduler kicks the Null sink in, consuming them all.

I also inserted my first block converting the streams into vectors and doing initial processing of my chain. This block still is consuming all samples. But as I add futher blocks the dropouts take off.

I haven't found a comprehensive documentation of GNU Radio's scheduler. Most sources just say that it is complex, and please do not change anything unless you know what you are doing. And I know nothing ;-). There seems to be a project called GNU Radio 4.0 comprising a total redesign of the scheduler. As the scheduler is very close to the OS kernel, I hope that it will be available with all its new goodies for Windows as well. Under Windows the 3.x scheduler is lacking some kernel-related functionality as e.g. thread priority link .

By analyzing the gaps between gap warnings I found out that (due to processing chain overload) I actually lose samples in my stream to vector conversion, so the assumed problem is existing indeed.

I initially rejected your proposal to decimate the streams because decimation would degrade processing quality as a side-effect. However skipping whole vectors would be acceptable, because I do need no-gaps sampling only within a vector. At that time I was assuming that this could only be achieved by doing the stream to vector conversion in your gr-sdrplay3, see my comment of August 15 above. But then I discovered the "Stream to Vec Decim" block which seems to have been designed for exactly this purpose.

And indeed, my processing results have improved! Problem solved!
(Unless I missed something ;-).

[fventuri]

@gozillah - glad to hear you got it working with decimation!
With GNU Radio being modular and the number of GNU Radio modules available either in-tree or out-of-tree, I found out that there's always a module that does what you need.

A couple of comments:

• for the GNU Radio scheduler there is an old presentation by Tom Rondeau that you can find here: https://static.squarespace.com/static/543ae9afe4b0c3b808d72acd/543aee1fe4b09162d0863397/543aee20e4b09162d0863578/1380223973117/gr_scheduler_overview.pdf There is also this paper by Bastian Bloessl and others with more information about its performance: https://www.bastibl.net/bib/bloessl2019benchmarking/bloessl2019benchmarking.pdf If you still have questions, you may want to ask them in the GNU Radio discussion mailing list
• the new GNU Radio 4 project is very interesting. I watched Dr Steinhagen's presentation at GRCon23: https://www.youtube.com/watch?v=Dxein1FJBHM and it is definitely very cool stuff. The week after next they'll show a lot more about it at the European GNU Radio Days '24 (https://events.gnuradio.org/event/23/timetable/?view=standard), but I am not sure if it'll be available on YouTube

Franco

[gozillah]

Thanks for the links. There is also a good scheduler presentation from Marcus Muller at link. GNU Radio 4 will be exciting indeed.

Problem solved!

Well, sort of. It is now about ten times better than before I applied the stream to vector decimator. But I am still losing some samples. My setup is as follows:

I optimized my processing a bit by applying Numba at some points in my code. So far my attempts to migrate processing via CUDA to the GPU have failed, but that is due to my current lack of CUDA knowledge.

What happens is that at the decimator's output I am receiving bursts of gaps followed by series of flawless vector data. It looks like so (I am printing the dynamic range of my processing after each vector);

...
Dynamic range 37 dB
Dynamic range 40 dB
Dynamic range 36 dB
Dynamic range 38 dB
Dynamic range 39 dB
Dynamic range 38 dB
Dynamic range 39 dB
rspduo ⚠️ sample num gap in stream 0: 15694 [1021913744:1021929438] -> 62+70
rspduo ⚠️ sample num gap in stream 1: 15694 [1021913744:1021929438] -> 62+70
rspduo ⚠️ sample num gap in stream 0: 3 [1021929690:1021929693] -> 0+3
rspduo ⚠️ sample num gap in stream 1: 3 [1021929690:1021929693] -> 0+3
rspduo ⚠️ sample num gap in stream 0: 9610 [1022041833:1022051443] -> 38+34
rspduo ⚠️ sample num gap in stream 1: 9610 [1022041833:1022051443] -> 38+34
rspduo ⚠️ sample num gap in stream 0: 4 [1022051695:1022051699] -> 0+4
rspduo ⚠️ sample num gap in stream 1: 4 [1022051695:1022051699] -> 0+4
Dynamic range 39 dB
Dynamic range 41 dB
Dynamic range 40 dB
rspduo ⚠️ sample num gap in stream 0: 31672 [1037252087:1037283759] -> 125+172
rspduo ⚠️ sample num gap in stream 1: 31672 [1037252087:1037283759] -> 125+172
rspduo ⚠️ sample num gap in stream 0: 9737 [1037380023:1037389760] -> 38+161
rspduo ⚠️ sample num gap in stream 1: 9737 [1037380023:1037389760] -> 38+161
rspduo ⚠️ sample num gap in stream 0: 1 [1037390012:1037390013] -> 0+1
rspduo ⚠️ sample num gap in stream 1: 1 [1037390012:1037390013] -> 0+1
rspduo ⚠️ sample num gap in stream 0: 9872 [1037405889:1037415761] -> 39+44
rspduo ⚠️ sample num gap in stream 1: 9872 [1037405889:1037415761] -> 39+44
rspduo ⚠️ sample num gap in stream 0: 4 [1037416013:1037416017] -> 0+4
rspduo ⚠️ sample num gap in stream 1: 4 [1037416013:1037416017] -> 0+4
Dynamic range 37 dB
Dynamic range 39 dB
Dynamic range 39 dB
Dynamic range 42 dB
Dynamic range 36 dB
Dynamic range 38 dB
Dynamic range 40 dB
Dynamic range 38 dB
Dynamic range 38 dB
Dynamic range 39 dB
Dynamic range 40 dB
Dynamic range 38 dB
Dynamic range 39 dB
Dynamic range 38 dB
Dynamic range 41 dB
Dynamic range 37 dB
Dynamic range 38 dB
Dynamic range 37 dB
Dynamic range 38 dB
rspduo ⚠️ sample num gap in stream 0: 31686 [1137066141:1137097827] -> 125+186
rspduo ⚠️ sample num gap in stream 1: 31686 [1137066141:1137097827] -> 125+186
rspduo ⚠️ sample num gap in stream 0: 1 [1137098079:1137098080] -> 0+1
rspduo ⚠️ sample num gap in stream 1: 1 [1137098079:1137098080] -> 0+1
rspduo ⚠️ sample num gap in stream 0: 9737 [1137194092:1137203829] -> 38+161
rspduo ⚠️ sample num gap in stream 1: 9737 [1137194092:1137203829] -> 38+161
rspduo ⚠️ sample num gap in stream 0: 1 [1137204081:1137204082] -> 0+1
rspduo ⚠️ sample num gap in stream 1: 1 [1137204081:1137204082] -> 0+1
rspduo ⚠️ sample num gap in stream 0: 9872 [1137219958:1137229830] -> 39+44
rspduo ⚠️ sample num gap in stream 1: 9872 [1137219958:1137229830] -> 39+44
rspduo ⚠️ sample num gap in stream 0: 15872 [1137245958:1137261830] -> 62+248
rspduo ⚠️ sample num gap in stream 1: 15872 [1137245958:1137261830] -> 62+248
rspduo ⚠️ sample num gap in stream 0: 1 [1137262082:1137262083] -> 0+1
rspduo ⚠️ sample num gap in stream 1: 1 [1137262082:1137262083] -> 0+1
rspduo ⚠️ sample num gap in stream 0: 9609 [1137374223:1137383832] -> 38+33
rspduo ⚠️ sample num gap in stream 1: 9609 [1137374223:1137383832] -> 38+33
rspduo ⚠️ sample num gap in stream 0: 2 [1137384084:1137384086] -> 0+2
rspduo ⚠️ sample num gap in stream 1: 2 [1137384084:1137384086] -> 0+2
Dynamic range 38 dB
Dynamic range 40 dB
....

Taking this gap as an example:
rspduo ⚠️ sample num gap in stream 0: 9609 [1137374223:1137383832] -> 38+33
What is the meaning of -> 38+33 ?

I have no idea whether the gaps are occurring in the stream data for the vectors I am actually receiving from the decimator or in the decimated data disposed off by the decimator.

The gaps explode whenever I am increasing the decimation rate beyond about 0.4 . Reducing the decimation on the fly via QT GUI Range back to 0.4 or below will not recover the situation. I will have to restart the flowgraph.

On the other hand the processing results of my application are not looking really bad as long as I am limiting decimation to about 0.4 .

I read in other GNU Radio posts in the Internet that other people are having problems with sample gaps as well. I suspect that the GNU Radio buffering and the single task / multiple threads concept of the scheduler are simply overstrained by my large vectors of between one and four megasamples. I can parallelize parts of my code with Numba but not the crucial buffering and scheduling. Is there any possibility to increase the GNU Radio buffer sizes?

[fventuri]

@gozillah 38+33 means 38*252 + 33, where 252 is the size of the SDRplay API RX callback vectors xi and xq.
In other words 9,609 samples are equal to 38 full RX callback vectors + 33 samples. I put it there as a quick way to see if the number of dropped samples is an exact multiple of the RX callback size, which could perhaps indicate a different type of problem.

Regarding the GNU Radio buffer sizes, I think you should ask that question to the GNU Radio discuss mailing list (https://www.gnuradio.org/community/) to see what those more familiar with GNU radio internals think about it.

Also thanks for the link to the presentation by Mark Muller on YouTube; I'll watch it tonight after work.

Franco

[gozillah]

Are these samples being lost between the SDRduo and your gr-sdrplay3 source block or between your source block and the first non-source block? I unsuccessfully tried to find your code implementing the gap analysis.

[fventuri]

@gozillah - the code that checks for gaps in the sample streams is here: https://github.com/fventuri/gr-sdrplay3/blob/main/lib/rsp_impl.cc#L1057-L1074 and it is invoked by the two RX callback functions here: https://github.com/fventuri/gr-sdrplay3/blob/main/lib/rsp_impl.cc#L709-L710 and here: https://github.com/fventuri/gr-sdrplay3/blob/main/lib/rsp_impl.cc#L730-L731

The samples being "lost" is simply because the GNU Radio scheduler doesn't call the work() function of the source block often enough, and therefore the RX callback functions are stuck waiting for the ring buffer condition variable overflow to indicate that there's room available in the ring buffer to put more samples: https://github.com/fventuri/gr-sdrplay3/blob/main/lib/rsp_impl.cc#L762-L764
The reason why the GNU Radio scheduler is not calling the work() function often enough is typically because one or more downstream blocks are still working processing samples. As we discussed a few days ago, if you replace the downstream blocks with a simple null sink, the problem goes away because the GNU Radio scheduler can now call the work() function as often as needed.

Franco

[gozillah]

In rsp_impl.cc's work() function is a line
int nsamples = ring_buffer.head - ring_buffer.tail;

If ring_buffer.tail is larger than ring_buffer.head (because ring_buffer.head has wrapped around the ring_buffer's linear top), then nsamples becomes negative, with the consequence that at
noutput_items = std::min(nsamples, noutput_items);
noutput_items becomes negative as well.

Then new_tail at
uint64_t new_tail = ring_buffer.tail + noutput_items;
runs backwards into already processed samples.

This does not happen very often as RingBufferSize is large compared to GNU Radio's usual noutput_items in the work() function's call parameters. But I will happen from time to time.

Or did I overlook something?

[gozillah]

Maybe I overlooked this part preceeding above code:
ring_buffer.empty.wait(lock, [&ring_buffer]() { return ring_buffer.tail < ring_buffer.head; });
It is cryptic to me as I am not deeply proficient in C++. Could you please explain its workings with respect to the ring_buffer?

[fventuri]

@gozillah - I am out of town these days, so I won't be able to take a look at the issue with negative values for nsamples until I get home.

Regarding the use of that condition variable (ring_buffer.empty), I followed some of the examples on how to use condition variables; for instance here: https://www.geeksforgeeks.org/cpp-multithreading-condition-variables/ (if I find a better example, I'll send you the link).

Franco

[gozillah]

Ok thanks, take your time. I will study the link.

I am trying to modify your code for my own purposes while still struggling with refreshing and rounding my C++ knowledge. For this it is of course essential for me to grasp what your code is doing. I succeeded compiling and installing your code with small test modifications in my Radioconda / Windows environment.

My objective is to make your code output large vectors (instead of streams) containing contiguous samples without gaps while maintaining coherence between the two cannels of SDRplay duo. It is ok if the downstream processing chain is losing samples between verctors due to ithe chain's inability to process the samples as fast as SDRplay is delivering them. But the vectors polled by the downsteam processing chain must be without gaps inside the vectors.

As a beneficial side-effect this will reduce latency by sidestepping downstream "stream to vector" blocks.

My understanding of your code is that the SRDplay API is pushing samples into the ring buffers via the callback rsp_impl::event_callback().

• Question 1: Is this callback being invoked independently of the GNU Radio scheduling? So I can be sure that the ring buffers contain contiguous samples, wherat old samples are automatically being overwritten after the ring buffers have become full?

On the other side of the ring buffers rsp_impl::work() is invoked by the GNU Radio scheduler and is transfering samples from the ring buffers to this source block's outputs. rsp_impl::work() will miss the samples overwritten by rsp_impl::event_callback() when the ring buffers become full (as detailed above).

• Question 2: Am I right with my desctription of rsp_impl::work()?

My approach is as follows:

• In the .yml file the outputs will be defined as vectors of length vec_len, and vec_len will be a new parameter.
• RingBufferSize will be dynamically allocated according to vec_len (as a power of two), typically one, two or four megasamples.
• rsp_impl::event_callback() will continue to run as-is.

I will have to modify rsp_impl::work():

• Due to the outputs having been defined as vectors the GNU Radio scheduler will call rsp_impl::work() with noutput_items = vec_len.
• If the ring buffers are locked because rsp_impl::event_callback() is processing a 252 samples frame then rsp_impl::work() will wait. This seems to have been implemented already.
• If the ring buffers are not full (i.e. do not contain vec_len samples) then rsp_impl::work() will return zero. The downstream processing chain will have to wait another GNU Radio scheduler cycle.
• If the ring buffers are full, then rsp_impl::work() will output vec_len samples as vectors and empty the ring buffers.
• rsp_impl::work() will then successfully supply another vector only after the ring buffer has been refilled by rsp_impl::event_callback(). RingBufferSize could be defined as a multiple of vec_len to speed things up in case the downstream processing chain is processing vectors efficiently, but this would complicate the cases to be handled in rsp_impl::work(). May be later.

Question 3: Is there anything probelmatic in my approach?

[fventuri]

• I found the link to the messages I posted a while ago about using condition variables (protected by a mutex) to manage the synchronization of the ring buffers between the RX callback and the work() function in the GNU Radio source: Setting LNA gain sometimes fails pothosware/SoapySDRPlay3#58 (comment) - even if that message refers to a different project, I though you might find useful the links and the discussion in that thread
• the samples from the SDRplay API are copied to the ring buffer in the stream_callback() function (the event_callback() function is only called for infrequent events like gain changes, ADC overloads, device failures, etc)
• finally, I think that you are barking at the wrong tree with your proposed changes. As we already discussed in the past, removing the downstream blocks or lowering the sample rate solves the problem with the sample gaps. In other words, if the work() function is called frequently enough by the GNU Radio scheduler, then the problem goes away. This is a problem of backpressure, i.e. the blocks downstream are not processing those samples fast enough, which means that no matter how big you make those buffers in between, eventually they will become full, and you'll start having gaps in the sample sequence. It is like trying to drain a tub being filled up by a firehose using a smaller pipe - no matter how big that tub is, eventually it will fill up.

Franco