Query - support for dynamic frequency changing at runtime

[jcfitzpatrick12]

Hi Franco,

I am writing a program called SPECTRE, which is a Python-based framework for recording and visualising radio spectrograms. With the help of your repository, I have integrated (so far!) an RSP1A and RSPduo into the program and I am currently using them for solar radio observations.

I am now looking to implement one of the program's critical functionalities: a receiver-agnostic frequency sweep, which will allow the program to capture spectrograms over a large frequency range. However, I cannot find any obvious way to dynamically change the centre frequency of the receivers at runtime. I have noted that message passing may be a route forward. I was hoping for your insight into:

1. Is it feasible to integrate message passing into gr-sdrplay3 source blocks?
2. Could this be used to update the centre frequency at run time?
3. If there is any way to do this currently (with no modifications to the gr-sdrplay3 block code)

I require good performance, in the sense that delay should be minimised between updating the centre frequency in the running flowchart and propagating to the receiver itself (I am looking to tag individual samples with the appropriate center frequency it was collected at). Currently, the program is limited to a fixed centre frequency, with the bandwidth capped by the sample rate of the receiver. So, being able to dynamically update the center frequency of the receivers would be a massive help in improving SPECTRE.

If you need any more information, please let me know.

For your interest, I have included an example image of a solar radio burst captured with SPECTRE and an RSPduo, thanks to gr-sdrplay3 :)

[fventuri]

@jcfitzpatrick12 - Jimmy, congratulations for the excellent work on SPECTRE, and I am happy to hear that the gr-sdrplay3 OOT module is useful in your research.

I had never thought about message ports in my source blocks, however tonight I took a look at a couple of GNU Radio in-tree modules, and I think they are easy to add.

To understand how they work and make sure they would fit your requirements, tonight I implemented a message port called freq in all the source blocks in the gr-sdrplay3 OOT module. The message handler for this port calls the set_center_freq() method to change the center frequency of the RSP.

I just pushed these changes to the new message-passing branch (https://github.com/fventuri/gr-sdrplay3/tree/message-passing); please give it a try and let me know how it goes.

If it works and it is useful, I think I'll add similar message ports for other commonly changed settings, like the IF and RF gains.

Franco

[jcfitzpatrick12]

@fventuri, thank you for the swift reply! This is great news, I will be able to start testing now. I will get back to you once I give the code on your new branch a go. It will take some time to do thorough testing (I need to create a custom sink block to drive the sweep and a custom sink block to save the swept IQ data to file), but I'll get back to you on this thread as soon as I can once I have done some preliminary work.

Jimmy

[jcfitzpatrick12]

@fventuri

I have done some preliminary testing and am keen for your insight. I have created a source block called "Sweep Driver" which outputs floats in steps according to user defined variables: the samples per step, the minimum frequency, the maximum frequency and the frequency step. An example output would be for 2, 10MHz, 30MHz, 3MHz respectively:

10, 10, 13, 13, 16, 16, ..., 28, 28 (then it resets) -> 10, 10

Though the samples per step is larger than that in practice. Every time the block steps upwards in frequency, it publishes a message to the output port with the new frequency. Via the flowchart, I connect this output message port to the input message port which tunes the center frequency of the RSP devices.

I have been experimenting with this and message passing to see how quickly I can update the center frequency of an RSP1A using your recent changes, however, I seem to be hitting some sort of limit with how quickly I can do this.

Outwith gnuradio (and neglecting all hardware + system delays) the physical time between tuning would be dt = (samples per step) * 1/(sample rate). I was hoping for a dt of the order of somewhere between 0.001s -> 0.01s, but in practice, I can only get away with dt ~ 1s which is much too slow for what I need ! If I go below this, the receiver stops collecting data and just crashes gnuradio (sometimes the SDRPlayAPI stops responding and I have to restart my computer).

I was wondering you might know:

1. Is this a gnuradio issue or an SDRPlay API issue?
2. Is there any way to tell outwith gnuradio what the limit to how fast the tuning frequency can be updated programmatically?
3. Are there any potential work arounds?

If message passing was too slow, I wonder if it would be possible to create an input stream into your source blocks, and work with the input stream from sweep driver directly rather than via messaging.

My knowledge of gnuradio runtime scheduling, the inner workings of the message passing API and your source blocks are limited, so any help is greatly appreciated.

I can give you more information if needed, and thank you for any time you have for this :)

Jimmy

[fventuri]

Jimmy,
I suspect you might be hitting the limits of the SDRplay API, which are perhaps dictated by how the internals of the center frequency changes work in the Mirics MSI001 tuner chip.

The reason I think this might be the case is because changing the center frequency is an asynchronous operation in the SDRplay API. First one has to change the center frequency setting (rfHz) and invoke the sdrplay_api_Update() API function, which tells the API you want to change it, but then one has to wait until the RX callback receives a parameter params->rfChanged that is non 0 to know that the center frequency has actually been changed.

I implemented a mechanism to wait for that parameter change in the SoapySDRPlay3 driver ( https://github.com/pothosware/SoapySDRPlay3/blob/master/Streaming.cpp#L94), but it is currently not in this GNU Radio OOT module.

Franco

[fventuri]

I forgot that a couple of years ago I started to implement this 'wait for updates' approach in the 3.9-wait_for_updates branch: https://github.com/fventuri/gr-sdrplay3/blob/3.9-wait_for_updates/lib/rsp_impl.cc#L891-L911

When I have some time I'll have a look at what I did there and I hopefully I'll be able to port it to the main branch.

On the other hand while this might definitely prove that the problem stems from the internals of the SDRplay API, it won't give you a way to address it, since the SDRplay API is proprietary and we can't change the way it works (but understanding the problem might give us some good ideas).

Franco

[jcfitzpatrick12]

Great, thanks for the info! Like you say, it will be good to understand exactly what's causing the problem so to hopefully come up with an alternative. In the meantime, I'll keep fiddling with it and see how I get on. Please let me know if you make any headway !

Jimmy

[jcfitzpatrick12]

Hi Franco,

Interesting developments, I think I've resolved the crashing issue! The flowchart I was using to test the frequency sweep included five blocks: the RSP1A source, the sweep driver source block, a throttle block, and two GUI sinks. In particular, there was some very odd behaviour regarding the throttle block: I noticed that the script completely crashed when I omitted it, and when I did include it, the script exhibited the cut-off behaviour I described before. I understand that throttle blocks should not be used in conjunction with hardware source blocks, so this was really puzzling behaviour.

I resolved the issue by including an input port to the "Sweep Driver" block and connecting it with the output of the RSP1A. It doesn't do anything with the samples internally, but I envisioned that this would bound the Sweep Driver's output with exactly the receiver's output. I can now decrease the samples per step further and tune more rapidly without crashing! Of course, I'll look into this further, but certainly progress!

As usual, thanks for all your help so far.

Jimmy

[jcfitzpatrick12]

Although I shouldn't be so hasty in celebration, there does seem to be a jitter at a certain cut-off point (maybe due to the API issue you were discussing), but at least it's not crashing!

[fventuri]

Jimmy, I am really happy to hear you are making good progress there.
I was out all day so I didn't have time to look at the 'wait for updates' code that I mentioned earlier, but I thought of something that you might find interesting and it might be related to the behavior you are seeing there.

When I was working with the SDRplay API on another SDR program (Linrad), I noticed something about how the SDRplay API and those RX callbacks work. I posted about it on Linrad group on Google groups: https://groups.google.com/g/linrad/c/3ySgbS592EY/m/HHwm1i-4AgAJ
What I found out is that the SDRplay API does "some work" internally (possibly some sort of FFT) on a block of raw samples, and then it sends the samples to the client application via the RX callback in USB data transfer blocks of say 252 samples. That might mean that when one submits a request for changing the center frequency, the SDRplay API won't be able to send samples for the new center frequency until the next FFT block, i.e. possibly up to 96 RX callbacks later.
In some cases changing center frequency may also mean changing the RF band, i.e. going from one RF filter (for instance the one from 0 to 2MHz) to the next RF filter (for instance the one from 2MHz to 12MHz), which perhaps causes some additional delay.

If you want to take a better look at this behavior, you may want to try my little utility program single_tuner_recorder (https://github.com/fventuri/single-tuner-experiments/blob/main/single_tuner_recorder.c), and run it with the -T option where it will show you the time elapsed between successive RX callbacks.

Franco

[jcfitzpatrick12]

No worries at all! I'll look at what you've described above and tinker with the single tuner recorder utility program. I've noticed that the "minimum time interval between tuning before jitter starts happening" is roughly constant for all sampling rates (around 30-50ms), and I'll explore this further too.

[fventuri]

Jimmy,
these days I found some time to add an option called Synchronous Updates to the source blocks in this GNU Radio module gr-sdrplay3. This option is off by default (i.e. asynchronous updates, so the normal behavior of these source blocks stays the same); when Synchronous Updates is set to Synchronous, the updates to sample rate, center frequency, and gains will wait for the notification from the SDRplay API that the value has actually been changed before returning. This way when the call to say set_center_frequency() returns, the client application knows that the change to the center frequency has fully taken effect, and the new I/Q samples are at the new center frequency.

Since this change for the option to enable synchronous updates is orthogonal to the message passing changes (and I plan to eventually merge them to the main branch separately), I first created a new branch called synchronous-updates off the main branch. I then recreated the message-passing branch off the newly created synchronous-updates branch, so that this new version of the message-passing branch contains both set of changes.

I also temporarily added a few debug lines to print the elapsed time (in ns) every time there's an update to one of the parameters of the RSP while streaming. I then ran a few tests changing the center frequency both in asynchronous mode (i.e. the way the blocks in the gr-sdrplay3 module work currently) and in synchronous mode (i.e. waiting for the SDRplay API to set the flag that the requested center frequency change has actually occurred).
I can definitely see the difference in elapsed time: in asynchronous mode these changes take on average 0.5ms, while in synchronous mode they take much longer, from 10ms to more than 38ms.
This is another sign IMHO that the SDRplay API performs some work internally on blocks of I/Q samples; it could be an FFT, or perhaps some block filtering, or something else.

Franco

[jcfitzpatrick12]

Hi Franco, I appreciate the detailed response! Apologies for the delay, I've had a hectic weekend. I am currently doing some testing following your changes and I'll report back in this thread soon :)

[jcfitzpatrick12]

I have been testing your changes by attempting to create a stitched spectrogram over the FM band, sweeping in non-overlapping 8MHz steps from 80MHz to 120MHz. What is interesting, is the latency is still very visible! Broadly speaking, the spectrogram creation is in two steps:

Binary data collection
I have a gnuradio script (pictured below) composed of three blocks: the RSPduo source block, and two custom spectre OOT module blocks - Sweep Driver and Batched File Vector Sink. Essentially, the "Sweep Driver" block ingests samples from the RSPduo, counting each sample that comes in and tagging it with the "current_frequency" variable internally defined in the block. After a user-defined number of samples (the so-called "samples_per_step" variable), the "current_frequency" variable is incremented internally in the block, and at the same time, a message is published to the RSPduo source block to change the center frequency to this new incremented value. The output to the Sweep Driver block is simply a vector of three floats: (current_frequency, real(IQ_sample), im(IQ_sample)). This is then streamed into the Batched File Vector Sink, which (basically) saves this stream to a binary file.

An important assumption in the above setup is that the frequency I am attaching to each sample (i.e. the internal current frequency in sweep driver) is actually representative of the centre frequency that the sample was collected at. It will be seen below that the latency plays a part in throwing this off.

Post-processing
This binary file is then read by my Python post-processing script to create the stitched spectrogram. It boils down to first creating the corresponding base-band spectrogram for the samples in each step. Each step is naturally a fixed length, dictated by the "samples_per_step" variable described above. I have confirmed via a conditional check that indeed each "samples_per_step" sequence of samples is tagged by the correct (ensured unique) frequency.

The base-band spectrograms are then displaced in frequency according to the tagged current frequency for that step, and stacked in a way we pretend they are time coincident over the duration of the sweep:

You can see the latency as follows. Consider the lower left portion of the spectogram:

The very bottom segment is the first step collected by the program (collected at 84MHz, with 8MHz bandwidth). After a set number of samples (in this case, 600000 samples), the current_frequency variable was incremented to 92MHz. This step is directly above in the plot, but as mentioned, this is due to the assumed time-coincidence of the steps in the post-processing. You can see that although the samples have been TAGGED at the new current frequency, in-fact, they are still being produced at the old center frequency, up until a short duration later.

Lowdown
It is clear that my first attempt to rapidly increment the frequency and tag each sample with the appropriate center frequency was affected by this latency. I can live with the latency, but I do need some way to accurately tag each sample such that the tagged frequency is actually representative of the center frequency for that sample! If you have any ideas, please let me know. In the meantime, I'll keep brainstorming.

[fventuri]

@jcfitzpatrick12 - thanks for your comments that help me understand better what you are trying to achieve.

Perhaps the solution to your problem would be by adding 'stream tags' (https://wiki.gnuradio.org/index.php?title=Stream_Tags) to the SDRplay RSP block work function whenever the center frequency (and sample rate and gains) are flagged as changed in the RX callback function.

From an initial look at that RX callback function, I think this might require adding some sort of buffer to keep track of when these changes are signaled besides the regular stream of data, and at this point I don't have a good idea of the best way to do it. I'll need some time to think this through.

Franco

[jcfitzpatrick12]

@fventuri

As always, there's no rush at all. I appreciate the time you're putting into this. I can look into what it would take on my side to adapt my post-processing and blocks to stream tags too :)

[jcfitzpatrick12]

Hi @fventuri

How are you? I haven't had as much time as usual to work on this; how significant life events get in the way! However, I've been making steady progress and should be able to handle stream tags appropriately in post-processing. Specifically, I can manage a variable number of samples at each frequency, which seems inevitable given the asynchronous nature of the messaging API and the behaviour of the SDRPlay API itself.

Tagged Staircase
Firstly, I created a source block called Tagged Staircase which outputs a well-defined (tagged) complex output stream for testing purposes. Essentially, it takes in four values:

• min_samples_per_step (m): The minimum number of samples per step.
• max_samples_per_step (M): The maximum number of samples per step.
• step_increment: The increment by which the number of samples per step increases.
• samp_rate: The (artificially imposed) sampling rate.

The block outputs a sequence of complex numbers where the imaginary part is always zero, and the real part forms a "staircase" pattern according to the user inputs. Each step in the real part has an increasing length as follows:

(1,...,1),(2,...,2),(3,...,3),...,(N,...,N),(1,...,1), and so on

In particular:

• The value "1" is output m times.
• The value "2" is output m + step_increment times.
• The value "3" is output m + 2*step_increment times.
• This continues until the value N is output M times.

Once the maximum step size M is reached, the pattern resets to the initial step size m and the sequence repeats.

Additionally, the block tags the first sample of each step with a frequency tag (rx_freq), which increments by the sampling rate for each new step. This allows testing of my sink block and post-processing on a known tagged output.

Modified Sink
I modified my Batched File Sink source block to save the raw IQ stream to a binary file and (optionally) simultaneously save the tagged stream metadata in a detached header file. The relevant part of the header file looks like:

(rx_freq_0, n_0, rx_freq_1, n_1, ..., rx_freq_N, n_N)

where n_i is the number of samples assigned with frequency "rx_freq_i". This number of samples is inferred from the tagged stream itself at runtime. In brief, for two neighbouring frequency tags in the stream tag_i, and tag_j (i<j), all samples z_k, with k in [i, j), correspond to the frequency described by tag_i.

Post-processing
I have been able to combine the raw data with the metadata in the detached header to construct the swept spectrogram! I can compare the output from the 'Tagged Staircase' to the expected output based on transforming via the STFFT to confirm the post-processing analytically.

I have done some refactoring in my own code to nicely accommodate the detached header, and I am currently neatening up the post-processing for the swept spectrogram.

Going Forward
Any progress on your front? The good news is that if you include a frequency tag whenever the centre frequency has re-tuned, I can handle that well enough, even if the number of samples at each frequency is variable! It should be plug-and-play with the code I've written. Currently, I only have the capability to handle a frequency tag, but if there were extra tags (with timing information, for example), I can include these in the detached header too and handle them appropriately in post-processing.

Jimmy

[fventuri]

@jcfitzpatrick12 - thanks for the update and the great news!
I am really glad to hear that you'll be able to support frequency tags in the new gr-sdrplay3 source blocks.

I started working on them a couple of weeks ago (or perhaps three), but then I got sidetracked with something else (another SDR that uses an FPGA and therefore should be a lot of fun), so I kind of put this project aside.
I am going to be out of town next weekend, but I should be able to resume working on it the weekend after next.

The SDRplay API provides only three pieces of information in the RX callback API:

• changes to the center frequency, as we are discussing here
• changes to the sample rate, which are probably unlikely in your use case
• changes to the gains (both RF gain reduction/LNA state and IF gain reduction), which you might be interested in capturing too

Unfortunately, as far as I know, the RSPs have no 'knowledge' of time, i.e. they just stream I/Q samples at the requested sample rate. They do have a port for an external clock reference, so you might be able to calibrate them with a known reference signal at the beginning of your flowgraph (and perhaps periodically every second or something like that), and be able to figure out the exact time of each sample this way. I am kind of waving hands here, but I hope you get the idea.

Franco

[jcfitzpatrick12]

Sounds cool about the SDR, a very reasonable sidetrack ! Take as much time as you need; there's a lot I can do on my end in the interim. Timing with an external reference is certainly something I'll look into later down the line.

The Sun was very active the last couple of days with some nice broad-band radio bursts. I'm looking forward to when I can get the sweep up and running to see if it works in practice :)

[jcfitzpatrick12]

Hi Franco,

How's it going ! Any movement on this? Now at a bit of a standstill. Is there a possibility it could be done this weekend ?

I was looking into testing my code with the USRP B200mini (https://www.ettus.com/all-products/usrp-b200mini/) which supports message passing and frequency tagging, but it's very pricey 😅

[fventuri]

Jimmy, thanks for the reminder!
I have been working on the other SDR project all this week, and I think I am about to be at a point where I can pause it for a little (you never know when writing software though).

After that, I plan to begin working on adding stream tags to the source blocks - thanks for mentioning the USRP B200mini; I am going to look at the code for that source block to see how they handle stream tags (I am very new to them), so I can figure out how to do it (since the SDRplay API sends the events in the receive callback function, I need to figure out the best way to make them available in the GNU Radio block work() function, which runs on a different thread).

When writing new code I already have problems giving and keep deadlines, let alone for something that I do as a hobby in my free time.

Franco

[jcfitzpatrick12]

Hi Franco,

I appreciate the update ! the time question was bluntly put only so I could figure out out how I'll proceed throughout the week and beyond. You've already done loads so far, the code will come when it comes 🤲

I might have to bite the bullet for the b200mini but it looks quite cool in any case.

Jimmy

[jcfitzpatrick12]

Minor update: after some deep internal reflection I went ahead with the purchase, so if it helps I'll let you know how I get on with the frequency sweep in practice !

Likewise, I appreciate this is a side gig for you, so if all works out with the USRP b200mini, I can even give a go at implementing the frequency tagging in a forked gr-sdrplay3 repo.

[fventuri]

Jimmy, excellent choice!

If you are using GNU Radio in a professional/business context (i.e. not as a hobby like many of us), you will not be disappointed by the USRP SDRs, since they are first class citizens of the GNU Radio ecosystem: their modules are in-tree, and from what I can tell most of the experts in the GNU Radio discussion mailing list either work for Ettus Research/NI or use GNU Radio with a USRP SDR.

On my side, this morning I looked at the GNU Radio code for the B200mini and other similar blocks where they do stream tags, and it doesn't look as bad as I initially thought. I still have to figure out the correct way to keep the different threads in sync and generate those tags with the right sample number, but hopefully I can figure that out too (and soon).

Franco

[jcfitzpatrick12]

Good to hear it's worth it, and it certainly sounds non-trivial with implementing the tags. I'll report back on this thread when I make a bit of progress; the new SDR should be arriving later this week and I've a bit of tinkering to do to integrate it into spectre.

Jimmy

[fventuri]

@jcfitzpatrick12 - I just pushed out a new version of the message-passing branch (https://github.com/fventuri/gr-sdrplay3/tree/message-passing), that contains some initial code for adding a stream tags (https://wiki.gnuradio.org/index.php?title=Stream_Tags) placed at the samples where changes in sample rate, center frequency, or gains occur.

To enable this feature, please go in the Other Options tab in the RSPduo source block, and enable Add Stream Tags.
The new code should compile without any errors, but I haven't done any testing yet.

The tag for changes to the center frequency is called freq (the others are rate and gains), similar to what I saw they use for the USRP B200.

Franco

[jcfitzpatrick12]

Hi Franco,

Brilliant! I will be back late after work today, but I'll take a look when I'm back, do some testing and get back to you.

Thanks,
Jimmy

[jcfitzpatrick12]

@fventuri

I am still doing some testing, but everything is looking really promising! It looks crude at the moment, but I managed to create my first swept spectrum over the FM band:

And the spectrogram at each "step" is being correctly resolved, without any of the issues I was having previously:

I still need to do some amount of further testing (and resolve some bugs in my post processing), but I'll get back on this thread once I've cleaned everything up.

As one notable point, there is an issue when "cmaking" the OOT module in the message passing branch (see the attached text file:

CMAKE_ERROR_MESSAGE.txt

I am currently applying a bandaid fix, by replacing "SDRPLAY_RSPdxR2_ID" with "SDRPLAY_RSPdx_ID" where it is complaining, but I thought to let you know in any case.

I know I've said it many times but thank you so much, I'm really excited that it is working!

Jimmy

[fventuri]

Jimmy, thanks for the kind words and I am really glad to see that your first tests with the FM band look really good!

And thanks for the hint about the USRP B200mini; I have been looking at their code a little bit more, and I see how they pass frequency changes and other commands via a message port. I think their approach is way more flexible than just a port for the frequency the way I do now, so I'll probably change the RSP source blocks to work the same way.
This way it will also be simpler for someone who has designed a GNU Radio Companion flowgraph for one of the SDRplay RSPs to use the same flowgraph for the USRPs with hopefully just minor changes.

Regarding that error message with your build not being able to find 'SDRPLAY_RSPdxR2_ID', I suspect it is because on your computer you have an older version of the SDRplay API, not the current one (3.15): https://www.sdrplay.com/api/
I personally would recommend that you update your computer to the current one even if you don't have an RSPdx-R2 because there are also other bug fixes that you don't have with the older version.

Franco

[jcfitzpatrick12]

@fventuri

Excellent, upgrading the API fixed the issue; thanks for the heads up.

Using your new tags and message port, I've also added a sweep mode to the RSP1A receiver. SPECTRE can now generate spectrograms over a frequency range and time/frequency resolution comparable to the Callisto spectrometer: (https://www.e-callisto.org/), using much cheaper equipment in a (relatively) receiver-agnostic way. Next steps will be to include more SDRplay devices (using gr-sdrplay3) and additionally try and integrate USRP receivers.

For interest, the set-up I am using with your block looks like this:

The Sweep Driver block manages frequency sweeping by counting samples from the receiver. After reaching a defined number of samples, it increments the frequency and sends a message to update the receiver's centre frequency. The tagged data stream from the receiver is then saved by the Batched File Sink, which saves the data in a way we can reconstruct the swept spectrograms.

While there are definitely a few artefacts (when collecting data for a long period of time, occasionally it looks like the data gets mixed up), I'd like to spend some time figuring these out before "reporting a bug" in case it's an issue at my end. There are definitely significant limits to this approach, but for the most part, things are looking very good!

In any case, I'm more than happy to close this issue 🕺

Another FM band spectrogram, this time with the RSP1A:

A (currently) dull-looking spectrogram covering a 45MHz frequency range:

Jimmy

[fventuri]

@jcfitzpatrick12 - very nice results!

It looks like you change frequency every 400k/8M = 50ms, which IIRC is in the same order of magnitude of the latency of the SDRplay API, which means that perhaps some of those frequency changes occur when the previous change is not fully competed, and in that case the behavior of my code and the SDRplay API might be undefined.
Perhaps you could say double the number of samples per step to see if those artifacts go away.

Another option to try could be using zero IF, no decimation, and disabling DC offset correction and I/Q imbalance correction in the source block to see if that helps with the artifacts (you'll have to do some post processing for DC offset and I/Q imbalance in another GNU Radio block downstream).

Franco