Acknowledgements¶

First of all I would like to thank Gunnar Elgered for making this project possible. I would also like to thank Lars Petersson and Peter Forkman for supervising the project and providing me with wonderful insights and supporting me at all times. The SALSA system has been a inspirational source throughout this project and I have Eskil Varenius to thank for that. Mikael Lerner has played a fundemental role in integrating the spectrometer in existing systems and it’s now continous usage would not have happend without him. The spectrometer is fully integrated in his software package BIFROST, providing a truly great interface for Radiometers in Sweden and around the world.

Personal background¶

I am a bachelor student at Chalmers University of Technology in Sweden and have been, part time, developing a new spectrometer solution for Radio Astronomy and Aeronomy. I am currently pursuing a degree in Master of Wireless and Space Engineering and eventually a Phd.

The Ettus USRP¶

Ettus Research and the National instruments company provides a broad product line of SDR platforms with e.g different frequency coverage and bandwidths [4]. The Ettus USRP X3x0 series is currently the most high performing platform and would be the most suitable for radio systems with high demands on bandwidth and accuracy. The provided General Purpose Input Output (GPIO) interface in the x3x0 series also makes implementations of Dicke-switching an easier task, and as we shall see is fundamental for the FFTS described here. A general USRP architecture is provided in the figure below, specifics may vary depending on the model [5].

A general Ettus USRP architecture demonstrating the signal chain and processing performed. Based on the Ettus N210 with daughterboard WBX [5].

The first part of the receiver (RX) chain consist of a fundamental Low Noise Amplifier (LNA) with variable attenuation exposed in software e.g GNURadio. The signal is further modulated into inphase and quadrature signals I and Q. Analog and Digital (A/D) converters handles the discretization of the RF providing a bandwidth determined by the daughterboard i.e the front-end of the Ettus USRP [2]. The onboard FPGA down converts the signal by an Digital Down Converter (DDC) implementation providing fine frequency tuning and filters for decimation. The data is subsequently exposed to the host computer through Ethernet or PCIe interfaces.

PCIe interface¶

The PCIe interface provides high data throughput together with low latency, where the latter is a important part for Dicke-switched purposes due to the importance of deterministic timings. However the 10 Gbit ethernet interface provides higher throughput.

Sampling rates¶

The x3x0 series delivers up to 200 M samples/s (I/Q), thus your host computer might have trouble keeping up to speed. Overflow, lost samples, is indicated by a an O in the terminal and is an indication that there is a bottle neck at your host. Several things can be done to address this issue.

First of all is to check that your hardware is up to speed. Check the CPU load, RAM usage and not least the harddrive. A mechanical drive would allow a sample rate up to about 30 Mhz of I/Q sampling depending on the harddrive. The other option would be to use a SSD and it can get you alot higher, however I experienced quite alot of performance issues when running under Linux, with write speeds way below the specifications. I eventually abandoned the SSD and right now I am using Ramdisk, which in theory should have no problem whatsoever for a complex sample rate of 200Mhz (800MB/s).

Other host side improvements that can be done is to expand the socket buffers, setting the governors for your CPU to run at maximum performance and several other parameters. More information and instructions on this can be found here.

If perfomance issues is still a problem go back to the base of your application and check for issues there. There is also another option, which is FPGA programming. Go to this section for more information.

GNU Radio¶

The software interface GNU Radio performs signal processing through the use of dedicated blocks, e.g. infinite impulse response filters (IIR) or FFT calculations, designed in C++. The signal processing blocks can be linked to perform the desired computations in the GNU Radio Companion (GRC) Graphical User Interface (GUI) or directly through the use of a programming language such as Python. GNU Radio provides a vast library of signal processing blocks that can easily be modified to custom specifications. GNU Radio is also published under the GNU General Public Licence (GPL), thus making it free to use and modify.

https://github.com/EttusResearch/uhd
https://github.com/gnuradio/gnuradio

sudo uhd_images_downloader

uhd_image_loader --args="type=x300,resource=RIO0,fpga=HGS"

/usr/local/lib/uhd/utils

./usrp_x3xx_fpga_burner --type=HGS --resource=RIO0

Interfacing with JTAG¶

The Ettus X310 has a JTAG interface that can be accessed using standard USB. This allows interfacing with several utilites such as the Xilinx design tool softwares. This also allows access to the FPGA temperature sensor which, at this time, is only exposed through this interface.

USRP, Computer and Server initialization¶

The USRP device has to be started before the computer in order for the PCIe communication to function properly. After bootup the PCIe drivers for the device has to be turned on. This can be accomplished by the following commands in the terminal:

cd /bin/niusrprio-installer
sudo ./niusrprio_pcie start

If the device needs to be turned off issue:

sudo ./niusrprio_pcie stop

Before turning it off. The radiometer software is located in the home directory under GNURadio-FFTS (Software_HRC for two channel version else Software_COO3) and the socket server is turned on by issuing the following command:

python Server.py

This will also initialize the USRP and the system is now ready. The listed actions as well as several optimization parameters (including setting up RAM-disk) have been summarized in the script USRP_start.sh located in the home folder. Always initiate this script on system reboot.

The computer can either be controlled locally or remotely. For remote control please generate ssh-keys using ssh-keygen and then issue:

ssh-copy-id user@remoteHost

The server can now be accessed through ssh or vnc. The installed vnc server is x11vnc, I recommend using ssvnc for the client which utilize ssh-keys. To install ssvnc issue:

sudo apt-get install tsvnc

then run:

ssvnc

in your terminal window. SSVNC is also available for Windows systems.

Specifications¶

OS Ubuntu 15.10 Desktop

Back-end: Ettus USRP X310 Motherboard + SBX(120MHz) for COO3 system and dual UBX(160MHz) for HRC replacement system

Interface: GNU Radio with UHD version 3.9.1

IP/PORT: Se internal docs

A/D Master Clock Rate = 120 MHz (COO3) 200 MHz (HRC Replacement) (Configurable at host i.e 184.32 MHz, 200 MHz, 120 MHz)

FFT Channels: 8192*2^(-n), where n zero or positive integer.

Integration time CO-O3: Integer multiples of 5 seconds are accepted for integrations <= 30 seconds Above 30 seconds only integer multiples of 30 seconds is accepted. Integrations >= 30 seconds is recommended for switched measurements for optimal processing performance. Processing time is <= 3% of integration time at 120 MHz. Total power measurements have significantly lower processing time <0.1%.

Integration time HRC Replacement: There are no limitations on integration times <= 1000s, above that only integer multiples of 60 seconds accepted (due to RAM-limitations). However you can of course make any integration time possible by creating appropriate loops in your control program, i.e. BIFROST. Processing time may vary depending on whether switched or unswitched measurements are used. Please use the “state?” command to make sure no overlapping occurs between two measurements. Proccessing time guidelines are however less than 3% for switched measurements and negligible for SR=DV=1 measurements.

Note: The integration time of the CO-O3 will be updated to the same specifications as the HRC Replacement on the next maintenance of the system.

Center frequency: Range 400-4400 MHz SBX-120, 10-6000 MHz UBX-160

USRP Gain settings: Range 0-31.5 dB (automatic adjustment recommended)

Clock source: External, 10 MHz square wave recommended

Time source: Ettus USRP Internal (alt external, GPSDO)

Recommended bandwidths [MHz]¶

CO-O3 System:

120  60  30  20  15  12  10  7.5  5  2.5  2  1.5  1  0.7  0.5  0.3  0.1

HRC Replacement:

50  25  20  10  5  2.5  2  1  0.8  0.5  0.2

Listed bandwidths are confirmed for 8k FFT:s. If other bandwidths are desired possibilities exist. Note: Switched measurements have confirmed functionality for bandwidths >= 20 MHz. Other bandwidths are possible, make a request to the GNURadio-FFTS and it will answer with closest possible value. Even decimations of the Master Clock Rate = 200 MHz are however recommended for the best filter characteristics. The upper limit is 50 MHz for the two channel edition GNURadio-FFTS and 120 MHz for COO3 system.

List of commands¶

Communication with the FFTS is handled by a socket server. Invalid commands will return an error message. The following commands are accepted.:

command value
e.g
conf:fft:channels 8192

Control:

meas:init #initilize measurement
meas:adjust #set gain automatically, se code explanation for detailed information
meas:stop #abort measurement
conf:usrp:bw #configure Ettus USRP bandwidth
conf:time:obs #integration time
conf:fft:channels #configure FFT channels
conf:usrp:cfreq #configure center frequency
conf:usrp:gain #manual gain setting

State variables:

state? #state of spectrometer i.e. integrating, adjusting, ready
read:settings? #returns configured settings and general information
conf:usrp:bw? #return configured bandwidth
conf:usrp:gain? #return USRP gain
conf:usrp:cfreq? #return center frequency
conf:fft:channels? #return FFT channels
conf:time:obs? #return set integration time

Read data CO-O3:

meas:read:sig? #read signal spectrum
meas:read:ref? #read reference spectrum
meas:read:sr? #read signal-reference spectrum
meas:read:srr? #read (signal-reference)/reference spectrum
meas:read:hist? #read sample values from latest meas:adjust, plot in a histogram to observe the sample distribution

Read data HRC Replacement: (Replace x with desired channel i.e. 0 or 1)

meas:read:sig_chx? #read signal spectrum
meas:read:ref_chx? #read reference spectrum
meas:read:sr_chx? #read signal-reference spectrum
meas:read:srr_chx? #read (signal-reference)/reference spectrum
meas:read:hist_chx? #read sample values from latest meas:adjust, plot in a histogram to observe the sample distribution

Effective bandwidth and offsets¶

The figure shows an example of a power spectrum at 120 MHz. Observe the filter roll off at band edges, implies that effective bandwidth is less than theoretical set bandwidth. In this case a configured bandwidth of 120 MHz quadrature returns approximately an effective bandwidth of 105 MHz. Ettus lists an effective bandwidth of approximately 80% of the Nyquist bandwidth which seems to be in accordance with the experimental results. E.g a set bandwidth of 20 MHz (+-10MHz from center) will result in an effective bandwidth of approximately 16 MHz (+-8MHz from center). Raw data may contain a center spike due to DC offset in the A/D converters, as shown in the figure. Interpolate the spike to obtain a clean spectrum.

USRP_start.sh¶

This is a optimization and initialization script for the GNURadio-FFTS. Creates RAMDisk for sampled data, optimizes GNURadio by configuring network buffers and starts the PCIE interface as well as the FFTS.

#!/bin/bash
#Title: Ettus USRP optimization and storage init
#Author: Simon Olvhammar

#Create and mount RAMDisk
sudo mkdir /tmp/ramdisk
sudo chmod 777 /tmp/ramdisk
sudo mount -t tmpfs -o size=16384M tmpfs /tmp/ramdisk

# Configure Network Buffers
sudo sysctl -w net.core.rmem_max=33554432
sudo sysctl -w net.core.wmem_max=33554432

# Initalize Ettus USRP PCIe drivers
sudo ~/bin/niusrprio-installer/niusrprio_pcie start

#Message to User
echo "Ettus USRP PCIe drivers activated"
echo "RAMDisk is 8GB in size and located at /tmp/ramdisk"
echo "Starting GNURadio-FFTS..."
#Start FFTS
python /home/olvhammar/GNURadio-FFTS/Software_HRC/Server.py

Server.py¶

This module is a socket server used to communicate with the receiver and basically just handles commands. The code is pretty much self explanatory and can be found at the GITHUB pages under Olvhammar/GNURadio-FFTS.

Measurement.py¶

This is the most important class in the application since this is the base for the Dicke-switching. This class controls the Gnuradio flowgraph, Receiver.py, in order to separate the samples corresponding to signal or reference. Separation of the samples is based upon a switch that is controlled by the state of external SR and DV signals that is transmitted to the GPIO of the Ettus X310. It also contains functions such as automatic gain adjustment. The gain adjustment is based on monitoring the raw samples from the Ettus USRP. The distribution is then checked and gain set accordingly for several loops until the full dynamic range of the A/D converter is covered.

Receiver.py¶

Receiver.py is the Gnuradio flowgraph and is best explained using a gnuradio-companion flowgraph. The flowgraph is abit simplified and the FFTS includes blocks for e.g. sample monitoring, however that would just clutter the graph, but the fundamentals are displayed. The situation gets a bit more complicated for a two channel setup but the principle is similar. Se Software_HRC/Receiver.py for details.

GNU Radio flow graph describing a general purpose FFT Dicke-switched solution for usage with front-ends implementing sky, load or frequency switching.

Basically it computes the Fast Fourier Transform (FFT) from the complex sample stream and saves it to file sinks. A in stream averaging procedure is implemented through a Single Pole IIR filter. The transfer function can be described by:

y(n) = (1-G)y(n-1) + Gx(n)

Which acts as a cheap and convenient way to perform integration. However it does not contain any decimation of it’s own why a keep 1 in N block is introduced. Consider the case of a complex sampling rate 120 MSps and a 8192 channel FFT produces approximately 14.6e3 FFT/s. Setting G=1/20 and N=1/G the output of the keep 1 in N block is instead only 732 FFTs/s. Thus the implemented python long term integrator stress is greatly reduced along with processing times. For switched measurements it is however extremely important to make a good tradeoff in the selection of G due to the introduced delay. Ideally a switched measurement should use very small 1/G values. It is however great for SR=DV=1 measurements where G, with great benefits, can be very small.

The stream selector at the sink side is fundamental for the Dicke-switching implementation. The stream will be switched based on the state of RF input. This state, signal or reference, is controlled by an external switch and it will transmit this information to the GPIO front panel. How this is done in Python can be seen in Measurement.py.

Analyze.py¶

This class stacks all FFT data and performs averaging. The switched data is stacked using the function pool.map to better utilize the cores and lower the processing time:

def stack_all_data(self, files):
        pool = Pool(processes=4)
        spectra = pool.map(self.stack_FFT_file, files)
        pool.terminate()
        return spectra

Where self.stack_FFT_file is defined by:

#According to https://github.com/varenius/salsa/tree/master/USRP/usrp_gnuradio_dev
def stack_FFT_file(self, infile):
        signal = np.memmap(infile, mode = 'r', dtype = np.float32)
        num_spec = int(signal.size/self.fftSize) #The number of spectra contained in the file
        length = num_spec*self.fftSize
        signal = signal[0:length] #Convert the array to an even number of spectras
        spec = signal.reshape((num_spec, self.fftSize)) #Reshape the array with FFT:s so it can be easily stacked
        spec = spec.sum(axis=0) #Stack the FFT:s
        spec = spec/(1.0*num_spec) #Average Spectrum
        del signal
        return spec

Which reads the file containing FFT data and returns one averaged FFT. Finally the mean value of the stacked FFTs is calculated:

def mean(self, spectra):
        sum_spec = np.sum(spectra, axis=0, dtype = np.float32)
        return sum_spec/float(len(spectra))

Finalize.py¶

This class finalizes the measurement and performs e.g. the final averaging and creates FITS-files according to OSO-standards.

Improvements and other toughts¶

Things I would like to improve, any input would be greatly appreciated.

RFNoC¶

RFNoC (RF Network on Chip) is a relatively new concept in GNURadio which allows for, in relation to raw VHDL/Verilog, an easy FPGA implementation of several functions. Such as:

FFT computations up to 2048 channels
Vector IIR averaging filters, extremely useful for FFT outputs
Keep one in N function, combined with the IIR filter allows for decimation of the stream and thus data rates

GNURadio-FFTS RFNoC modification¶

Code have been develop to implement a RFNoC version of the GNURadio-FFTS described here and right now there are mainly two points that prevents a update of the system:

- Number of FFT-channels, where max for RFNoC is in practice only 2048 with a theoretical max of 4096 (Xilinx coregen based). At OSO we require atleast 8192 channels to achieve the desired resolution.
- RFNoC, at this time, only fully supports ethernet communication.

The ethernet equipment is obviously not the biggest problem since we can aquire that but the FFT is trickier. There is basically two things that needs be updated in order for a 8k channel FFT to be implemented in RFNoC. One: the Xilinx coregen based FPGA FFT implementation needs to be updated to 8k channels. And secondly the GNURadio developers needs to add support for fragmentation of vectors in RFNoC which is why only 2048 channel FFT is possible in RFNoC right now. I have however spoken to several of them and an update is expected and it is on their todolist. It should also be noted that RFNoC is yet in it’s alpha stage of development rendering instability as possible problem as well. I expect that within one year a 8k FFT in RFNoC is implemented and thus the GNURadio-FFTS can be implemented on the FPGA of the Ettus X310 instead of the host computer. This is useful for the CO-O3 systems where high bandwidths are desired, for the HRC replacement system the benefits are small.

Ettus USRP X310 Data sheet¶

UBX-160 Daughterboard Data sheet¶