This reposistory contains an implementation for a music visualiser, operating on the NXP FRDM KL03 evaluation board equipped with an SSD1331 OLED screen and microphone input to the ADC. On boot up, the hardware will configure itself before dynamically splitting up incoming audio into 6 frequency bins online. These will be shown as an evolving chart on the OLED screen, where each bar will correspond to different frequency depending on the sampling frequency specified in the firmware.
The codebase is built off of the Warp firmware, which this repository is forked off of. This program was severely memory constrained due to the need to store sampled audio values and perform intensive calculations on them, and as such the Warp firmware had to be stripped down signficantly - code for un-needed components on the freedom board was removed, and the more 'investigative' warp capabilities were removed in order to capitalise on their space. Relative to previous courseworks, the following files have been altered or added:
boot.chas been significantly simplifed, with the main loop now repeatedly calling a small set of functions that lay out the steps needed to perform frequency decomposition.fft.chas been added. This contains the simplest implementation of the FFT, which is therefore low memory. Credit goes to https://github.com/brendanashworth/fft-small for this, which was added to this project as allowed under their MIT license. Extra code was added to this in order to deal with the complex output of a real FFT.devADC.chas been added. This contains functions to set up the board's ADC and start taking readings continously. This assumes an analogue audio input is connected to pin PTB1 on the FRDM board.devSSD1331.chas been altered. Code to draw a green square has been removed, and replaced with a number of methods that are utilised to draw a frequency chart on the OLED screen. This includes code to dynamically scale the bars in order to keep then interesting.
The software functionality can be described through the functions called in the main loop:
warpStartcalls all the initialising functions that haven't been cut out. This includes setting up pins, power modes among other things.ADCinitconfigures the ADC with our desired level of accuracy and starts 'continuous conversion' mode. This means repeatedly samples from the microphone such that a new value will be available when we perform a read.devSSD1331initdoes as in previous courseworks - sends the OLED screen all the initial commands needed before graphics commands can be processed.chart_calibrationsets up the data structures needed before frequency charts can be repeatedly drawn. In the chart drawing function, we track the minimum and maximum value seen for each frequency bin such that we can dynamically scale the bars in order to keep them interesting. This calibration function populates the min /max data structures through two initial reads such that there is a baseline to compare against.ADC_read_setpopulates the ADC sample buffer through 16 reads.ffttakes these values and performs a real FFT on them, giving a size 8 complex output.process_powerstakes the magnitude of each of these, but only taking the central bins, as the first and last bins are quite noisy and contain unwanted components.draw_frequency_charttakes these magnitudes and recursively draws scaled bars to represent the amount of each frequency.
Connect the board up as above. The microphone package I have used is: https://www.adafruit.com/product/1063.
Having been built off of Warp, getting the code running is relatively simple. Simply clone the reposistory with git clone https://github.com/Finn-Ashley/Warp-firmware/ and navigate into it with a terminal.
Subsequently call make warp and copy the resulting file at build/ksdk1.1/work/demos/Warp/armgcc/Warp/release/Warp.srec onto your board using the appropriate JLINK tools. This process is extensively detailed below, such that only a brief explanation has been provided here.
Baseline firmware for the Warp family of hardware platforms
This is the firmware for the Warp hardware and its publicly available and unpublished derivatives. This firmware also runs on the Freescale/NXP FRDM KL03 evaluation board which we use for teaching at the University of Cambridge. When running on platforms other than Warp, only the sensors available in the corresponding hardware platform are accessible.
Prerequisites: You need an arm cross-compiler such as arm-none-eabi-gcc installed as well as a working cmake (installed, e.g., via apt-get on Linux or via MacPorts on macOS). On Ubuntu, the package you need is gcc-arm-none-eabi. You will also need an installed copy of the SEGGER JLink commander, JlinkExe, which is available for Linux, macOS, and Windows (here are direct links for downloading it for macOS, and Linux tgz 64-bit).
First, edit setup.conf to set the variable ARMGCC_DIR and JLINKPATH. If your arm-none-eabi-gcc is in /usr/local/bin/arm-none-eabi-gcc, then you want to set ARMGCC_DIR to /usr/local. In the following, this README.md will refer to the top of the repository as $TREEROOT.
Second, edit tools/scripts/glaux.jlink.commands and tools/scripts/warp.jlink.commands to replace <full-path-to-warp-firmware> with the full path to your Warp firmware directory.
Third, build the Warp firmware by
make warp
Fourth, load the Warp firmware to hardware by
make load-warp
To build for the Glaux variant, use make glaux and make load-glaux in steps three and four instead.
The build process copies files from src/boot/ksdk1.1.0/ into the build/, builds, and converts the binary to SREC. See Warp/src/boot/ksdk1.1.0/README.md for more. When editing source, edit the files in src/boot/ksdk1.1.0/, not the files in build location, since the latter are overwritten during each build.
To connect to the running hardware to see output, you will need two terminal windows. In a separate shell window from the one in which you ran make load-warp (or its variants), launch the JLink RTT client See note 1 below:
JLinkRTTClient
The SEGGER firmware allows you to use SEGGER’s JLink software to load your own firmware to the board, even without using their specialized JLink programming cables. You can find the SEGGER firmware at the SEGGER Page for OpenSDA firmware.
To build the Warp firmware for the FRDM KL03, you will need to modify this line in src/boot/ksdk1.1.0/config.h.
The firmware is currently all in src/boot/ksdk1.1.0/, in particular, see src/boot/ksdk1.1.0/warp-kl03-ksdk1.1-boot.c and the per-sensor drivers in src/boot/ksdk1.1.0/dev*.[c,h].
The firmware builds on the Kinetis SDK. You can find more documentation on the Kinetis SDK in the document doc/Kinetis SDK v.1.1 API Reference Manual.pdf.
The firmware is designed for the Warp and Glaux hardware platforms, but will also run on the Freescale FRDM KL03 development board. In that case, the only sensor driver which is relevant is the one for the MMA8451Q. For more details about the structure of the firmware, see src/boot/ksdk1.1.0/README.md.
When the firmware boots, you will be dropped into a menu with a rich set of commands. The Warp boot menu allows you to conduct most of the experiments you will likely need without modifying the firmware:
[ * W a r p (rev. b) * ]
[ Cambridge / Physcomplab ]
Supply=0mV, Default Target Read Register=0x00
I2C=200kb/s, SPI=200kb/s, UART=1kb/s, I2C Pull-Up=32768
SIM->SCGC6=0x20000001 RTC->SR=0x10 RTC->TSR=0x5687132B
MCG_C1=0x42 MCG_C2=0x00 MCG_S=0x06
MCG_SC=0x00 MCG_MC=0x00 OSC_CR=0x00
SMC_PMPROT=0x22 SMC_PMCTRL=0x40 SCB->SCR=0x00
PMC_REGSC=0x00 SIM_SCGC4=0xF0000030 RTC->TPR=0xEE9
0s in RTC Handler to-date, 0 Pmgr Errors
Select:
- 'a': set default sensor.
- 'b': set I2C baud rate.
- 'c': set SPI baud rate.
- 'd': set UART baud rate.
- 'e': set default register address.
- 'f': write byte to sensor.
- 'g': set default SSSUPPLY.
- 'h': powerdown command to all sensors.
- 'i': set pull-up enable value.
- 'j': repeat read reg 0x00 on sensor #3.
- 'k': sleep until reset.
- 'l': send repeated byte on I2C.
- 'm': send repeated byte on SPI.
- 'n': enable SSSUPPLY.
- 'o': disable SSSUPPLY.
- 'p': switch to VLPR mode.
- 'r': switch to RUN mode.
- 's': power up all sensors.
- 't': dump processor state.
- 'u': set I2C address.
- 'x': disable SWD and spin for 10 secs.
- 'z': dump all sensors data.
Enter selection>
By default on Unix, you will likely see characters you enter shown twice. To avoid this, do the following:
- Make sure you are running
bash(and notcsh) - Execute
stty -echoat the command line in the terminal window in which you will run theJLinkRTTClient.
You can probe around the menu to figure out what to do. In brief, you will likely want:
-
Menu item
bto set the I2C baud rate. -
Menu item
rto switch the processor from low-power mode (2MHz) to "run" mode (48MHz). -
Menu item
gto set sensor supply voltage. -
Menu item
nto turn on the voltage regulators. -
Menu item
zto repeatedly read from all the sensors whose drivers are compiled into the build.
NOTE: In many cases, the menu expects you to type a fixed number of characters (e.g., 0000 or 0009 for zero and nine) See note 1 below. If using the JLinkRTTClient, the menu interface eats your characters as you type them, and you should not hit RETURN after typing in text. On the other hand, if using telnet you have to hit return.
If you see repeated characters, you can set your terminal to not echo typed characters using stty -echo.
b(set the I2C baud rate to0300for 300 kb/s).g(set sensor supply voltage to3000for 3000mV sensor supply voltage).n(turn on the sensor supply regulators).j(submenu for initiating a fixed number of repeated reads from a sensor):
Enter selection> j
Auto-increment from base address 0x01? ['0' | '1']> 0
Chunk reads per address (e.g., '1')> 1
Chatty? ['0' | '1']> 1
Inter-operation spin delay in milliseconds (e.g., '0000')> 0000
Repetitions per address (e.g., '0000')> 0000
Maximum voltage for adaptive supply (e.g., '0000')> 2500
Reference byte for comparisons (e.g., '3e')> 00
This will perpetually stream data from the 90+ sensor dimensions at a rate of about 90-tuples per second. Use the following command sequence:
b(set the I2C baud rate to0300for 300 kb/s).r(enable 48MHz "run" mode for the processor).g(set sensor supply voltage to3000for 3000mV sensor supply voltage).n(turn on the sensor supply regulators).z(start to stream data from all sensors that can run at the chosen voltage and baud rate).
From your local clone:
git remote add upstream https://github.com/physical-computation/Warp-firmware.git
git fetch upstream
git pull upstream master
Phillip Stanley-Marbell and Martin Rinard. “A Hardware Platform for Efficient Multi-Modal Sensing with Adaptive Approximation”. ArXiv e-prints (2018). arXiv:1804.09241.
BibTeX:
@ARTICLE{1804.09241,
author = {Stanley-Marbell, Phillip and Rinard, Martin},
title = {A Hardware Platform for Efficient Multi-Modal
Sensing with Adaptive Approximation},
journal = {ArXiv e-prints},
archivePrefix = {arXiv},
eprint = {1804.09241},
year = 2018,
}
Phillip Stanley-Marbell and Martin Rinard. “Warp: A Hardware Platform for Efficient Multi-Modal Sensing with Adaptive Approximation”. IEEE Micro, Volume 40 , Issue 1 , Jan.-Feb. 2020.
BibTeX:
@ARTICLE{8959350,
author = {P. {Stanley-Marbell} and M. {Rinard}},
title = {Warp: A Hardware Platform for Efficient Multi-Modal
Sensing with Adaptive Approximation},
journal = {IEEE Micro},
year = {2020},
volume = {40},
number = {1},
pages = {57-66},
ISSN = {1937-4143},
month = {Jan},
}
This research is supported by an Alan Turing Institute award TU/B/000096 under EPSRC grant EP/N510129/1, by Royal Society grant RG170136, and by EPSRC grants EP/P001246/1 and EP/R022534/1.
1 On some Unix platforms, the JLinkRTTClient has a double echo of characters you type in. You can prevent this by configuring your terminal program to not echo the characters you type. To achieve this on bash, use stty -echo from the terminal. Alternatively, rather than using the JLinkRTTClient, you can use a telnet program: telnet localhost 19021. This avoids the JLink RTT Client's "double echo" behavior but you will then need a carriage return (↵) for your input to be sent to the board. Also see Python SEGGER RTT library from Square, Inc. (thanks to Thomas Garry for the pointer).