Audiokit: Record to RAM @ 96k sample-rate _while_ playing back at resampled rate?

[z-l-p]

Hi. I've done several projects with arduino-audio-tools and it surprises me all the time. Simply wonderful!
I'm building a device to sonify ultrasound with an audiokit, by sampling at stereo 96k and playing back (via ResampleStream) at a much lower rate. (Similar to a "time expansion" bat detector.)
I modified the streams_audiokit_ram_audiokit example in 2 ways:

1. Set up the audiokit at 96k SR and resample the output SR to a selectable factor of the input SR
2. Start playing the buffer (slowly) while recording it, so you hear "slow motion" immediately as you press key 1. It continues to loop after recording. This is where I'm confused. In the original example, the streams are re-arranged at runtime by calling copy.begin() and specifying the source and sink (triggered by the edge of the record button). I'm using 2 copiers instead, and disabling them in the loop based on the rec/play conditions. I'm sure this is wrong, but it works!

The problem: I get choppy playback during the "full duplex" moment of recording (but it sounds OK during playback). If I reduce the SR to around 88000 then the choppiness is gone, but Ideally I'd like to adapt this code to an I2S ADC with 192k SR for better detail in the ultrasound. Is there something obvious that I'm missing? Many thanks for all you've done already!
(This sketch is easy to test: Upload to an audiokit. Press key1 and jingle your keys over the audiokit mics to hear bit-crushed church bells in your headphones at 16x "slo-mo"!)

/**
 * ultrasound to audible sound resampling test, based on streams_audiokit_ram_audiokit.ino by Phil Schatzmann
 * Hold Key 1 to sample ultrasound and immediately hear it at slower (audible) speed. Playback will loop when button is released.
 * Remember to enable PSRAM in Arduino IDE Tools menu
 * Requires ESP32 Arduino core 3.x (tested on 3.2.0)
 */
 
#include "AudioTools.h"
#include "AudioTools/AudioLibs/AudioBoardStream.h"
#include "AudioTools/AudioLibs/MemoryManager.h"

/// USER VARS ///
int recSampleRate = 96000; // input sampling rate
int speedFactors[] = {16, 8, 4, 2, 1, 2, 4, 8}; // available speed factors to use for playback (1/speedFactor)
int sfIndex = 0; // select factor from array to use as startup default

/// INTERNAL VARS ///
bool nowRecording = 0; // are we filling the buffer?
bool nowPlaying = 0; // is there something in the buffer to play?

AudioInfo info(recSampleRate, 2, 16); // for ADC
AudioInfo out_info(recSampleRate / speedFactors[sfIndex],2,16); // for DAC
AudioBoardStream kit(AudioKitEs8388V1);
MemoryManager memory(500); // Activate SPI RAM for objects > 500 bytes
DynamicMemoryStream recording(true); // Audio stored on heap [args: loop (bool), buffer-size (int)]
ResampleStream resample(recording);
StreamCopy copier1(recording, kit); // copies data
StreamCopy copier2(kit, resample); // copies data 
auto rcfg = resample.defaultConfig();


 void change_sr(bool pinStatus, int pin, void* ref) {
  sfIndex++;
  if (sfIndex > sizeof(speedFactors) / sizeof(speedFactors[0]) - 1) {
    sfIndex = 0;
  }
  Serial.print("Speed: 1/");
  Serial.println(speedFactors[sfIndex]);
  out_info.sample_rate = recSampleRate / speedFactors[sfIndex]; // update output sample-rate
  rcfg.copyFrom(out_info);
  rcfg.step_size = 1.0 / speedFactors[sfIndex]; // update resampler
  resample.end(); // restart resampler for changes to take effect
  resample.begin(rcfg);
}

void record_start(bool pinStatus, int pin, void* ref){
 
  Serial.print("Recording... (sample-rate: ");
  Serial.print(info.sample_rate);
  Serial.println(" kHz)");
  Serial.print("Playing... (sample-rate: ");
  Serial.print(out_info.sample_rate);
  Serial.print(" kHz, speed: 1/");
  Serial.print(speedFactors[sfIndex]);
  Serial.println(")");
  recording.end();
  recording.begin();
  copier1.begin();  // start recording
  copier2.begin(); // start playback
  nowRecording = 1;
  nowPlaying = 1;

}

void record_end(bool pinStatus, int pin, void* ref){
  Serial.println("Recording stopped (playback continues) ");
  nowRecording = 0;
}

void setup(){
  Serial.begin(115200);
  while(!Serial); // wait for serial to be ready
  AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Warning);  

  // setup input and output
  auto cfg = kit.defaultConfig(RXTX_MODE);
  cfg.sd_active = false;
  cfg.copyFrom(info);
  cfg.input_device = ADC_INPUT_LINE2;
  kit.begin(cfg);
  kit.setInputVolume(1.0); // input
  kit.setVolume(0.4); // output

  // define resampling info
  //auto rcfg = resample.defaultConfig(); // now global
  rcfg.copyFrom(out_info);
  rcfg.copyFrom(info);
  rcfg.step_size = 1.0 / speedFactors[sfIndex];
  resample.begin(rcfg);
  

  // record when key 1 is pressed
  kit.audioActions().add(kit.getKey(1), record_start, record_end);
  // chang playback sample-rate when key 6 is pressed
  kit.audioActions().add(kit.getKey(6), change_sr);
  Serial.println("Press Key 1 to record.");
  Serial.println("Press Key 6 to change playback speed.");

}

void loop(){

  // record and play recording (only if a recording has started, otherwise lib throws memory allocation error)
  if (nowRecording) {
    copier1.copy();  
  }
  if (nowPlaying) {
    copier2.copy();  
  }

  //t Process keys
  kit.processActions();

}

[pschatzmann]

I think the following happens: If you play back at a lower speed (while still recording) the copier2.copy(); is taking more time then the copier1.copy() requires data so you run into an underflow.

Solution appraches:

• Don't play back while recording
• Put the copier1.copy(); or copier2.copy(); into a separate task, so that they don't impact each other
• Reduce the copy size of the copier2 by the resampling factor to allign the timing with the copier1

[z-l-p]

I decided to use tasks for higher performance, since I hope to increase the sample-rate later. This is my first experience with multi-core processing, so I had trouble with the ESP32 watchdog resetting. I think that streamcopy.copy is safe to put in a task, but since my copiers are conditional, it seemed to lock up the task and trigger the wdt when the condition wasn't met?
This fixed the watchdog problem, and the system works great now, except for occasional crashes when the record cycle is initiated ( Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled. )
Any ideas for optimization? Current code:

/**
 * ultrasound to audible sound resampling test, based on streams_audiokit_ram_audiokit.ino by Phil Schatzmann
 * Hold Key 1 to sample ultrasound and immediately hear it at slower (audible) speed. Playback will loop when button is released.
 * Remember to enable PSRAM in Arduino IDE Tools menu
 * Requires ESP32 Arduino core 3.x (tested on 3.2.0)
 */
 
#include "AudioTools.h"
#include "AudioTools/AudioLibs/AudioBoardStream.h"
#include "AudioTools/AudioLibs/MemoryManager.h"
#include "AudioTools/Concurrency/RTOS.h"

/// USER VARS ///
int recSampleRate = 96000; // input sampling rate
int speedFactors[] = {16, 8, 4, 2, 1, 2, 4, 8}; // available speed factors to use for playback (1/speedFactor)
int sfIndex = 0; // select factor from array to use as startup default

/// INTERNAL VARS ///
bool nowRecording = 0; // are we filling the buffer?
bool nowPlaying = 0; // is there something in the buffer to play?

AudioInfo info(recSampleRate, 2, 16); // for ADC
AudioInfo out_info(recSampleRate / speedFactors[sfIndex],2,16); // for DAC
AudioBoardStream kit(AudioKitEs8388V1);
MemoryManager memory(500); // Activate SPI RAM for objects > 500 bytes
DynamicMemoryStream recording(true); // Audio stored on heap [args: loop (bool), buffer-size (int)]
ResampleStream resample(recording);
StreamCopy copier1(recording, kit); // copies data
StreamCopy copier2(kit, resample); // copies data 
auto rcfg = resample.defaultConfig(); // set up the resampler config

Task writeTask("rec-copy", 3000, 10, 0);
Task readTask("play-copy", 3000, 10, 1);

void rec_function() {
     if (nowRecording) {
      copier1.copy();  
    } else {
      delay(1);
    }
  }

void play_function() {
    if (nowPlaying) {
     copier2.copy();  
    } else {
      delay(1);
    }
  }

 void change_sr(bool pinStatus, int pin, void* ref) {
  sfIndex++;
  if (sfIndex > sizeof(speedFactors) / sizeof(speedFactors[0]) - 1) {
    sfIndex = 0;
  }
  Serial.print("Speed: 1/");
  Serial.println(speedFactors[sfIndex]);
  out_info.sample_rate = recSampleRate / speedFactors[sfIndex]; // update output sample-rate
  rcfg.copyFrom(out_info);
  rcfg.step_size = 1.0 / speedFactors[sfIndex]; // update resampler
  resample.end(); // restart resampler for changes to take effect
  resample.begin(rcfg);
}

void record_start(bool pinStatus, int pin, void* ref){
 
  Serial.print("Recording... (sample-rate: ");
  Serial.print(info.sample_rate);
  Serial.println(" kHz)");
  Serial.print("Playing... (sample-rate: ");
  Serial.print(out_info.sample_rate);
  Serial.print(" kHz, speed: 1/");
  Serial.print(speedFactors[sfIndex]);
  Serial.println(")");
  recording.end();
  recording.begin();
  copier1.begin();  // start recording
  copier2.begin(); // start playback
  nowRecording = 1;
  nowPlaying = 1;

}

void record_end(bool pinStatus, int pin, void* ref){
  Serial.println("Recording stopped (playback continues) ");
  nowRecording = 0;
}

void setup(){
  Serial.begin(115200);
  while(!Serial); // wait for serial to be ready
  AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Warning);  

  // setup input and output
  auto cfg = kit.defaultConfig(RXTX_MODE);
  cfg.sd_active = false;
  cfg.copyFrom(info);
  cfg.input_device = ADC_INPUT_LINE2;
  kit.begin(cfg);
  kit.setInputVolume(1.0); // input
  kit.setVolume(0.4); // output

  // define resampling info
  //auto rcfg = resample.defaultConfig(); // now global
  rcfg.copyFrom(out_info);
  rcfg.copyFrom(info);
  rcfg.step_size = 1.0 / speedFactors[sfIndex];
  resample.begin(rcfg);
  

  // record when key 1 is pressed
  kit.audioActions().add(kit.getKey(1), record_start, record_end);
  // chang playback sample-rate when key 6 is pressed
  kit.audioActions().add(kit.getKey(6), change_sr);
  Serial.println("Press Key 1 to record.");
  Serial.println("Press Key 6 to change playback speed.");

 // Start audio tasks
  writeTask.begin(rec_function);
  readTask.begin(play_function);

}

void loop(){
  // Process keys
  kit.processActions();

}

[z-l-p]

One more thing, the examples on the multicore wiki page https://github.com/pschatzmann/arduino-audio-tools/wiki/Multicore-Processing use this include:
#include "AudioTools/Concurrency/All.h"
but I think it should be:
#include "AudioTools/Concurrency/RTOS.h" ?

[pschatzmann]

Yes, if you don't add any delay, the task will never yield and you will run into a watchdog exception.
I think you should also synchronize the access to the DynamicMemoryStream e.g. with a LockGuard

[z-l-p]

Thanks for you patience and support. I checked the LockGuard class docs and studied the occurrences in the repo but I'm not yet confident to "connect the dots". I understand a mutex conceptually, and how it protects a resource that is shared between threads. I think LockGuard makes mutexes easier / safer (using RAII) because it removes the need to explicitly acquire and release the lock in each function. (Sorry, no formal CS training here, so I'm learning by doing!) So is this the right idea?

#include "AudioTools/Concurrency/RTOS.h"

Create a global mutex to use as the lock:
mutex myMutex;

Then prefix each task's copier like this:

LockGuard guard(myMutex);
copier1.copy();

Translation: LockGuard waits until the lock is acquired (insuring exclusive access), then the copier runs, then LockGuard implicitly releases the lock (even if copier fails).

Or maybe I'm misunderstanding, and the stream buffer itself needs to be somehow declared as mutex? I would usually test before asking, but I'm away from my ESP32 for the day!

For future travelers, I found understandable examples of mutexes and lock_guard (LockGuard) here: https://stackoverflow.com/questions/35252119/stdlock-guard-example-explanation-on-why-it-works

[pschatzmann]

You got it right but use a MutexRTOS myMutex;

[z-l-p]

Unfortunately it isn't working. If I use LockGuard then the recording crashes almost every time (The panicked core is always the one that the playback task is pinned to), or it records a tiny fraction of a second and loops it (instead of continuing to record while key is pressed).

If I remove the LockGuards in either the play or record functions it works "OK", but occasionally crashes when recording starts.
Maybe I'm missing something obvious? The task memory is currently 3000 for both tasks, but I have no idea how much to request. The (broken) code is below.

/**
 * ultrasound to audible sound resampling test, based on streams_audiokit_ram_audiokit.ino by Phil Schatzmann
 * Hold Key 1 to sample ultrasound and immediately hear it at slower (audible) speed. Playback will loop when button is released.
 * Remember to enable PSRAM in Arduino IDE Tools menu
 * Requires ESP32 Arduino core 3.x (tested on 3.2.0)
 */
 
#include "AudioTools.h"
#include "AudioTools/AudioLibs/AudioBoardStream.h"
#include "AudioTools/AudioLibs/MemoryManager.h"
#include "AudioTools/Concurrency/RTOS.h"

MutexRTOS myMutex; // to handle LockGuard concurrency checks

/// USER VARS ///
int recSampleRate = 96000; // input sampling rate
int speedFactors[] = {16, 8, 4, 2, 1, 2, 4, 8}; // available speed factors to use for playback (1/speedFactor)
int sfIndex = 0; // select factor from array to use as startup default

/// INTERNAL VARS ///
bool nowRecording = 0; // are we filling the buffer?
bool nowPlaying = 0; // is there something in the buffer to play?

AudioInfo info(recSampleRate, 2, 16); // for ADC
AudioInfo out_info(recSampleRate / speedFactors[sfIndex],2,16); // for DAC
AudioBoardStream kit(AudioKitEs8388V1);
MemoryManager memory(500); // Activate SPI RAM for objects > 500 bytes
DynamicMemoryStream recording(true); // Audio stored on heap [args: loop (bool), buffer-size (int)]
ResampleStream resample(recording);
StreamCopy copier1(recording, kit); // copies data
StreamCopy copier2(kit, resample); // copies data 
auto rcfg = resample.defaultConfig(); // set up the resampler config

Task writeTask("rec-copy", 3000, 10, 0);
Task readTask("play-copy", 3000, 10, 1);

void rec_function() {
     if (nowRecording) {
      LockGuard guard(myMutex);
      copier1.copy();  
    } else {
      delay(1);
    }
  }

void play_function() {
    if (nowPlaying) {
     LockGuard guard(myMutex);
     copier2.copy();  
    } else {
      delay(1);
    }
  }

 void change_sr(bool pinStatus, int pin, void* ref) {
  sfIndex++;
  if (sfIndex > sizeof(speedFactors) / sizeof(speedFactors[0]) - 1) {
    sfIndex = 0;
  }
  Serial.print("Speed: 1/");
  Serial.println(speedFactors[sfIndex]);
  out_info.sample_rate = recSampleRate / speedFactors[sfIndex]; // update output sample-rate
  rcfg.copyFrom(out_info);
  rcfg.step_size = 1.0 / speedFactors[sfIndex]; // update resampler
  resample.end(); // restart resampler for changes to take effect
  resample.begin(rcfg);
}

void record_start(bool pinStatus, int pin, void* ref){
 
  Serial.print("Recording... (sample-rate: ");
  Serial.print(info.sample_rate);
  Serial.println(" kHz)");
  Serial.print("Playing... (sample-rate: ");
  Serial.print(out_info.sample_rate);
  Serial.print(" kHz, speed: 1/");
  Serial.print(speedFactors[sfIndex]);
  Serial.println(")");
  recording.end();
  recording.begin();
  copier1.begin();  // start recording
  copier2.begin(); // start playback
  nowRecording = 1;
  nowPlaying = 1;

}

void record_end(bool pinStatus, int pin, void* ref){
  Serial.println("Recording stopped (playback continues) ");
  nowRecording = 0;
}

void setup(){
  Serial.begin(115200);
  while(!Serial); // wait for serial to be ready
  AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Warning);  

  // setup input and output
  auto cfg = kit.defaultConfig(RXTX_MODE);
  cfg.sd_active = false;
  cfg.copyFrom(info);
  cfg.input_device = ADC_INPUT_LINE2;
  kit.begin(cfg);
  kit.setInputVolume(1.0); // input
  kit.setVolume(0.4); // output

  // define resampling info
  //auto rcfg = resample.defaultConfig(); // now global
  rcfg.copyFrom(out_info);
  rcfg.copyFrom(info);
  rcfg.step_size = 1.0 / speedFactors[sfIndex];
  resample.begin(rcfg);
  

  // record when key 1 is pressed
  kit.audioActions().add(kit.getKey(1), record_start, record_end);
  // chang playback sample-rate when key 6 is pressed
  kit.audioActions().add(kit.getKey(6), change_sr);
  Serial.println("Press Key 1 to record.");
  Serial.println("Press Key 6 to change playback speed.");

 // Start audio tasks
  writeTask.begin(rec_function);
  readTask.begin(play_function);

}

void loop(){
  // Process keys
  kit.processActions();
}

[pschatzmann]

Using multiple tasks is tricky and you should avoid unsynchronized global state:

• I replaced the global state variables with starting and stoping the tasks
• DynamicMemoryStream is risky and should not be working: I added a small delay between the starting of the tasks to minimize the risks of concurrency conflicts.

The following is not giving any crashes:

#include "AudioTools.h"
#include "AudioTools/AudioLibs/AudioBoardStream.h"
#include "AudioTools/AudioLibs/MemoryManager.h"
#include "AudioTools/Concurrency/RTOS.h"

/// USER VARS ///
int recSampleRate = 96000;                         // input sampling rate
int speedFactors[] = { 16, 8, 4, 2, 1, 2, 4, 8 };  // available speed factors to use for playback (1/speedFactor)
int sfIndex = 0;                                   // select factor from array to use as startup default/// INTERNAL VARS ///

AudioInfo info(recSampleRate, 2, 16);                              // for ADC
AudioInfo out_info(recSampleRate / speedFactors[sfIndex], 2, 16);  // for DAC
AudioBoardStream kit(AudioKitEs8388V1);
MemoryManager memory(500);            // Activate SPI RAM for objects > 500 bytes
DynamicMemoryStream recording(true);  // Audio stored on heap [args: loop (bool), buffer-size (int)]

ResampleStream resample(recording);
StreamCopy copier1(recording, kit);    // copies data
StreamCopy copier2(kit, resample);     // copies data
auto rcfg = resample.defaultConfig();  // set up the resampler config

Task writeTask("rec-copy", 3000, 10, 0);
Task readTask("play-copy", 3000, 10, 1);

void rec_function() {
  copier1.copy();
}

void play_function() {
  copier2.copy();
}

void change_sr(bool pinStatus, int pin, void* ref) {
  sfIndex++;
  if (sfIndex > sizeof(speedFactors) / sizeof(speedFactors[0]) - 1) {
    sfIndex = 0;
  }
  Serial.print("Speed: 1/");
  Serial.println(speedFactors[sfIndex]);
  out_info.sample_rate = recSampleRate / speedFactors[sfIndex];  // update output sample-rate
  rcfg.copyFrom(out_info);
  rcfg.step_size = 1.0 / speedFactors[sfIndex];  // update resampler
  resample.end();                                // restart resampler for changes to take effect
  resample.begin(rcfg);
}

void record_start(bool pinStatus, int pin, void* ref) {
  readTask.end();
  writeTask.end();

  Serial.print("Recording... (sample-rate: ");
  Serial.print(info.sample_rate);
  Serial.println(" kHz)");
  Serial.print("Playing... (sample-rate: ");
  Serial.print(out_info.sample_rate);
  Serial.print(" kHz, speed: 1/");
  Serial.print(speedFactors[sfIndex]);
  Serial.println(")");
  recording.end();
  recording.begin();
  writeTask.begin(rec_function);
  delay(10);
  resample.begin();
  readTask.begin(play_function);
}

void record_end(bool pinStatus, int pin, void* ref) {
  Serial.println("Recording stopped (playback continues) ");
  writeTask.end();
}

void setup() {
  Serial.begin(115200);
  while (!Serial)
    ;  // wait for serial to be ready
  AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Warning);

  // setup input and output
  auto cfg = kit.defaultConfig(RXTX_MODE);
  cfg.sd_active = false;
  cfg.copyFrom(info);
  cfg.input_device = ADC_INPUT_LINE2;
  kit.begin(cfg);
  kit.setInputVolume(1.0);  // input
  kit.setVolume(0.4);       // output

  // define resampling info
  //auto rcfg = resample.defaultConfig(); // now global
  rcfg.copyFrom(out_info);
  rcfg.copyFrom(info);
  rcfg.step_size = 1.0 / speedFactors[sfIndex];
  resample.begin(rcfg);


  // record when key 1 is pressed
  kit.audioActions().add(kit.getKey(5), record_start, record_end);
  // chang playback sample-rate when key 6 is pressed
  kit.audioActions().add(kit.getKey(6), change_sr);
  Serial.println("Press Key 1 to record.");
  Serial.println("Press Key 6 to change playback speed.");

}

void loop() {
  // Process keys
  kit.processActions();
  delay(1);
}

I will try to change the implementation of DynamicMemoryStream to support a writing and reading task properly...

[z-l-p]

Amazing. Works great! I really appreciate that you took the time!
To make it play back at whatever slower rate is configured (rather than the recorded rate) I changed the resample.begin method in the record_start() function so it refers to the settings object: resample.begin(rcfg);
Thanks so much!