Sending/Receiving Audio Packets via Websockets

[mittlc]

Hello there,
first of all:
thanks a lot for this library - it made approaching audio on the esp32 a lot simpler for me.

But currently i am running into a wall - so i figured i might open a discussion here:

I started of with the sinewave and then microphone loopback to i2s amplifier examples, listening to url streams and hosting a microphone webserver so far and at last managed to setup a continious stream between two identical esp32 boards each equiped with an i2s inmp441 microphone and i2s max98357a amplifier and speaker using the ip sender/receiver examples ( that is after i figured out that i need to read 32bit from the microphone first and then transform it to 16 bit rather than trying to read 16bit directly as done by other audio examples online using the i2sdrive.h library)

Currently one board is always acting as the sender and one as the receiver (the goal is to make a project with two way communication).
This works quite fine so far to transmit audio in realtime from the sender to the receiver in the same network and putting it out on the i2s speaker.

Now i am trying to adapt these examples to use a central relay server and websocket communication, to be able to

• send audio from an i2s Stream (coming from the microphone) via websocket
• reveive audio data from websocket events and write it to an i2s Stream (going to be outputed via the i2s amplifier board)
• still be able use the websocket connection to send and receive text commands besides audio data between the two esp32 boards
• potentially use the server in the middle for some further processing of the audio
• make this working across network boundaries as long as both esp32 clients can reach the central relay server

The issue is that i can not figure out how i can correctly send the converted i2s stream from the microphone via websockets and push received binary data from the websocket messages toward the i2s output stream.
I am trying to use the arduinoWebsockets library by Link2004 which provides a sendBIN(payload,length) function and also allows to receive binary data.
https://github.com/Links2004/arduinoWebSockets/blob/master/examples/esp32/WebSocketClient/WebSocketClient.ino

I had a look on https://github.com/pschatzmann/arduino-audio-tools/wiki/Writing-your-Custom-Input&Output-Class but unfortunately cannot wrap my head around this.

I also found that the websocket library seems to only beeing able to work with 8bit data packets, meaning that i need to split my 16bit i2s output packets into 8bit packets and then sticht them back together at the receiver side - but i can´t figure out how i get messages from the webSocketEvent WStype_BIN into the i2s output stream.

I guess i need to use some Form of Buffer and maybe the QueueStream? to send the websocket packets to the i2s output stream - but thats where i get lost.

Any ideas on this?

Thanks

[pschatzmann]

I haven't done anything with websockets yet, but here are my 5 cents:

• I would use some defined audio format: convert your data e.g. to a 1 channel 8 or 16 bit wav file. This minimizes the amount of data that needs to be sent and gives the receiver the information how to play it.
• Do your projects is steps: first build and test the sending of the microphone data to the server: this let's you easily confirm that your approach is working. Having wav will also help you with your tests because you can play it easily on the server.
• You don't really need any custom components: Just read the data from I2S via some conversion streams and write it with sendBIN in the loop or better in a seperate task
• When you implement the speaker just do the oposite: receive it in the loop and write it to an I2SStream.
• Implement some start/stop logic so that the wav header is generated when started and that the speaker closes properly when the sending is stopped, so that it knows that the first data it receives is a wav header.

I suggest you read the Wikis about

• Introduction
• It's not working
• Format Conversion
• Codecs
• Multicore Processing

Just a simpler alternative:

• You could use my TinyHttp server which supports multiple (audio) connections in the same Wifi network
• If you want to make it available on the internet just use a public proxy server: e.g. NGINX

[mittlc]

Thanks! that was of great help and i got it working - up next fixing the white noise in the background :)

[SriramKeerthi]

@mittlc I'm trying to do the same as well - can you please share your code?

[mittlc]

Hi there,
the project currently consists of one esp32 sketch to be uploaded to two identical boards (esp32-s3-devkitc-1-n16r8) equiped with an inmp441 Microphone, max98357a amplifier, micro sd card module and a push button.

Wifi credentials, target websocket server, websocket port, microphone, speaker and sd module pins are defined within a seperate Settings.h file.

I am working on a project including some calling/ringing logic to have both esp32 boards interact with each other - so i guess you can delete the most from it depending on what you need for your project. So the main focus can be put on the websocket sending which is just sending 8bit packets from the arduino audio tools via websocket and in the receiving end it is written to the i2s stream from the websocket message. While i first got confused with the 16bit audio data and the 8bit websocket library what got me on the right track was pschatzman answer in this post together with this post over in the websocket library:
Links2004/arduinoWebSockets#213

In between the two esp32 boards sits a python server relaying the websocket messages to all other connected client other than the sender itself.

Currently i am trying to see if it is possible to analyze and then overwrite the stream on the server side with silence if there was no voice detected, but this currently causes issues so that nothing is relayed to the other client at all, so still work in progress.

And i am also trying to reduce the amount of noise in the background from the microphone stream (not sure if that is picked up by wifi or if my breadboard connections are messy, about that you can find many different posts on the topic of noise on the inmp441).

Hope this helps in your project:

Cheers

esp32_clients.txt
python_relay_server.txt

ESP32 Clients:

#include <Arduino.h>
#include "AudioTools.h"
#include <WiFi.h>
#include <WebSocketsClient.h>
#include <Settings.h>  // Include your settings (SSID, password, server details)
#include <SD.h>
#include <SPI.h>

// Audio Settings
AudioInfo from(16000, 1, 32);  // Input audio format
AudioInfo to(16000, 1, 16);    // Output audio format
AudioInfo volumeconfig(16000, 1, 16);    // Output audio format
// I2S Input Stream (Microphone)
I2SStream in;
FormatConverterStream conv(in);
VolumeStream volume(conv);

// I2S Output Stream (Speaker)
I2SStream i2sStream;

// WebSocket Client
WebSocketsClient webSocket;

// Audio buffer for streaming
const size_t bufferSize = 256;  // 256 samples of int16_t
int16_t audioBuffer[bufferSize];

// SD Card and WAV playback
WAVDecoder wav;
EncodedAudioOutput decoder(&i2sStream, &wav);  // Decoding stream
StreamCopy copier;
File audioFile;

// Button handling
const int buttonPin = 15;        // GPIO pin connected to the button
bool buttonState = HIGH;         // Current reading from the input pin
bool lastButtonState = HIGH;     // Previous reading from the input pin
unsigned long lastDebounceTime = 0;  // Last time the button state changed
unsigned long debounceDelay = 50;    // Debounce time in milliseconds

// State Machine
enum State { IDLE, RINGING, ACTIVECALL };
State currentState = IDLE;

// Sound Types
enum SoundType { NONE, RINGING_SOUND, ACCEPT_SOUND };
SoundType currentSound = NONE;
bool isPlaying = false;

// Flags
bool isStreaming = false;
bool isWebSocketConnected = false;
bool isCaller = false;  // Flag to indicate if this device is the caller

// Function to connect to Wi-Fi
void connectWifi() {
  Serial.print("Connecting to Wi-Fi");
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("\nConnected to Wi-Fi");
  Serial.print("IP Address: ");
  Serial.println(WiFi.localIP());

  // Optional: Disable Wi-Fi power save for better performance
  esp_wifi_set_ps(WIFI_PS_NONE);
}

// Function to start playback of a WAV file
void startPlayback(const char* filename) {
  audioFile = SD.open(filename);
  if (!audioFile) {
    Serial.print("Failed to open audio file ");
    Serial.println(filename);
    return;
  }
  // Start decoder and copier
  decoder.begin();
  copier.begin(decoder, audioFile);
  isPlaying = true;

  if (strcmp(filename, "/ringingsound.wav") == 0) {
    currentSound = RINGING_SOUND;
  } else if (strcmp(filename, "/acceptsound.wav") == 0) {
    currentSound = ACCEPT_SOUND;
  } else {
    currentSound = NONE;
  }

  Serial.print("Playback started: ");
  Serial.println(filename);
}

// Function to stop playback
void stopPlayback() {
  // Stop decoder and copier
  copier.end();
  decoder.end();
  audioFile.close();
  isPlaying = false;
  currentSound = NONE;
  Serial.println("Playback stopped");
}

// Function to initialize SD card
void setupSDCard() {
  Serial.println("Initializing SD card...");
  SPI.begin(SD_SCK, SD_MISO, SD_MOSI, SD_CS);
  if (!SD.begin(SD_CS)) {
    Serial.println("SD card initialization failed!");
    while (1) {
      delay(1000); // Stay here if SD card initialization fails
    }
  }
  Serial.println("SD card initialized.");
}

// Button handling with debounce logic
void checkButton() {
  int reading = digitalRead(buttonPin);

  if (reading != lastButtonState) {
    lastDebounceTime = millis();  // Reset the debouncing timer
  }

  if ((millis() - lastDebounceTime) > debounceDelay) {
    // If the button state has changed
    if (reading != buttonState) {
      buttonState = reading;

      // Only act on the button press when it goes LOW (button pressed)
      if (buttonState == LOW) {
        if (currentState == IDLE) {
          // Send "RINGING" message
          if (isWebSocketConnected) {
            webSocket.sendTXT("RINGING");
            currentState = RINGING;
            isCaller = true; // This device is the caller
            startPlayback("/ringingsound.wav");
          }
        } else if (currentState == RINGING) {
          if (isCaller) {
            // Caller cancels the call
            if (isWebSocketConnected) {
              webSocket.sendTXT("ABORT");
              stopPlayback();
              currentState = IDLE;
            }
          } else {
            // Callee accepts the call
            if (isWebSocketConnected) {
              webSocket.sendTXT("ACCEPT");
              stopPlayback();
              startPlayback("/acceptsound.wav");
              currentState = ACTIVECALL;
              isStreaming = true;
            }
          }
        } else if (currentState == ACTIVECALL) {
          // Send "ABORT" message
          if (isWebSocketConnected) {
            webSocket.sendTXT("ABORT");
            isStreaming = false;
            startPlayback("/acceptsound.wav"); // Play accept sound when aborting call
            currentState = IDLE;
          }
        }
      }
    }
  }
  lastButtonState = reading;
}

// WebSocket Event Handler
void webSocketEvent(WStype_t type, uint8_t* payload, size_t length) {
  switch (type) {
    case WStype_DISCONNECTED:
      Serial.println("WebSocket Disconnected!");
      isWebSocketConnected = false;
      break;

    case WStype_CONNECTED:
      Serial.println("WebSocket Connected!");
      isWebSocketConnected = true;
      break;

    case WStype_BIN: {
      if (currentState == ACTIVECALL) {
        // Write the received audio data to the I2S output
        size_t bytesWritten = i2sStream.write(payload, length);
        if (bytesWritten != length) {
          Serial.printf("Error: Expected to write %u bytes, but wrote %u bytes\n", length, bytesWritten);
        }
      }
      break;
    }

    case WStype_TEXT: {
      String message = String((char*)payload);
      Serial.printf("Received message: %s\n", message.c_str());

      if (message == "RINGING") {
        if (currentState == IDLE) {
          currentState = RINGING;
          isCaller = false; // This device is being called
          startPlayback("/ringingsound.wav");
        }
      } else if (message == "ACCEPT") {
        if (currentState == RINGING) {
          stopPlayback();
          startPlayback("/acceptsound.wav");
          currentState = ACTIVECALL;
          isStreaming = true;
        }
      } else if (message == "ABORT") {
        if (currentState == ACTIVECALL) {
          stopPlayback();
          startPlayback("/acceptsound.wav"); // Play accept sound when call is aborted
          currentState = IDLE;
          isStreaming = false;
        } else if (currentState == RINGING) {
          stopPlayback();
          currentState = IDLE;
          isStreaming = false;
        }
      }
      break;
    }

    case WStype_ERROR:
      Serial.println("WebSocket Error!");
      break;

    default:
      break;
  }
}

void setup() {
  Serial.begin(115200);
  AudioLogger::instance().begin(Serial, AudioLogger::Info);

  // Initialize button
  pinMode(buttonPin, INPUT_PULLUP);  // Set pin as INPUT_PULLUP for the button

  // Initialize SD card
  setupSDCard();

  // Connect to Wi-Fi
  connectWifi();

  // Initialize WebSocket client
  webSocket.begin(serverAddress, port, "/");
  webSocket.onEvent(webSocketEvent);
  webSocket.setReconnectInterval(5000);

  // Initialize I2S input (Microphone)
  Serial.println("Initializing I2S input...");
  auto config_in = in.defaultConfig(RX_MODE);
  config_in.sample_rate = from.sample_rate;
  config_in.bits_per_sample = from.bits_per_sample;
  config_in.channels = from.channels;
  config_in.pin_bck = AUDIO_MIC_SCK;   // Bit Clock (BCLK) pin
  config_in.pin_ws = AUDIO_MIC_WS;     // Word Select (LRCLK) pin
  config_in.pin_data = AUDIO_MIC_SD;   // Serial Data In (DOUT) pin
  config_in.port_no = I2S_NUM_0;
  in.begin(config_in);
  conv.begin(from, to);
  

  auto vol_config = volume.defaultConfig();
  vol_config.volume = 10;
  vol_config.allow_boost = true;

  volume.begin(vol_config);  



  // Initialize I2S output (Speaker)
  Serial.println("Initializing I2S output...");
  auto config_out = i2sStream.defaultConfig(TX_MODE);
  config_out.sample_rate = to.sample_rate;
  config_out.bits_per_sample = to.bits_per_sample;
  config_out.channels = to.channels;
  config_out.buffer_size = 512;
  config_out.buffer_count = 6;
  config_out.pin_bck = AUDIO_SPEAKER_BCLK;  // Bit Clock pin
  config_out.pin_ws = AUDIO_SPEAKER_LRC;    // Word Select pin
  config_out.pin_data = AUDIO_SPEAKER_DIN;  // Serial Data Out pin
  config_out.port_no = I2S_NUM_1;
  i2sStream.begin(config_out);

  Serial.println("Setup complete.");
}

void loop() {
  // Handle WebSocket communication
  webSocket.loop();

  // Check button state
  checkButton();

  // Handle playback
  if (isPlaying) {
    if (!copier.copy()) {
      // Playback finished
      stopPlayback();
      if (currentState == RINGING && currentSound == RINGING_SOUND) {
        // Loop the ringing sound
        startPlayback("/ringingsound.wav");
      } else if (currentState == ACTIVECALL && currentSound == ACCEPT_SOUND) {
        // Continue as per state
        currentSound = NONE;
      } else if (currentState == IDLE && currentSound == ACCEPT_SOUND) {
        // After aborting the call and playing accept sound
        currentSound = NONE;
      } else {
        currentSound = NONE;
      }
    }
  }

  // Handle audio streaming
  if (isStreaming && currentState == ACTIVECALL && isWebSocketConnected) {
    // Read audio data from the microphone
    size_t bytesGenerated = volume.readBytes((uint8_t*)audioBuffer, sizeof(audioBuffer));

    // Send the audio data over WebSocket
    webSocket.sendBIN((uint8_t*)audioBuffer, bytesGenerated);
  }
}

Python Relay Server:

import asyncio
import websockets
import torch
import time
import json  # Import json module for JSON formatting

last_voice_debounce = 1
# Load the Silero VAD model
model, utils = torch.hub.load(repo_or_dir='snakers4/silero-vad', model='silero_vad', force_reload=False)
(get_speech_timestamps, save_audio, read_audio, VADIterator, collect_chunks) = utils

# Settings for the VAD
sampling_rate = 16000  # We're working directly with 16kHz audio

# Dictionary to hold connected clients and their data
connected_clients = {}

class ClientData:
    def __init__(self, websocket):
        self.websocket = websocket
        self.audio_buffer = torch.tensor([])
        self.voice = False
        self.last_voice_time = time.time()
        self.ip = websocket.remote_address[0]

async def print_voice_status():
    while True:
        status_list = []
        for client_data in connected_clients.values():
            status_list.append(f"{client_data.ip}: {client_data.voice}")
        status_str = ", ".join(status_list)
        print(f"Voice status: {status_str}")
        await asyncio.sleep(0.5)  # Print every 0.5 seconds

async def broadcast_voice_status():
    while True:
        for client_data in connected_clients.values():
            # Check if any other client has voice=True
            other_clients_voice = any(
                other_client.voice for other_client in connected_clients.values()
                if other_client != client_data
            )
            message = '1' if other_clients_voice else '0'
            if client_data.websocket.open:
                try:
                    await client_data.websocket.send(message)
                except websockets.exceptions.ConnectionClosed:
                    pass
        await asyncio.sleep(0.5)  # Send every 0.5 seconds

async def relay_server(websocket, path):
    # Register new client
    connected_clients[websocket] = ClientData(websocket)
    client_data = connected_clients[websocket]
    print(f"Client connected: {websocket.remote_address}")

    try:
        async for message in websocket:
            if isinstance(message, bytes):
                # Message contains binary audio data, process it
                audio_chunk = torch.frombuffer(message, dtype=torch.int16).float() / 32768.0
                client_data.audio_buffer = torch.cat((client_data.audio_buffer, audio_chunk))

                # If we have more than 1 second of audio (16000 samples for 16kHz)
                if client_data.audio_buffer.shape[0] >= 16000:
                    # Perform VAD
                    timestamps = get_speech_timestamps(
                        client_data.audio_buffer[:16000],
                        model,
                        threshold=0.1
                    )

                    # Store previous voice status
                    prev_voice = client_data.voice

                    # Check if there was speech in the last second
                    if len(timestamps) > 0:
                        client_data.voice = True
                        client_data.last_voice_time = time.time()
                    else:
                        # Reset voice status if no speech detected for last_voice_debounce seconds
                        if time.time() - client_data.last_voice_time > last_voice_debounce:
                            client_data.voice = False

                    # Reset the buffer
                    client_data.audio_buffer = client_data.audio_buffer[16000:]

                # Relay audio message to all other connected clients
                for other_client in connected_clients.values():
                    if other_client.websocket != websocket and other_client.websocket.open:
                        try:
                            await other_client.websocket.send(message)
                        except websockets.exceptions.ConnectionClosed:
                            pass
            else:
                # Message is text, relay it to other clients
                message_str = message.strip()
                print(f"Received text message from {client_data.ip}: {message_str}")
                for other_client in connected_clients.values():
                    if other_client.websocket != websocket and other_client.websocket.open:
                        try:
                            await other_client.websocket.send(message_str)
                        except websockets.exceptions.ConnectionClosed:
                            pass
    except websockets.exceptions.ConnectionClosed:
        pass
    finally:
        # Unregister client
        del connected_clients[websocket]
        print(f"Client disconnected: {websocket.remote_address}")

async def main():
    # Start WebSocket relay server
    server = await websockets.serve(relay_server, "0.0.0.0", 8555)
    print_task = asyncio.create_task(print_voice_status())  # Start the voice status printing task
    
    # disable the broadcast voice status for now - not used in the esp32 clients
    #broadcast_task = asyncio.create_task(broadcast_voice_status())  # Start the voice status broadcasting task
    #await asyncio.gather(server.wait_closed(), print_task, broadcast_task)
    await asyncio.gather(server.wait_closed(), print_task)
if __name__ == "__main__":
    asyncio.run(main())

[pschatzmann]

Here is the reference how to post code propery, so that it does not get messed up: if you add C++ just after the triple backticks, you even get the language specific syntax colouring.

[mittlc]

@pschatzmann - i hope its ok to make a follow up in this discussion as it is directly connected (but i am happy to create a new post for this question as well if thats better)

In the ESP32 Code Above - how would you go about implementing some sort of buffering on the receiving end?

The sending and receiving of audio via websocket in the local network is working fine by now - but i run into issues when trying to run the setup across network boundaries (using a cloudflare tunnel to expose my websocket server).
I think this might be related to the latency introduced and that i simply try to write the data received from the websocket messages directly to the I2S Output Stream.

so i guess in this part of the sketch where i receive the audio data i would need some kind of buffering to collect enough audio data to then allow for continous writting to the speaker - but i am not sure how a buffer might be integrated based on the examples i found.

case WStype_BIN: {
      if (currentState == ACTIVECALL) {
        // Write the received audio data to the I2S output
        size_t bytesWritten = i2sStream.write(payload, length);
        if (bytesWritten != length) {
          Serial.printf("Error: Expected to write %u bytes, but wrote %u bytes\n", length, bytesWritten);
        }
      }
      break;
    }

[pschatzmann]

I suggest that you have another look at this Wiki