AFTER_Training_Template--Kaggle.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AFTER: Audio Features Transfer and Exploration in Real-time\n",
    "\n",
    "[AFTER](https://github.com/acids-ircam/AFTER) is a diffusion-based generative model that creates new audio by blending two sources: one audio stream to set the style or timbre, and another input (either audio or MIDI) to shape the structure over time.\n",
    "\n",
    "This repository is a real-time implementation of the [research paper](https://arxiv.org/abs/2408.00196) *Combining audio control and style transfer using latent diffusion* by Nils Demerlé, P. Esling, G. Doras, and D. Genova.\n",
    "\n",
    "----\n",
    "\n",
    "Please note that right now, this notebook currently *only* covers training **AFTER audio-to-audio models** using **pre-trained autoencoders** (e.g. RAVE models). \n",
    "\n",
    "Therefore, in order to use this notebook, you need: \n",
    "* an autoencoder model exported **without streaming** for preprocessing and training\n",
    "* the same autoencoder model exported **with streaming** enabled for model export \n",
    "\n",
    "----\n",
    "\n",
    "Notebook author: [Martin Heinze](https://github.com/devstermarts)\n",
    "\n",
    "Last updated: 16.04.2025"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup runtime\n",
    "\n",
    "Set up Miniconda with Python 3.11, then install AFTER from GitHub on the runtime. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Install Miniconda\n",
    "\n",
    "!mkdir /kaggle/temp\n",
    "%cd /kaggle/temp\n",
    "!curl -L https://repo.anaconda.com/miniconda/Miniconda3-py311_24.11.1-0-Linux-x86_64.sh -o miniconda.sh\n",
    "!chmod +x miniconda.sh\n",
    "!sh miniconda.sh -b -p /kaggle/temp/miniconda"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Install AFTER\n",
    "\n",
    "%cd /kaggle/temp\n",
    "!git clone -b wip https://github.com/devstermarts/AFTER.git\n",
    "%cd /kaggle/temp/AFTER\n",
    "!/kaggle/temp/miniconda/bin/pip install -e ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess dataset for audio-to-audio model\n",
    "\n",
    "Preprocessing needs to be done once per training. You can use both autoencoders trained with AFTER source code (not covered here) or e.g. RAVE. In this notebook, the assumption is that you have a pre-trained autoencoder at hand. Note that for both preprocessing and training later, you need that autoencoder **without streaming** enabled. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#AFTER dataset preprocessing\n",
    "\n",
    "!mkdir /kaggle/temp/processed\n",
    "\n",
    "!/kaggle/temp/miniconda/bin/after prepare_dataset \\\n",
    "--input_path /kaggle/input/your-audio-folder \\\n",
    "--output_path /kaggle/temp/processed \\\n",
    "--emb_model_path /kaggle/input/your-rave-model-without-streaming.ts \\\n",
    "--gpu 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train audio-to-audio model\n",
    "\n",
    "Below section covers training an audio-to-audio AFTER model. Again, use your pre trained autoencoder model **without streaming** here.\n",
    "\n",
    "In order to resume training, you need to transform the output data of the first run into a new dataset from the output tab of your notebook after the first run is complete. You then add the new dataset to the notebook, enable below copy command from your earlier training output to your runtime's working directory and set the --restart flag in training section to the steps of the checkpoint you want to resume from. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Copy files to /kaggle/working folder to continue training from an earlier checkpoint.\n",
    "#!cp -r /kaggle/input/your-earlier-training-output/* /kaggle/working"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#AFTER model training. Use --restart flag when you continue an earlier training. \n",
    "\n",
    "!/kaggle/temp/miniconda/bin/after train  \\\n",
    "--name your-training-name \\\n",
    "--db_path /kaggle/temp/processed \\\n",
    "--emb_model_path /kaggle/input/your-rave-model-without-streaming.ts \\\n",
    "--config base \\\n",
    "--out_path /kaggle/working/after_runs/ #\\\n",
    "#--restart steps-of-latest-checkpoint "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export audio-to-audio model\n",
    "\n",
    "To export your AFTER model for real time use with nn~, you need to use an autoencoder with **streaming enabled** (opposed to pre processing and training, where you used the model *without* streaming). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#AFTER model export\n",
    "\n",
    "!/kaggle/temp/miniconda/bin/after export \\\n",
    "--model_path your-training-name \\\n",
    "--emb_model_path /kaggle/input/your-rave-model-WITH-streaming.ts \\\n",
    "--step 800000"
   ]
  }
 ],
 "metadata": {
  "kaggle": {
   "accelerator": "none",
   "dataSources": [],
   "isGpuEnabled": false,
   "isInternetEnabled": true,
   "language": "python",
   "sourceType": "notebook"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}