AudioMIR is an open-source, research-grade Music Information Retrieval (MIR) and Machine Listening framework that automates the joint discovery of audio representations, deep neural network architectures, and training hyperparameters for rhythm understanding tasks (Tempo Estimation & Rhythm Style Classification).
Rather than treating model accuracy as an isolated objective, AudioMIR formulates neural architecture search as a multi-objective Pareto optimization problem, simultaneously optimizing predictive accuracy, inference latency (ms), and model footprint (MB).
- 1. Scientific Motivation & Research Questions
- 2. System Architecture
- 3. Audio Representations & DSP Engine
- 4. Deep Learning Model Architectures
- 5. Multi-Objective AutoML & NSGA-II Search
- 6. Datasets & Zero-Leakage Guarantee
- 7. Evaluation Metrics & Benchmarks
- 8. Installation & Environment Setup
- 9. Command-Line Interface (CLI) Guide
- 10. Interactive Streamlit Dashboard
- 11. Reproducibility & Experiment Tracking
- 12. Repository Structure
- 13. License & Citation
Deep learning models in Music Information Retrieval often suffer from excessive parameter counts and high computational latency, making them impractical for real-time applications such as Digital Audio Workstations (DAWs), live performance plugins, and edge/mobile devices.
AudioMIR investigates the following core research questions:
- RQ1 (Pareto Frontier Discovery): Can automated multi-objective search discover compact, low-latency rhythm models that retain strong predictive accuracy with negligible performance degradation compared to oversized models?
- RQ2 (Audio Representations): How do Log-Mel Spectrograms, Fourier Tempograms, and Dual-Tower representations compare across continuous tempo regression and discrete rhythm style classification?
- RQ3 (Algorithmic Comparison): Under identical candidate-evaluation budgets, does native evolutionary NSGA-II discover superior Pareto fronts compared to Random Search and Bayesian Optuna/TPE baselines?
- RQ4 (Domain Generalization): How robustly do Pareto-optimal models transfer across acoustic domains (e.g., from public MIDI-aligned drums to custom studio drum loop libraries)?
+-----------------------+
| Audio Stream |
| (WAV / MP3 / FLAC) |
+-----------+-----------+
|
+-----------v-----------+
| Deterministic Audio |
| Preprocessing |
+-----------+-----------+
|
+------------------------+------------------------+
| |
+-----------v-----------+ +-----------v-----------+
| Log-Mel Spectrogram | | Fourier Tempogram |
| (64, 96, 128 Mel) | | (Periodicity Bins) |
+-----------+-----------+ +-----------+-----------+
| |
+------------------------+------------------------+
|
+-----------v-----------+
| Candidate Model |
| TinyCNN / CRNN / Dual |
+-----------+-----------+
|
+----------------+----------------+
| |
+-----------v-----------+ +-----------v-----------+
| Tempo Head log2(BPM) | | Style / Meter Head |
| (Log-space SmoothL1) | | (Softmax CrossEntropy)|
+-----------+-----------+ +-----------+-----------+
| |
+----------------+----------------+
|
+-----------v-----------+
| Evaluation Engine |
| Quality, Latency, Size|
+-----------+-----------+
|
+-----------v-----------+
| Multi-Objective AutoML|
| (Random / TPE / NSGA) |
+-----------+-----------+
|
+-----------v-----------+
| 2D/3D Pareto Frontier |
| (Best / Fast / Small) |
+-----------------------+
AudioMIR provides deterministic audio preprocessing and feature extraction with automatic disk caching:
-
Log-Mel Spectrogram (
logmel): Captures spectral timbre and harmonic distribution across configurable Mel bins ($N_{\text{mels}} \in {64, 96, 128}$ ), with amplitude-to-dB conversion. -
Fourier Tempogram (
tempogram): Extracts rhythmic periodicity and tempo harmonics by computing the Short-Time Fourier Transform of the onset strength envelope. -
Dual Representation (
logmel_tempogram): Concatenates both spectral (timbral) and periodicity (rhythmic) features into complementary input streams. - Deterministic SHA-256 Feature Caching: Hashes audio content identity, sample rate, segment duration, and representation parameters to guarantee stale features are never silently reused.
In traditional computer vision, CNNs process 2D RGB pixel images. In AudioMIR, we treat sound as a 2D Time-Frequency Energy Image:
flowchart TD
subgraph AudioDSP ["1. DSP & Feature Extraction"]
WAV["Raw Audio (1D Waveform)"] --> STFT["STFT & Mel Filterbank / Tempogram"]
STFT --> SPEC["2D Feature Matrix [1, Freq/Bins, Time]"]
end
subgraph CNNBackbone ["2. Hierarchical 2D CNN Feature Extractor"]
SPEC --> CB1["Conv Block 1: Conv2D(1βC, 3Γ3) + BatchNorm + ReLU + MaxPool(2Γ2)"]
CB1 --> CB2["Conv Block 2: Conv2D(Cβ2C, 3Γ3) + BatchNorm + ReLU + MaxPool(2Γ2)"]
CB2 --> CB3["Conv Block 3: Conv2D(2Cβ4C, 3Γ3) + BatchNorm + ReLU + MaxPool(2Γ2)"]
end
subgraph TemporalModeling ["3. Temporal Sequence Modeling (CRNN)"]
CB3 --> GRU["Bidirectional GRU Layer (Hidden Dim = 64)"]
GRU --> POOL["Temporal Mean / Adaptive Pooling"]
POOL --> LATENT["Compact Rhythmic Latent Vector [128-d]"]
end
subgraph MultiTaskHeads ["4. Multi-Task Output Neurons"]
LATENT --> HEAD_TEMPO["Tempo Head: Linear(128β64) β ReLU β Linear(64β1)"]
LATENT --> HEAD_STYLE["Style Head: Linear(128β64) β ReLU β Linear(64βN_styles)"]
LATENT --> HEAD_METER["Meter Head: Linear(128β32) β ReLU β Linear(32βN_meters)"]
HEAD_TEMPO --> BPM_OUT["Predicted Tempo: BPM = 2^(log2_bpm)"]
HEAD_STYLE --> STYLE_OUT["Style Probabilities: Softmax (Rock, Funk, Jazz, Latin...)"]
HEAD_METER --> METER_OUT["Meter Probabilities: Softmax (4/4, 3/4, 6/8)"]
end
classDef dsp fill:#1e293b,stroke:#64748b,stroke-width:2px,color:#f8fafc;
classDef cnn fill:#0f172a,stroke:#3b82f6,stroke-width:2px,color:#f8fafc;
classDef rnn fill:#0f172a,stroke:#8b5cf6,stroke-width:2px,color:#f8fafc;
classDef heads fill:#0f172a,stroke:#10b981,stroke-width:2px,color:#f8fafc;
class WAV,STFT,SPEC dsp;
class CB1,CB2,CB3 cnn;
class GRU,POOL,LATENT rnn;
class HEAD_TEMPO,HEAD_STYLE,HEAD_METER,BPM_OUT,STYLE_OUT,METER_OUT heads;
For multi-modal fusion of timbral and rhythmic periodicity spectra:
flowchart LR
subgraph Tower1 ["Log-Mel Tower (Timbre & Energy)"]
IN_MEL["Log-Mel Spectrogram [1, 96, T]"] --> CNN_MEL["CNN Backbone / CRNN"]
CNN_MEL --> EMB_MEL["Latent Timbre Vector z_mel"]
end
subgraph Tower2 ["Tempogram Tower (Periodicity & Pulse)"]
IN_TEMPO["Fourier Tempogram [1, 193, T]"] --> CNN_TEMPO["CNN Backbone / CRNN"]
CNN_TEMPO --> EMB_TEMPO["Latent Periodicity Vector z_tempo"]
end
subgraph FusionHead ["Dense Fusion & Multi-Task Prediction"]
EMB_MEL & EMB_TEMPO --> CAT["Concatenation [z_mel || z_tempo]"]
CAT --> DENSE["Dense Fusion: Linear(256β128) + ReLU + Dropout"]
DENSE --> OUT_BPM["Tempo Prediction log2(BPM)"]
DENSE --> OUT_STYLE["Style Classification (CrossEntropy)"]
end
classDef tower1 fill:#1e293b,stroke:#3b82f6,stroke-width:2px,color:#f8fafc;
classDef tower2 fill:#1e293b,stroke:#f59e0b,stroke-width:2px,color:#f8fafc;
classDef fusion fill:#0f172a,stroke:#10b981,stroke-width:2px,color:#f8fafc;
class IN_MEL,CNN_MEL,EMB_MEL tower1;
class IN_TEMPO,CNN_TEMPO,EMB_TEMPO tower2;
class CAT,DENSE,OUT_BPM,OUT_STYLE fusion;
Taking an input audio segment (
| Layer / Stage | Layer Type | Output Tensor Dimension |
Musical Feature Extracted |
|---|---|---|---|
| Input | Audio Tensor | (B, 1, 96, 345) |
|
| Conv Block 1 | Conv2D(1β32) + BN + ReLU + MaxPool |
(B, 32, 48, 172) |
Transient onsets, drum attacks, cymbal sizzle |
| Conv Block 2 | Conv2D(32β64) + BN + ReLU + MaxPool |
(B, 64, 24, 86) |
Harmonic overtones, recurring rhythmic subdivisions |
| Conv Block 3 | Conv2D(64β128) + BN + ReLU + MaxPool |
(B, 128, 12, 43) |
Full beat patterns, syncopation accents |
| BiGRU Layer | BiGRU(Hidden=64, Bidirectional) |
(B, 43, 128) |
Long-range bar counts, swing groove, meter phase |
| Pooling | Temporal Mean Pooling |
(B, 128) |
Fixed-size global rhythm embedding vector |
| Tempo Head | Linear(128β64) β Linear(64β1) |
(B, 1) |
Scalar tempo |
| Style Head | Linear(128β64) β Linear(64βN) |
(B, N_classes) |
Style logits (Rock, Funk, Jazz, Latin...) |
- Spectrogram as an Image: The vertical axis represents frequency (low-end bass/kick to high-end cymbals), and the horizontal axis represents time. The pixel intensity represents energy in decibels (dB).
-
2D Convolutional Kernels (
$3\times 3, 5\times 5$ ): Small learnable filter matrices slide (convolve) across the spectrogram:- Vertical Edges: Detect instantaneous broadband transient attacks (kick drum downbeats, snare backbeats, hi-hat ticks).
- Horizontal Stripes: Detect sustained tonal frequencies and basslines.
- Repetitive Textures: Detect periodic tempo patterns and recurring rhythmic motifs.
- Batch Normalization & Non-linear Activation (ReLU): Stabilizes layer activations, prevents internal covariate shift, and enables the network to learn non-linear musical relationships.
-
Hierarchical Pooling: Max-pooling layers progressively downsample spatial dimensions, allowing deeper layers to see larger temporal receptive fields (from milliseconds
$\to$ individual beats$\to$ full musical bars).
+------------------------------------------------------------------------------------------------------+
| Model Family | Architectural Components | How It Learns & Operates |
+--------------------+---------------------------------------+-----------------------------------------+
| 1. TinyCNN | 2D Conv + BatchNorm + ReLU + Adaptive | Extracts spatial-frequency features and |
| | 2D Pooling + Linear Multi-Task Heads | collapses time into a compact embedding.|
+--------------------+---------------------------------------+-----------------------------------------+
| 2. CRNN | 2D Conv Extractor + Bidirectional | Conv layers detect per-frame hits; |
| | GRU (Sequence Modeling) + Mean Pool | BiGRU tracks sequence timing & accents. |
+--------------------+---------------------------------------+-----------------------------------------+
| 3. DualInputNet | Parallel Log-Mel & Tempogram Towers | Jointly models acoustic timbre and |
| | + Dense Fusion Layer (128 units) | explicit tempo periodicity spectra. |
+--------------------+---------------------------------------+-----------------------------------------+
- Configurable convolution blocks (
$B \in [2, 4]$ ) with channel progression:$C \to 2C \to 4C \to 8C$ ($C \in {16, 32, 64}$ ). - Uses
AdaptiveAvgPool2d((1, 1))to convert variable-length audio spectrograms into fixed-size latent vectors. - Optimized for minimal CPU/MPS latency (< 5 ms) and sub-megabyte footprints.
- Why Recurrent? Rhythm is inherently sequential. A pure CNN sees local patches, but a recurrent layer remembers what happened 2 bars ago.
- The 2D CNN downsamples frequency while preserving the temporal dimension (
$T$ ). - The feature sequence is passed to a Bidirectional Gated Recurrent Unit (BiGRU): $$\vec{h}t = \text{GRU}{\text{fwd}}(\vec{x}t, \vec{h}{t-1}), \quad \overleftarrow{h}t = \text{GRU}{\text{bwd}}(\vec{x}t, \overleftarrow{h}{t+1})$$
- Captures meter subdivisions (
$4/4$ vs$3/4$ ), syncopated swing feels, and tempo stability over time.
- Log-Mel Tower: Analyzes acoustic timbre, attack characteristics, and frequency distribution.
- Fourier Tempogram Tower: Analyzes localized autocorrelation and tempo harmonics.
- Fusion Layer: Concatenates both latent representations: $$\mathbf{z}{\text{fused}} = \text{ReLU}\left(\mathbf{W}f [\mathbf{z}{\text{mel}} ,|, \mathbf{z}{\text{tempo}}] + \mathbf{b}_f\right)$$
- Delivers the highest predictive accuracy across complex cross-genre audio.
AudioMIR optimizes both tempo estimation and style classification simultaneously using a joint loss function:
-
Log-Space Tempo Regression: Instead of predicting raw linear BPM directly (which has high variance across 60β200 BPM), the model predicts
$z = \log_2(\text{BPM})$ . The predicted tempo in BPM is reconstructed during inference via$\hat{\text{BPM}} = 2^{\hat{z}}$ . - Backpropagation: Gradients from both tempo regression and style classification backpropagate through shared convolutional layers using the AdamW optimizer with Cosine Annealing learning rate schedules.
-
Tempo Regression Head: Predicts
$\hat{z} = \log_2(\text{BPM})$ using Smooth L1 Loss. The predicted linear tempo is recovered via:$$\hat{\text{BPM}} = 2^{\hat{z}}$$ This formulation handles octave jumps and wide tempo ranges smoothly. - Rhythm / Style Head: Multi-class classification predicting genre/style logits (Rock, Funk, Jazz, Latin, etc.) via Cross-Entropy Loss.
-
Meter Head (Optional): Time-signature classification (
$4/4, 3/4, 6/8$ ) with automatic disablement if class imbalance exceeds thresholds.
AudioMIR implements a fully transparent, native NSGA-II (Non-dominated Sorting Genetic Algorithm II):
-
Multi-Objective Problem Formulation: $$\max_{c \in \mathcal{C}} \quad \mathbf{F}(c) = \begin{bmatrix} \text{Tempo Accuracy}{\pm 4%}(c) \ \text{Style Macro-F1}(c) \ -\text{Median Latency}{\text{ms}}(c) \ -\text{Model Size}_{\text{MB}}(c) \end{bmatrix}$$
-
Core Algorithmic Components:
-
Fast Non-dominated Sorting: Partitions candidate populations into Pareto fronts
$\mathcal{F}_1, \mathcal{F}_2, \dots, \mathcal{F}_k$ . - Crowding Distance Calculation: Enforces solution diversity along the frontier.
- Binary Tournament Selection: Prefers individuals with lower Pareto rank, breaking ties using crowding distance.
- Hyperparameter Crossover & Mutation: Discrete architectural mutation and continuous Gaussian jitter in log-learning-rate and weight-decay spaces.
-
Elitist Selection:
$(N + N) \to N$ pool selection guaranteeing best Pareto candidates are preserved across generations.
-
Fast Non-dominated Sorting: Partitions candidate populations into Pareto fronts
-
Baseline Comparison:
- Random Search: Uniform parameter sampling under equal candidate budget.
- Optuna TPE: Bayesian optimization using Tree-structured Parzen Estimator.
- 1,150+ real drum performances recorded on Roland V-Drums by professional drummers (13.6+ hours).
- Preserves the official Train, Validation, and Test splits.
- User-supplied drum loop collections formatted via CSV manifests:
audio_path,bpm,meter,genre,source_id loops/001.wav,120,4/4,rock,pack01_loop01 loops/002.wav,120,4/4,rock,pack01_loop01_var2 loops/003.wav,90,4/4,hiphop,pack03_loop07
-
Group-Aware Anti-Leakage Protocol: Strictly guarantees that all variations sharing the same
source_idremain in exactly ONE split: $$\text{Train}{\text{groups}} \cap \text{Val}{\text{groups}} = \emptyset, \quad \text{Train}{\text{groups}} \cap \text{Test}{\text{groups}} = \emptyset, \quad \text{Val}{\text{groups}} \cap \text{Test}{\text{groups}} = \emptyset$$
- Real-time synthesis of kicks, snares, and hi-hats for instant unit testing and CI without downloading large datasets.
| Metric | Category | Description |
|---|---|---|
| Tempo MAE (BPM) | Tempo | Mean absolute error in BPM |
| Tempo Median AE | Tempo | Median absolute error in BPM |
| Tempo |
Tempo | Percentage of predictions within |
| Tempo |
Tempo | Percentage of predictions within |
| Octave-Aware Accuracy | Tempo | Tolerance-aware accuracy considering |
| Half / Double Rate | Tempo | Ambiguity tracking for half-tempo and double-tempo errors |
| Macro-F1 (Primary) | Style / Meter | Class-balanced harmonic mean of precision and recall |
| Weighted-F1 / Accuracy | Style / Meter | Global classification accuracy and weighted F1 |
| Median / p95 Latency | Efficiency | Per-sample inference time (ms) at batch size = 1 with warm-up |
| Model Size (MB) | Efficiency | Serialized state_dict disk footprint in megabytes |
| Parameter Count | Efficiency | Total learnable weights |
# 1. Clone the repository
git clone https://github.com/srknskr/AudioMIR.git
cd AudioMIR
# 2. Create Python virtual environment (Python 3.11+)
python3 -m venv .venv
source .venv/bin/activate
# 3. Install package and dependencies in editable mode
pip install --upgrade pip
pip install -e ".[dev]"
# 4. Run test suite
pytest -vHardware Acceleration: AudioMIR automatically detects and utilizes
CUDA(Nvidia GPUs),Apple MPS(Apple Silicon M1/M2/M3/M4), or falls back toCPU.
python scripts/download_groove.pypython scripts/create_manifest.py \
--audio-dir "/path/to/your/drum_loops" \
--output data/custom_loops_manifest.csv \
--infer-from-filenamepython scripts/train_baseline.py --config configs/standard.yaml# Random Search Baseline (50 candidates)
python scripts/run_search.py --strategy random --evaluations 50 --config configs/standard.yaml
# Optuna TPE Bayesian Baseline (50 candidates)
python scripts/run_search.py --strategy tpe --evaluations 50 --config configs/standard.yaml
# Evolutionary Multi-Objective Pareto Search (NSGA-II)
python scripts/run_search.py --strategy evolutionary --evaluations 50 --config configs/standard.yaml# Full-fidelity retraining of discovered Pareto models
python scripts/retrain_pareto.py --run-id <RUN_ID>
# Final test evaluation on untouched Test Set
python scripts/benchmark.py --run-id <RUN_ID>python -m automir.experiments.reproduce <RUN_ID>Launch the live interactive web demo:
streamlit run dashboard/app.py- Audio Upload: Drag and drop
WAV,MP3,FLAC, orOGGfiles (or use built-in synthetic rhythm generator). - 4 Pareto Presets:
- π Best Accuracy: Highest predictive capability.
- βοΈ Balanced: Optimal trade-off between latency, size, and accuracy.
- β‘ Fastest: Ultra-low latency model.
- πͺΆ Smallest: Minimum memory and disk footprint.
- Audio Visualizers: Interactive waveform and dynamic Log-Mel / Tempogram heatmaps.
- Interactive Pareto Front: 2D scatter plots (Plotly) exploring multi-objective trade-offs with hover inspections.
Every experiment is persistently stored in results/experiments.sqlite and JSON records:
- Global random seeds for Python, NumPy, and PyTorch.
- Git commit hash at the time of execution.
- Hardware device metadata (processor, GPU/MPS name, OS version).
- Per-candidate hyperparameters and objective evaluation metrics.
- Generated trade-off scatter plots in
results/<RUN_ID>/plots/*.png.
AudioMIR/
βββ .github/workflows/ci.yml # GitHub Actions CI workflow
βββ .gitignore # Data, caches, and weight exclusions
βββ LICENSE # MIT License
βββ README.md # Comprehensive documentation
βββ pyproject.toml # Python package metadata
βββ requirements.txt # Dependency specifications
βββ configs/
β βββ quick.yaml # Fast local/CI testing configuration
β βββ standard.yaml # Standard local experiment configuration
β βββ research.yaml # High-budget GPU workstation configuration
βββ automir/
β βββ utils/ # Device selector (CUDA/MPS/CPU) & Seed manager
β βββ audio/ # Transforms (Mel, Tempogram) & SHA-256 cache
β βββ datasets/ # Groove, Serkan Loops, Synthetic data loaders
β βββ models/ # TinyCNN, CRNN, DualInputNet & Multi-Task Heads
β βββ training/ # Multi-Task Loss & Multi-Fidelity Trainer
β βββ evaluation/ # Tempo & Style metrics, Latency benchmark, Pareto
β βββ automl/ # Search Space, Random, TPE, Evolutionary (NSGA-II)
β βββ experiments/ # SQLite store, JSON records, Reproduce runner
β βββ inference/ # Production inference engine for audio files
βββ dashboard/
β βββ app.py # Streamlit interactive application
βββ scripts/
β βββ download_groove.py # Groove MIDI downloader
β βββ create_manifest.py # Custom manifest generator with anti-leakage
β βββ train_baseline.py # Baseline model training CLI
β βββ run_search.py # AutoML search runner CLI
β βββ retrain_pareto.py # Full-fidelity Pareto retraining CLI
β βββ benchmark.py # Test set evaluation CLI
βββ tests/ # 25 unit and integration tests
βββ docs/ # Methodology, experiments, reproducibility docs
βββ results/ # SQLite database, run directories, checkpoints
This project is licensed under the MIT License - see the LICENSE file for details.
@misc{automir2026,
author = {Serkan Seker},
title = {AudioMIR: Multi-Objective Automated Machine Learning for Rhythm Understanding},
year = {2026},
publisher = {GitHub},
url = {https://github.com/srknskr/AudioMIR}
}
