Nanoflare

Nanoflare runs calibrated PyTorch models inside C++ audio plugins — real-time inference without linking Libtorch.

It is a header-only C++17 library designed as a fast, lightweight alternative to Libtorch for real-time inference of Pytorch models, and was originally developed for using calibrated Pytorch models in audio plugins. It drops into a project as a Git submodule, so you ship a plugin with a small static dependency instead of the full Libtorch toolkit.

• Header-only C++17 — no Libtorch, no runtime libraries to deploy alongside your plugin
• Made for real-time — causal layers and block-based forward() into buffers you own
• Extensible — new layers and models live in your own repository and register themselves at runtime
• Numerically verified — every layer and model is accuracy-tested against Libtorch references
• Used in shipping audio plugins by Rockedge Audio

Usage

Nanoflare consists of a Python library located in the pynanoflare folder which acts as a wrapper for various Pytorch modules, and a corresponding header-only C++ library located in the include folder.

Calibrating and exporting a Nanoflare compatible Pytorch model to C++ is easy and involves the following steps:

• Calibrating the model implemented in Python using the pynanoflare module
• Serialising it to JSON with the generate_doc method
• Loading its C++ instance using the Nanoflare::ModelBuilder class

If you would like to use your own neural network architecture, you would just:

• Define its Python class using the pynanoflare module
• Add a generate_doc function that handles its JSON serialisation
• Write a C++ equivalent version that derives from the Nanoflare::BaseModel virtual abstract class.

New models can be trained and exported as any other built-in network architectures. Examining the Python and C++ code for the models provided and Nanoflare::ModelBuilder are great ressources for understanding how the code is structured.

Models are registered to Nanoflare::ModelBuilder at runtime, so new models can be defined in their own Python and C++ modules while still being managed by this class.

Layers & Models

The basic layer types currently available are:

• BatchNorm1d
• Biquad
• Conv1d
• GRU
• GRUCell
• Linear
• LSTM
• LSTMCell
• PReLU

They were used to define the following custom block types:

• CausalDilatedConv1d
• FiLM
• MicroTCNBlock
• PlainSequential
• ResidualBlock
• TCNBlock

Which were in turn used to define the following models:

• MicroTCN
• ResRNN e.g. ResGRU or ResLSTM
• TCN
• WaveNet

Building and Dependencies

The library uses Eigen3 for fast matrix computation, and nlohmann::json for saving and loading models to file. Both are defined as Git submodules and built with the library.

A simple way of using the library is to register it as a Git submodule to your project, add it as a sub-directory, and define the include folders with the NANOFLARE_INCLUDE_DIRS variable:

add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/nanoflare)

include_directories(${NANOFLARE_INCLUDE_DIRS})

Tests

The tests are handled by the Catch2 testing framework also defined as a Git submodule.

The accuracy tests use Libtorch as a reference to verify numerical correctness. When configuring, pass the path to the Libtorch directory via CMAKE_PREFIX_PATH:

mkdir build && cd build
cmake .. -DNANOFLARE_TESTING=ON -DCMAKE_PREFIX_PATH=<path/to/libtorch>
cmake --build . --config Release 

Run accuracy tests:

make test

The generate_tests_data.py script generates the test data used by the accuracy tests. Install the Python dependencies first with uv, then run the script:

uv sync
uv run generate_tests_data.py

Benchmarks

Inference cost per model against the same architectures exported to TorchScript (the standard way of deploying PyTorch to C++), measured at a 128-sample block — roughly 2.9 ms of real-time budget at 44.1 kHz — with both implementations running single-threaded on the same machine:

Model	Nanoflare	TorchScript	Speedup
MicroTCN	0.24 ms	1.48 ms	6.2x
TCN	0.35 ms	2.05 ms	5.8x
WaveNet	0.46 ms	1.85 ms	4.0x
ResGRU	0.86 ms	17.46 ms	20.3x
ResLSTM	0.76 ms	1.58 ms	2.1x

Every model completes well inside a single 128-sample block, which is what makes callback-safe deployment possible. Per-layer benchmarks live in tests/layers_benchmarking.cpp.

Extending Nanoflare with Custom Models

Nanoflare uses static initialization to register models automatically. This allows you to add custom models in separate repositories without modifying nanoflare's code. This is useful for proprietary models or research projects.

Creating a Private Model Repository

1. Repository Structure:

your-private-models/
├── CMakeLists.txt
├── models/
│   ├── YourModel.h              # C++ implementation
│   └── PrivateModels.h          # Registration header
├── python/
│   ├── your_model.py            # Python implementation
│   └── __init__.py
└── nanoflare/                   # Git submodule pointing to this repo

2. C++ Model Registration:

Create a header file that registers your models using static initialization:

// models/PrivateModels.h
#pragma once

#include "nanoflare/ModelBuilder.h"
#include "YourModel.h"
#include "AnotherModel.h"

namespace YourNamespace
{
    namespace {
        inline bool registerPrivateModels()
        {
            // Register your custom models
            Nanoflare::registerModel<YourModel>("YourModel");
            Nanoflare::registerModel<AnotherModel>("AnotherModel");
            return true;
        }

        // Auto-register during static initialization (before main())
        static const bool _privateModelsRegistered = registerPrivateModels();
    }
}

3. CMake Integration:

In your private repository's CMakeLists.txt:

cmake_minimum_required(VERSION 3.24)
project(YourPrivateModels)

# Add nanoflare as subdirectory (git submodule)
add_subdirectory(nanoflare)

# Create your models library
add_library(your_models INTERFACE)
target_link_libraries(your_models INTERFACE nanoflare)
target_include_directories(your_models INTERFACE
    ${CMAKE_CURRENT_SOURCE_DIR}/models
)

4. Using Custom Models in C++:

Simply include your registration header before using ModelBuilder:

#include "nanoflare/ModelBuilder.h"
#include "models/PrivateModels.h"  // Auto-registers via static initialization

// Now your models are registered and can be loaded from JSON
std::ifstream model_file("your_model.json");
std::shared_ptr<Nanoflare::BaseModel> model;
nlohmann::json j = nlohmann::json::parse(model_file);
Nanoflare::ModelBuilder::getInstance().buildModel(j, model);

License

Apache License 2.0 — see LICENSE.

Dependencies are permissive and included as Git submodules: Eigen (MPL-2.0), nlohmann/json (MIT) and Catch2 (BSL-1.0, tests only). Libtorch is needed to build the accuracy tests, never to use the library.