mxfp4

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

A SOTA quantization toolkit for high-accuracy low-bit LLM inference|简洁且高效的量化工具包

CPU inference for Kimi K3, a 2.78T-parameter MoE LLM, in pure Rust. No GPU, no BLAS, no PyTorch. Streams the checkpoint from disk. Byte-identical port of kimi-k3-in-c.

The Kimi K3 and DeepSeek-V4-Flash-0731 architectures, reimplemented from scratch in pure Rust (zero dependencies) and verified 1:1 against the official code.

Tiny engine, immense models — run large MoE LLMs (gpt-oss, Mixtral, Qwen3-MoE) on ordinary machines by streaming experts from disk. OpenAI-compatible server with tool calling + hybrid cloud relay; CPU, Apple Silicon (MLX) & CUDA.

vLLM for AMD RDNA4 (gfx1201): Radeon AI PRO R9700 & RX 9070 XT — native MXFP4 linear kernel + spec-decode verify fix, carried as rebased branches (fork-carry model)

面向 8×RTX 4090（SM89）的 DeepSeek-V4-Flash-0731 MXFP4 SGLang 实验分支

Run Kimi-K3 (2.8T MoE) from SSD with a tiny readable C++17 engine—native MXFP4, io_uring, and O_DIRECT.

Run the official Kimi K3 MoE checkpoint on one 128 GB AMD Strix Halo box. ROCm-resident static weights, MXFP4 experts streamed from NVMe via io_uring. C engine, chat client, OpenAI-compatible server. Very experimental.

Local inference engine for DeepSeek-V4-Flash (284B MoE, MXFP4 experts streamed from disk) on consumer hardware — C11, zero deps, bit-exactness gated. macOS / Linux / Windows.

Train and serve MoE models that do not fit in VRAM: fused 4-bit experts, QLoRA, CPU/NVMe offload, and fast inference on consumer NVIDIA GPUs.

Triton kernels for 4-bit MoE inference: grouped NF4/MXFP4 GEMM, INT4 GEMV, FP8 paged attention, and CPU/NVMe expert streaming.

Qwen3-8B quantization study across vLLM, TensorRT-LLM, AutoRound, INT8, and MXFP4

Out-of-core LoRA fine-tuning and expert-routing measurement for Kimi K3 (2.78T MoE) on a 7.6 GB laptop: 93 layers streamed from a USB disk, forward checked against an independent C implementation, gradients checked by finite differences

Run a 2.78T parameter MoE LLM on CPU in pure Rust with only 8 GB RAM.

Field notes from optimizing LLM inference on an NVIDIA DGX Spark. Everything measured on the machine.

GRIT — Grouped Reduced-precision Interchange Type: a checkable 64-byte descriptor + contract for block-scaled, sparse, sub-byte tensors. Spec, conformance vectors, and 5 zero-dependency implementations (C, C++, Rust, Python, TypeScript). grit scan audits GGUF/safetensors files you already have.

CPU-first Rust research implementation of OpenAI GPT-OSS with exact MXFP4 kernels, reproducible benchmarks, and archived accelerator research.

Run OpenAI's gpt-oss-20b on AMD Radeon (RDNA2) via a from-scratch Vulkan compute backend. No CUDA, no ROCm, no Triton.

MXFP/NVFP/FP16 to GGUF conversion with automatic safe routing