cuda-graph

PINNs-Torch, Physics-informed Neural Networks (PINNs) implemented in PyTorch.

Foundry materializes CUDA graphs along with its execution context to disk to support fast cold start of serving engines.

A light, transparent, and modular inference & quantization engine for studying LLMs.

A comprehensive guide to using CUDA Graphs effectively with PyTorch, covering CUDA fundamentals, PyTorch integration, Megatron-LM implementations, and practical troubleshooting.

Handwritten Qwen2.5-0.5B inference engine — no vLLM, no model.generate(); CUDA Graph + hand-written Triton flash kernels, 3.34x vLLM decode TPS on a single RTX 2080 Ti (96.15/100). Independent work at ECNU summer school.

PyTorch CUDA Graph Capture Examples

This repository contains the source code for Grape.

A minimalist, production-ready framework for training small language models through 4 sequential stages (Pretraining, SFT, Alignment, Reasoning). Optimized for extreme performance with a hardware-aware inference engine featuring CUDA Graphs, Static KV Caching, and Topic-Aware Conversational Memory.

A vLLM-class inference and RL-rollout engine for embodied models with cross-environment batching, episode-affine data parallelism, and CUDA-graph capture.

Compression-aware Qwen3-VL inference engine with torch.compile, CUDA Graph, scaled-FP8 KV, visual compaction, and multimodal prefix caching.

🚀 从零手写的 Qwen3 高性能推理引擎 —— 1,500 行纯 Python 实现 Paged KV Cache · 连续批处理 · FlashAttention-2 · CUDA Graph,零框架依赖(不基于 vLLM/SGLang),单卡 256 并发 1,200+ tok/s

Enhancing CUDA Intra-Streaming-Multiprocessor Parallelism for Large Language Models via Fine-Grained Task Graph

Profiling CUDA Graph vs torch.compile vs eager decode on RTX 2070 — measuring per-token latency reduction across batch sizes, sequence lengths, and GPT-2 model sizes.

Batched speculative decoding in nano-vLLM: 1.50x at B=1, 1.17x at B=8 with Qwen3-8B/0.6B on RTX PRO 6000 Blackwell.

实时语音转写系统：FireRedASR2S + CUDA Graph 解码加速（RTF 0.024，20 秒音频 0.47 秒）+ 声纹说话人分离（在线句内声学分段 + 离线定版两遍式）。Real-time ASR with voiceprint diarization.

Qwen3.8-27B 双 RTX 2080 Ti 22GB + NVLink 的 vLLM 部署（上游 main 路线）：SM75 移植、MTP 下保住 FULL cudagraph（单并发解码 54.1→37.5 ms/步）、256K/500K 上下文与 128K tiered offload profile