evals

Mastra is the modern TypeScript framework for AI-powered applications and agents.

AI Observability & Evaluation

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI

Framework for evaluating and improving agents

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

Observability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement.

AI observability platform for production LLM and agent systems.

Evaluation and Tracking for LLM Experiments and AI Agents

AI system design guide for engineers building production AI systems and evals.

Laminar - open-source observability platform purpose-built for AI agents. YC S24.

Build tested agent skills and govern their lifecycle through a user-defined marketplace: evidence, discovery, updates, rollback, quarantine, and 17-platform distribution.

Testing and evaluation platform to chat, inspect, and debug MCP servers, MCP apps, and ChatGPT apps.

Waku Waku! Waku Agent is a local-first AI agent harness you actually own, including loop, memory, eval, all in code built to stay legible as it grows.

Evaluate your LLM-powered apps with TypeScript

OpenSource Production ready Customer service with built in Evals and monitoring

Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.

Build agentic systems. Run them with confidence. Orchestrate agents, automate business processes, inspect every execution, and keep humans in control. Deploy Heym on your own infrastructure.

🥤 RAGLite is a Python toolkit for Retrieval-Augmented Generation (RAG) with DuckDB or PostgreSQL

Low-level agentic design patterns. Turning old engineering wisdom into reflexes your agent reaches for on its own—on any agent.

A curated, non-BS library of the best resources for building and evaluating AI agents — papers, blogs, talks, tools, benchmarks. Maintained by BenchFlow.